System

Abstract

A system is provided.SOLUTION: A system including means for a user to photograph and upload an image of a food, means for a server to analyze the received image and identify the food, means for the server to acquire nutrition information of the identified food and chart a nutrition balance, means for the terminal to display the nutrition balance chart to a user, means for the user to input a chronic disease information and a health goal and for the server to individually make a suitable diet proposal, means for supporting purchase of foodstuffs according to a budget in a convenience store and a supermarket, and means for linking the nutrition balance information with a medical institution.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, many people find dietary management difficult, especially for those with busy lifestyles or certain health conditions. Choosing nutritionally balanced meals is not easy, and the difficulty is increasing with the increasing use of eating out and convenient foods. It is also difficult to regularly track dietary information and efficiently share that information with medical institutions. To address these challenges, a system is needed that allows users to easily record their dietary information, visualize nutritional balance, and provide dietary suggestions tailored to individual health goals. [Means for solving the problem]

[0005] The present invention provides a system that facilitates dietary management by allowing users to take photos of foods, chart their nutritional balance based on those photos, and provide dietary suggestions based on individual health goals. This system includes a means for users to take and upload food images, a means for a server to analyze the received images and identify the foods, a means for the server to acquire nutritional information for the identified foods and chart their nutritional balance, a means for displaying the nutritional balance chart to the user via a terminal, a means for the user to input information about chronic illnesses and health goals and for the server to provide individually appropriate dietary suggestions, a means for supporting budget-based food purchasing at convenience stores and supermarkets, and a means for linking nutritional balance information with medical institutions. This allows users to easily understand their diet and receive appropriate support for maintaining a healthy lifestyle.

[0006] "User" refers to an individual who uses this system to manage their diet and analyze their nutritional balance.

[0007] "Terminal" refers to electronic devices used by users, such as smartphones, tablets, and personal computers.

[0008] "Server" refers to a computer system that provides functions such as data analysis, processing, and storage, and communicates with user terminals.

[0009] "Food Images" refers to photographs taken by users and uploaded to the system to record their meals.

[0010] "Analyzing images" refers to the process by which the server uses machine learning and image recognition technology to identify the type and quantity of food.

[0011] "Identifying food" refers to identifying the labels and attributes of each food item recognized through image analysis.

[0012] "Nutrition information" refers to data on nutrients such as calories, protein, fat, carbohydrates, vitamins, and minerals contained in a particular food.

[0013] "Nutritional balance" refers to the distribution and balance of nutrients based on the food consumed by the user.

[0014] "Charting" refers to converting collected nutrition information into graphs or charts for visual display.

[0015] "Pre-existing medical information" refers to data about a user's existing illnesses or health conditions.

[0016] "Health goals" refer to the health status or nutritional management goals that a user wants to achieve (e.g., dieting, measures against metabolic syndrome, etc.).

[0017] "Meal suggestions" refers to recommending appropriate meals and recipes based on the user's health goals and nutritional balance.

[0018] "Food purchase" refers to the user's behavior of purchasing healthy foods at convenience stores or supermarkets.

[0019] "Medical institution" refers to a hospital, clinic, medical office, etc. that manages or treats the user's health.

[0020] "Collaboration" refers to the system sharing users' nutritional balance information with medical institutions and providing appropriate feedback. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0022] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0023] First, the terms used in the following description will be explained.

[0024] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0025] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0026] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0027] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0028] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0029] [First embodiment]

[0030] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0031] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0033] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0034] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0035] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0037] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0038] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0040] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0041] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0042] This invention is a system that allows users to take pictures of foods, chart their nutritional balance based on those pictures, and make dietary suggestions based on individual health goals. This system includes a means for users to take and upload food images, a means for a server to analyze the received images and identify the foods, a means for the server to obtain nutritional information for the identified foods and chart their nutritional balance, a means for displaying the nutritional balance chart to the user via a terminal, a means for the user to input information about chronic illnesses and health goals and for the server to make individually appropriate dietary suggestions, a means for supporting budget-based food purchasing at convenience stores and supermarkets, and a means for linking nutritional balance information with medical institutions.

[0043] Program processing

[0044] Image upload

[0045] A user opens the application and takes a photo of a food. The image is saved on the device and an upload button is displayed. When the user presses the upload button, the image is sent to the server.

[0046] Image analysis

[0047] The server adds the received image to the analysis queue. It invokes the image analysis module, which first preprocesses the image (e.g., resizes the image, removes noise, etc.). The server then uses a machine learning model to recognize food in the image. Specifically, it segments the image region, identifies each food item, and labels each one. At this stage, the type and quantity of food are estimated.

[0048] Acquiring food data

[0049] The server queries the food database based on the label of each identified food, and retrieves detailed nutritional information (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) for each food in the database.

[0050] Nutritional balance chart

[0051] The server aggregates the acquired nutritional information and creates a chart of nutritional balance based on the generated information. The nutritional balance chart is displayed in a visually easy-to-understand format (radar chart, bar graph, etc.).

[0052] Displaying the results

[0053] The server sends the chart data in JSON format to the device, which then visually displays the chart to the user using a drawing library. The user can refer to this chart and understand their diet at a glance.

[0054] Customized Offers

[0055] Users enter their health goals and chronic illness information within the app. The device sends this information to the server, which uses this information to update the user's profile and provide personalized dietary suggestions. These suggestions are then sent to the device and displayed to the user.

[0056] Food suggestions according to your budget

[0057] The user enters their desired budget amount using a budget input form. The device sends the budget information to the server. The server queries the product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within the budget. The server filters the list based on budget and nutritional balance and sends the selected list to the device. The device displays this list to the user.

[0058] Collaboration with medical institutions

[0059] The user enables the "Link with medical institutions" option in the app settings. The device sends nutritional balance data to the server with the user's permission. The server takes security measures and shares the user data through an API dedicated to medical institutions. Medical institutions use this data to provide the user with appropriate feedback and treatment plans.

[0060] Specific examples

[0061] A user eats a hamburger steak set meal (rice, hamburger steak, salad) for lunch and uploads a photo of it to the app. The device sends the captured image to the server, which analyzes the image and identifies each food item. The server obtains nutritional information based on the identification results and creates a chart of nutritional balance. The chart is then sent to the device and the user can visually confirm it. Furthermore, if the user has a diet goal, they input that goal into the app. The server takes this into consideration and suggests a low-calorie dinner menu (e.g., a recipe for salad and grilled chicken). If a user sets a budget of 1,000 yen for the next day and wants to buy a healthy lunch, the server will suggest convenience store items (e.g., sandwiches, salad, and tea) that can be purchased within that budget. Users can maintain a healthy diet by referring to the ingredient list when making purchases. Users can also share their nutritional balance information with medical institutions and engage in regular health management.

[0062] In this way, the system allows users to easily and efficiently manage their daily diet and receive appropriate support according to their individual health conditions.

[0063] The processing flow will be explained below.

[0064] Step 1:

[0065] A user opens the application and takes a photo of the food. After taking the photo, the user presses the "upload" button.

[0066] Step 2:

[0067] The terminal stores the captured image file in a temporary storage area within the application, and transmits the image data to the server.

[0068] Step 3:

[0069] The server adds the received image data to the analysis queue and invokes the image analysis module. As initial preprocessing, it performs image resizing, noise removal, color adjustment, etc.

[0070] Step 4:

[0071] The server uses machine learning models to analyze the image, identify food item regions, label each item, and estimate the quantity of each food item.

[0072] Step 5:

[0073] The server queries the food database based on the label of each recognized food item, and obtains detailed nutritional information for each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.).

[0074] Step 6:

[0075] The server compares the collected nutritional information with the user's recommended daily intake and generates a nutritional balance chart in a visually easy-to-understand format such as a radar chart or bar graph.

[0076] Step 7:

[0077] The server sends the generated chart data in JSON format to the terminal.

[0078] Step 8:

[0079] The device uses a drawing library to display the chart data received to the user, who can then check the chart to understand their own nutritional balance.

[0080] Step 9:

[0081] Users enter and update their health goals and chronic illness information within the app.

[0082] Step 10:

[0083] The terminal transmits the input information to the server.

[0084] Step 11:

[0085] The server updates the user's profile based on their health goals and chronic illnesses, and generates suitable meal suggestions, including recipes and menus for dining out.

[0086] Step 12:

[0087] The server sends meal suggestions to the terminal.

[0088] Step 13:

[0089] The device displays suggested recipes and meal menus to the user, who can then use them to select a meal.

[0090] Step 14:

[0091] The user inputs the budget for purchasing ingredients, for example, "1000 yen."

[0092] Step 15:

[0093] The terminal transmits the budget information to the server.

[0094] Step 16:

[0095] The server queries product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within a budget. The list is filtered to take nutritional balance into consideration.

[0096] Step 17:

[0097] The server transmits the generated food list to the terminal.

[0098] Step 18:

[0099] The terminal displays a food list to the user, and the user uses the list as a reference when purchasing ingredients.

[0100] Step 19:

[0101] The user enables the "Connect with healthcare providers" option in the app settings.

[0102] Step 20:

[0103] The terminal confirms the user's permission and transmits the permission information along with the nutritional balance data to the server.

[0104] Step 21:

[0105] The server takes appropriate security measures and shares user data through an API dedicated to medical institutions.

[0106] Step 22:

[0107] Medical institutions receive the shared data and provide appropriate feedback and treatment plans to users.

[0108] Example 1

[0109] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0110] Currently, there are many dietary management tools and applications on the market, but most of them have limitations in their ability to effectively understand the user's dietary content and nutritional balance and make personalized suggestions based on that information. They also lack the functionality to suggest ingredients based on the user's budget or to support health management in collaboration with medical institutions. There is a need for a system that can improve these points and enable users to manage their health more effectively.

[0111] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0112] In this invention, the server includes means for preprocessing the received images and identifying foods using a machine learning model, means for acquiring nutritional information for the identified foods and charting their nutritional balance, and means for linking the nutritional balance information with medical institutions. This enables users to easily manage their diet through food images, receive personalized dietary suggestions, and even manage their health consistently through collaboration with medical institutions.

[0113] "User" refers to an individual who uses the system to take pictures of food and manage their diet.

[0114] "Server" refers to a central processing unit for receiving and analyzing data sent by users and providing necessary information.

[0115] "Terminal" refers to a computing device (such as a smartphone, tablet, or PC) used by a user.

[0116] "Food image" refers to photographic data containing food that a user takes and uploads to the system.

[0117] "Preprocessing" refers to the initial processing such as resizing and noise removal that is performed before the server analyzes the image.

[0118] A "machine learning model" refers to an algorithm that learns patterns from data and identifies and classifies foods.

[0119] A "food database" refers to a collection of data that stores nutritional information about each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.).

[0120] "Charting nutritional balance" refers to presenting collected nutritional information in a visually easy-to-understand format (such as a radar chart or bar graph).

[0121] "Health goal" refers to a goal related to a user's health status (e.g., dieting, muscle building, managing a chronic illness, etc.).

[0122] "Meal suggestions" refers to recommendations for providing an individualized meal plan based on the user's health goals and chronic illness information.

[0123] "Support for food purchasing" refers to providing users with a list of foods that can be purchased within their budget, and assisting them in their purchasing activities at convenience stores and supermarkets.

[0124] "Collaboration with medical institutions" refers to a system in which users' nutritional balance information is shared with medical institutions and feedback and treatment plans are provided as needed.

[0125] "Drawing library" refers to a software component for visually displaying data (e.g., D3.js, Chart.js, Matplotlib, etc.).

[0126] The present invention is a system that allows a user to take images of food, chart the nutritional balance based on the images, and make dietary suggestions based on individual health goals. This system includes means for the user to take and upload images of food, means for a server to analyze the received images and identify the food, means for the server to obtain nutritional information for the identified foods and chart the nutritional balance, means for displaying the nutritional balance chart to the user via a terminal, means for the user to input health goals and chronic illness information and for the server to make individually appropriate dietary suggestions, means for supporting food purchasing according to budget, means for linking nutritional balance information with medical institutions, and means for pre-processing the captured images by resizing and removing noise.

[0127] Hardware and software used

[0128] To realize this system, the following hardware and software are used.

[0129] 1. Hardware

[0130] User device: smartphone, tablet, or computer.

[0131] Server: Server equipment equipped with a high-performance CPU and GPU.

[0132] 2. Software

[0133] Image analysis module: Uses Python and utilizes libraries such as OpenCV and TensorFlow.

[0134] Database: A relational database such as MySQL or PostgreSQL.

[0135] Drawing libraries: Visualization libraries such as D3.js, Chart.js, Matplotlib, etc.

[0136] API: Implements a RESTful API to communicate between the server and the terminal.

[0137] Example of operation

[0138] A specific example of the operation of the system of the present invention will be described below.

[0139] Specific examples

[0140] A user eats a hamburger steak set meal (rice, hamburger steak, salad) for lunch and uploads a photo of it to the app. The user takes a photo of the food using the app's camera function and presses the upload button, which sends the image to the server. The server preprocesses the received image (for example, resizes the image to 256x256 pixels and removes noise) and begins analysis using a machine learning model using TensorFlow. The server analyzes the image, identifies the foods - hamburger steak, rice, and salad - and labels each food.

[0141] The server then queries a MySQL database based on the identified food labels to obtain detailed nutritional information for each food (e.g., calories, protein, fat, carbohydrates, vitamins, and minerals). The obtained nutritional information is aggregated and a nutritional balance chart is created. The server sends this chart information in JSON format to the device, which then visually displays the chart using a drawing library such as D3.js. Users can refer to the chart to understand their diet at a glance.

[0142] Furthermore, when a user sets "diet" as a "health goal" in the app, the server will suggest corresponding low-calorie menus (e.g., salad and grilled chicken recipes). If a user sets a budget of 1,000 yen and wants to buy a healthy lunch, the server will generate a list of foods that can be purchased within that budget (e.g., sandwiches, salads, and tea) and send it to the device. By referring to the list of ingredients when making purchases, the user can maintain a healthy diet.

[0143] Prompt Sentence Examples

[0144] Example inputs to a generative AI model:

[0145] "Write a program that analyzes an image of a person eating oatmeal, banana, and yogurt for breakfast, along with the detailed ingredients, and then charts and displays the nutritional balance of this meal."

[0146] This system allows users to easily and efficiently manage their daily diet and receive appropriate support according to their individual health conditions.

[0147] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0148] Step 1:

[0149] The user takes a picture of a food item and presses the upload button in the application. The device saves the image to its internal storage and sends an upload request to the server. The input data is the food image taken by the user, and the output is the image sent to the server.

[0150] Specific behavior:

[0151] A user takes a photo of a hamburger steak set meal using the camera on their smartphone.

[0152] The device saves the image file (e.g., lunch.jpg) to its internal storage.

[0153] The user taps the upload button in the app and the image is sent to the server.

[0154] Step 2:

[0155] The server receives the uploaded images and adds them to the analysis queue. The server uses image analysis modules to preprocess the images and convert them into an analyzable format. The input data is the original uploaded image, and the output is the preprocessed image data.

[0156] Specific behavior:

[0157] The server resizes the image to 256x256 pixels and denoises it.

[0158] The preprocessed image data is fed into a machine learning model.

[0159] Step 3:

[0160] The server uses a machine learning model (e.g., TensorFlow) to analyze the preprocessed images and identify the foods. It identifies each food item in the image and assigns a label to each. The input data is the preprocessed image data, and the output is the label and estimated quantity of the identified food.

[0161] Specific behavior:

[0162] The server uses the YOLO model to detect food items (e.g., hamburger, rice, salad) in the image.

[0163] The server labels each food item and estimates the quantity (e.g., "hamburger": 1 piece, "rice": 1 bowl, "salad": 1 plate).

[0164] Step 4:

[0165] The server queries the food database based on the label of each identified food and retrieves detailed nutritional information for each food (calories, protein, fat, carbohydrates, vitamins, minerals, etc.). The input data is the label of the identified food, and the output is the retrieved nutritional information.

[0166] Specific behavior:

[0167] The server uses a SELECT query to retrieve nutritional information from the MySQL database based on the label "hamburger."

[0168] Similarly, obtain nutritional information for each food item (e.g., "Calories": 350kcal, "Protein": 25g).

[0169] Step 5:

[0170] The server aggregates the acquired nutritional information and generates chart data to visually represent nutritional balance. The input data is detailed nutritional information for each food, and the output is chart data.

[0171] Specific behavior:

[0172] A server aggregates the nutritional information of all foods.

[0173] The server uses the Matplotlib library to plot the nutritional balance as a radar chart.

[0174] Step 6:

[0175] The server generates chart data and sends it to the terminal in JSON format. The terminal uses a drawing library (e.g., D3.js) to display the chart to the user based on the data received. The input data is the chart data, and the output is a visual chart that is displayed to the user.

[0176] Specific behavior:

[0177] The server converts the chart data into JSON format and sends it to the terminal.

[0178] The device uses Chart.js to draw the nutritional balance in the form of a bar graph and displays it on the app's UI.

[0179] Step 7:

[0180] The user enters their health goals and chronic illness information into the app. The device sends this information to the server, which then makes personalized dietary suggestions. The input data is the user's health goals and chronic illness information, and the output is dietary suggestions.

[0181] Specific behavior:

[0182] The user enters "health goal" as "diet."

[0183] The server will then suggest a low-calorie option (e.g., salad and grilled chicken) based on that information.

[0184] Step 8:

[0185] The user uses a budget input form to input the desired budget amount. The device sends the budget information to the server, which then executes a query to generate a list of foods that can be purchased within the budget. The input data is the user's budget information, and the output is a list of foods that can be purchased within the budget.

[0186] Specific behavior:

[0187] The user sets the "budget" to "1,000 yen" within the app.

[0188] The server generates a list of foods that can be purchased for under 1,000 yen, suggesting sandwiches, salads, and tea.

[0189] Step 9:

[0190] The user enables the "Link with medical institutions" option in the app settings. Based on the user's permission, the device sends nutritional balance data to the server, and the server shares the user data through an API dedicated to medical institutions. The input data is the user's nutritional balance data and medical institution information, and the output is data shared with the medical institution.

[0191] Specific behavior:

[0192] The user enables the "Connect with Healthcare Providers" option.

[0193] The server encrypts the data and sends it to an HTTPS endpoint dedicated to the medical institution.

[0194] (Application example 1)

[0195] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0196] In modern society, there is a demand for appropriate nutritional management and dietary suggestions tailored to each individual's health condition. However, daily dietary management is complicated, and it is particularly difficult to confirm the nutritional balance of food when eating out or purchasing it. Furthermore, when purchasing at convenience stores or supermarkets, it is not easy to select ingredients that take nutritional balance and budget into consideration. Furthermore, there is a demand for more appropriate health management by linking nutritional information with medical institutions, but current systems are not able to adequately address this demand.

[0197] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0198] In this invention, the server includes means for a user to photograph and upload images of foods, means for the server to analyze the received images and identify the foods, means for the server to obtain nutritional information for the identified foods and create a nutritional balance chart, means for the server to display the nutritional balance chart to the user via a terminal, means for the user to input chronic illness information and health goals and for the server to make individually appropriate meal suggestions, means for supporting budget-based food purchasing at sales facilities, means for linking the nutritional balance information with medical institutions, means for analyzing photographed images of food shelves and obtaining nutritional information for each food item to display to the user, and means for suggesting foods that can be purchased within the user's budget based on the user's input data. This allows users to check nutritional information on the spot when purchasing food at a physical store, make healthy food choices within their budget, and even manage their health in cooperation with medical institutions.

[0199] "User" refers to an individual who uses the system to take pictures of food and receive nutritional balance and meal suggestions.

[0200] "Server" refers to a central computer system for analyzing food images, obtaining nutritional information, and displaying it to the user.

[0201] The "means for taking and uploading images of food" refers to the part that provides the function for users to take images of food using a smartphone or smart glasses and send those images to the system.

[0202] "Means for identifying food" refers to technology that analyzes the image received by the server and identifies the food in the image.

[0203] "Means for obtaining nutritional information and charting nutritional balance" refers to the function of collecting nutritional data for identified foods and displaying it in a visually easy-to-understand format.

[0204] "Means for displaying a nutritional balance chart to a user on a terminal" refers to a function that allows a user to see a nutritional balance chart on a terminal such as a smartphone or computer.

[0205] "A means for inputting health goals and for the server to make individually appropriate meal suggestions" refers to a function in which the user inputs information about chronic illnesses and health goals, and the server creates customized meal suggestions based on that information.

[0206] "Means to support food purchasing according to budget at sales facilities" refers to a function that suggests foods that can be purchased taking into account the user's budget and supports the purchase.

[0207] "Means for linking nutritional balance information with medical institutions" refers to a function for sharing a user's nutritional information with medical institutions and supporting health management.

[0208] "Means for analyzing images of food shelves, obtaining nutritional information for each food item, and displaying this information to the user" refers to a function that analyzes images of food shelves taken at a sales facility, obtains nutritional information for each food item, and provides this information to the user.

[0209] "Means for suggesting foods that can be purchased within a budget based on input data" refers to a function that creates and suggests a list of foods that can be purchased within a budget based on the user's input data (budget and health goals).

[0210] This invention is a system that allows a user to take an image of food, charts the nutritional balance based on the image, and makes meal suggestions based on individual health goals. The system includes the following means:

[0211] 1. Image uploading method: Users take pictures of the food shelves using their smartphones or smart glasses and upload them to the server from their devices. The uploaded images are stored on the server.

[0212] 2. Image analysis: The server analyzes the received images and identifies the food in the image. This image analysis is performed using a machine learning model (e.g., using TensorFlow). The server first preprocesses the images (resizes, removes noise, etc.), then identifies and labels the food.

[0213] 3. Nutritional information acquisition means: Based on the food identified from the analyzed image, the server acquires the detailed nutritional information of the food from a food database (e.g., an internally stored nutritional database), including information such as calories, protein, fat, carbohydrates, vitamins, and minerals.

[0214] 4. Nutritional balance charting: The server aggregates the nutritional balance based on the acquired nutritional information and displays it in a visually easy-to-understand format (for example, a radar chart or bar graph). This allows users to understand the nutritional balance of the foods they have consumed at a glance.

[0215] 5. Display of results: The charted nutritional balance is sent from the server to the device, which then uses a visualization library to render it and display it to the user, allowing the user to understand their own dietary habits.

[0216] 6. Customized Meal Suggestion: Users can input their health goals and chronic illness information within the application. This information is sent to the server, which updates the user's profile and provides personalized meal suggestions. These suggestions are sent to the device, providing the user with guidelines for maintaining a healthy diet.

[0217] 7. Budget-based food recommendation: The user inputs their desired budget amount into the application. The server connects to the database of sales facilities to generate a list of foods that can be purchased within the budget. It filters based on budget and nutritional balance and sends the selected food list to the terminal. The terminal displays this list to the user, who can receive assistance in purchasing healthy foods within their budget.

[0218] 8. Linking with medical institutions: When the user enables the "Linking with medical institutions" option, the device will send nutritional balance data to the server with the user's permission. The server will then share the user data through an API dedicated to medical institutions, after taking security measures. Medical institutions will use this data to provide the user with appropriate feedback and treatment plans.

[0219] For example, when a user takes a photo of a food shelf in a physical store and uploads it, the server analyzes the image, obtains nutritional information for each food item, and displays it to the user. Based on this information, the user can select foods that take into account nutritional balance and budget on the spot. Furthermore, by inputting the user's health goals and budget, the server can make optimal meal suggestions.

[0220] Additionally, examples of prompts that are useful as input to generative AI models include:

[0221] "A user takes a photo of a food shelf in a physical store and uploads it. Please explain how the system uses the image to create a nutritional balance chart and provide meal suggestions aligned with the user's health goals. Specifically, please explain in detail the process of image upload, image analysis, nutritional information acquisition, and meal suggestions. Please also include suggestions based on a budget."

[0222] In this way, the system allows users to easily and efficiently manage their daily diet and receive appropriate support according to their individual health conditions.

[0223] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0224] Step 1: Upload an image

[0225] A user takes a photo of the food shelf with a smartphone or smart glasses. The device saves the image and displays an upload button. When the user presses the upload button, the image is sent from the device to the server. The input is the image taken by the user, and the output is the image data sent to the server.

[0226] Step 2: Image analysis

[0227] The server adds the received image to the analysis queue. It invokes the image analysis module to first preprocess the image (resize, remove noise, etc.). The server then uses a generative AI model to recognize food in the image. This process involves segmenting the image area, identifying each food item, and labeling each one. The input is the image data sent to the server, and the output is the label information of the identified food and its location information.

[0228] Step 3: Obtain nutritional information

[0229] The server queries an internal food database based on the label of each identified food, and obtains detailed nutritional information (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) for each food in the database. The input is the label information of the identified food, and the output is the nutritional information.

[0230] Step 4: Nutritional Balance Chart

[0231] The server aggregates the acquired nutritional information and charts the nutritional balance in a visually easy-to-understand format (such as a radar chart or bar graph). The chart data is generated in JSON format. The input is the aggregated nutritional information, and the output is the JSON data of the nutritional balance chart.

[0232] Step 5: View the results

[0233] The server sends the chart data to the terminal, which uses a visualization library to draw the chart and display it visually to the user. The input is the JSON data of the chart, and the output is the nutritional balance chart displayed to the user.

[0234] Step 6: Customized Meal Suggestions

[0235] The user enters their health goals and chronic illness information within the application. The device sends this information to the server. The server updates the user's profile and generates personalized meal suggestions. The suggestions are sent to the device and displayed to the user. The input is the user's health goals and chronic illness information, and the output is personalized meal suggestions.

[0236] Step 7: Proposing ingredients according to your budget

[0237] The user enters their budget in the app. The device sends the budget information to the server. The server queries a database of sales establishments to generate a list of foods that can be purchased within the budget. After filtering, the server sends the selected list of ingredients to the device. The device displays this list to the user. The input is the user's budget information, and the output is a list of foods that the user can purchase.

[0238] Step 8: Collaborate with medical institutions

[0239] The user enables the "Link with medical institutions" option in the app settings. The device sends nutritional balance data to the server with the user's permission. The server takes security measures and shares the user data through an API dedicated to medical institutions. Based on this data, medical institutions provide the user with appropriate feedback and treatment plans. The input is the user's nutritional balance data, and the output is the data sent to the medical institution.

[0240] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0241] The present invention is a system that allows a user to take an image of food, charts the nutritional balance based on the image, and makes meal suggestions based on individual health goals and emotional state. This system includes means for a user to take and upload an image of food, means for a server to analyze the received image and identify the food, means for the server to obtain nutritional information for the identified food and chart the nutritional balance, means for displaying the nutritional balance chart to the user via a terminal, means for the user to input information about chronic illnesses and health goals and for the server to make individually appropriate meal suggestions, means for supporting budget-based food purchasing at convenience stores and supermarkets, means for linking nutritional balance information with medical institutions, and means for recognizing the user's emotions using an emotion engine.

[0242] Program processing

[0243] Image upload and emotion recognition

[0244] The user opens the application and takes a photo of the food. After taking the photo, the device's emotion recognition module analyzes the user's face and estimates their current emotion. When the user presses the "Upload" button, the image and emotion data are sent to the server.

[0245] Image analysis

[0246] The server adds the received image data to an analysis queue and invokes the image analysis module. Initial preprocessing includes image resizing, noise reduction, and color adjustment. The server then analyzes the image using a machine learning model to identify and label each food item. The quantity of each food item is also estimated.

[0247] Emotional Data Processing

[0248] The server analyzes the received emotional data to determine the user's emotional state, which is used as input data for dynamically adjusting the user's meal suggestions.

[0249] Acquiring food data

[0250] The server queries the food database based on the label of each identified food item, and retrieves detailed nutritional information (e.g., calories, protein, fat, carbohydrates, vitamins, minerals) for each food item in the database.

[0251] Nutritional balance chart

[0252] The server compares the nutritional information it obtains with the user's recommended daily intake and generates a nutritional balance chart in a visually easy-to-understand format, such as a radar chart or bar graph.

[0253] Displaying the results

[0254] The server sends the generated chart data in JSON format to the device. The device displays the chart to the user using a drawing library. The user can check the chart and understand their own nutritional balance.

[0255] Customized Offers

[0256] Users input and update their health goals and chronic illness information within the app. The device then sends this information and emotional data to the server. The server then updates the user profile based on the user's health goals, chronic illness information, and emotional state, and generates appropriate meal suggestions. These suggestions include recipes and menus for dining out.

[0257] Food suggestions according to your budget

[0258] The user enters their desired budget amount using a budget input form. The device sends the budget information to the server. The server queries convenience store and supermarket product databases to generate a list of foods that can be purchased within the budget. The server filters the list based on budget, nutritional balance, and emotional state, and sends the selected list to the device. The device displays this list to the user.

[0259] Collaboration with medical institutions

[0260] The user enables the "Link with Medical Institutions" option in the app settings. With the user's permission, the device sends nutritional balance data and emotional data to the server. The server takes appropriate security measures and shares the user data through an API dedicated to medical institutions. Medical institutions use this data to provide the user with appropriate feedback and treatment plans.

[0261] Specific examples

[0262] A user eats a hamburger steak set meal (rice, hamburger steak, salad) for lunch and uploads a photo of it to the app. The device sends the captured image to the server, which analyzes the image and identifies each food item. The server obtains nutritional information based on the identification results and creates a chart of the nutritional balance. The chart is then sent to the device and the user can visually check it.

[0263] Furthermore, if a user has a goal of losing weight and inputs that goal into the app, if the emotion engine detects that the user is feeling low, the server will take that into account and suggest a high-protein, low-calorie dinner menu (e.g., a recipe for salad and grilled chicken) to help them regain their energy.

[0264] If a user sets a budget of 1,000 yen for the next day and wants to buy a healthy lunch, the server will suggest convenience store items (e.g., sandwiches, salads, tea) that can be purchased within that budget. The user can purchase ingredients based on this list and maintain a healthy diet.

[0265] Users can also share their nutritional balance information and emotional data with medical institutions and receive regular health management advice from doctors. This system allows users to efficiently manage their daily diet and receive appropriate support tailored to their individual health and emotional states.

[0266] The processing flow will be explained below.

[0267] Step 1:

[0268] The user opens the application and takes a photo of the food. After taking the photo, the device's emotion engine analyzes the user's face in real time and estimates their current emotional state. The user then presses the "Upload" button.

[0269] Step 2:

[0270] The terminal stores the captured image file and emotion data in a temporary storage area within the application, and transmits the data to the server.

[0271] Step 3:

[0272] The server adds the received image data to the analysis queue and invokes the image analysis module. As initial preprocessing, it performs image resizing, noise removal, color adjustment, etc.

[0273] Step 4:

[0274] The server uses machine learning models to analyze the image, identify food item regions, label each item, and estimate the quantity of each food item.

[0275] Step 5:

[0276] The server queries the food database based on the label of each recognized food item, and obtains detailed nutritional information for each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.).

[0277] Step 6:

[0278] The server compares the collected nutritional information with the user's recommended daily intake and generates a nutritional balance chart in a visually easy-to-understand format such as a radar chart or bar graph.

[0279] Step 7:

[0280] The server analyzes the emotional data to determine the user's emotional state, which is used as input data for dynamically adjusting the user's meal suggestions.

[0281] Step 8:

[0282] The server sends the generated chart data and the user's emotional state in JSON format to the terminal.

[0283] Step 9:

[0284] The device uses a drawing library to display the chart data and emotion data received to the user, who can then check the chart to understand their own nutritional balance and emotional state.

[0285] Step 10:

[0286] Users enter and update their health goals and chronic illness information within the app, and the device sends this information and emotional data to the server.

[0287] Step 11:

[0288] The server updates the user's profile based on the user's health goals, chronic illness information, and emotional data, and generates suitable meal suggestions, including recipes and menus for dining out.

[0289] Step 12:

[0290] The server sends meal suggestions to the terminal.

[0291] Step 13:

[0292] The device displays suggested recipes and meal menus to the user, who then uses them to select a meal.

[0293] Step 14:

[0294] The user inputs the budget for purchasing ingredients, for example, "1000 yen."

[0295] Step 15:

[0296] The terminal transmits the budget information to the server.

[0297] Step 16:

[0298] The server queries convenience store and supermarket product databases to generate a list of food items that can be purchased within a budget. The food list is filtered based on nutritional balance and emotional state.

[0299] Step 17:

[0300] The server transmits the generated food list to the terminal.

[0301] Step 18:

[0302] The terminal displays a food list to the user, and the user uses the list as a reference when purchasing ingredients.

[0303] Step 19:

[0304] The user enables the "Connect with healthcare providers" option in the app settings.

[0305] Step 20:

[0306] The terminal confirms the user's permission and transmits the permission information along with the nutritional balance data and emotion data to the server.

[0307] Step 21:

[0308] The server takes appropriate security measures and shares user data through an API dedicated to medical institutions.

[0309] Step 22:

[0310] Medical institutions receive the shared data and provide appropriate feedback and treatment plans to users.

[0311] Example 2

[0312] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0313] Conventional dietary management systems make it difficult for users to understand the nutritional balance of foods and receive dietary suggestions tailored to their health goals. They also lack the ability to provide dietary suggestions based on emotional state or budget, and lack the ability to connect with medical institutions, making comprehensive health management difficult.

[0314] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: [means for an emotion recognition module of the terminal to analyze the user's face and estimate the emotion;] [means for the server to add received images to an analysis queue, and for the image analysis module to preprocess the images and identify foods using a machine learning model;] [means for the server to obtain nutritional information of the identified foods from a database and chart their nutritional balance;] [means for the server to select and provide foods according to a budget entered by the user; and] [means for linking nutritional balance information and emotion data with a medical institution. This allows the user's food selection to be optimized according to their emotional state and budget, and also enables health management in collaboration with a medical institution.

[0315] A "user" is someone who uses the system to take and upload images of food and receive nutritional balance and meal suggestions.

[0316] "Terminal" refers to a device operated by a user, such as a smartphone or tablet, and includes an emotion recognition module and a chart drawing library.

[0317] The "server" is the central computer of the system, and is a device that performs processes such as image analysis, database queries, meal suggestion generation, and medical institution collaboration.

[0318] An "emotion recognition module" is software or hardware built into a device that has the function of analyzing a user's face and estimating their emotional state.

[0319] "Image Analysis Module" means software within the server that pre-processes received images and uses machine learning models to identify food products.

[0320] A "machine learning model" is an algorithm that makes predictions and classifications based on data, and is a type of artificial intelligence used in image analysis modules.

[0321] A "food database" is a data store containing detailed nutritional information for various foods, to which a server can send queries to retrieve the required data.

[0322] A "nutritional balance chart" is a diagram that visually shows the user's nutritional intake status, and is displayed in the form of a radar chart or bar graph.

[0323] A "visualization library" is a software library for drawing charts and graphs, used on a device to visually display data in JSON format.

[0324] "Health goals" are goals that users set within the app based on their individual health status and lifestyle goals, and are taken into consideration when making meal suggestions.

[0325] The "budget" is a price range set by the user when purchasing ingredients, and is a guideline used to generate the food list provided by the server.

[0326] "Medical institution" refers to a hospital, clinic, medical office, etc., and is a facility that can share users' nutritional balance information and emotion data by linking with a server.

[0327] "Meal suggestions" refers to suggestions such as recipes and dining out menus generated by the server based on the user's health goals, emotional state, and budget.

[0328] The present invention is a system that allows users to take pictures of food, charts the nutritional balance based on the pictures, and suggests meals based on individual health goals and emotional state. Specific methods for implementing this system are described below.

[0329] The system includes devices such as smartphones and tablets, a server, a food database, and related software modules, including emotion recognition modules, image analysis modules, machine learning models (e.g., TensorFlow, PyTorch), and visualization libraries (e.g., Chart.js, D3.js).

[0330] First, the user opens the application and takes a picture of the food using the device's camera. An emotion recognition module (e.g., Face++ API) on the device analyzes the user's face and estimates their current emotion. This emotion data is later used to customize meal suggestions. After taking the picture, the user presses the "Upload" button, and the image and emotion data are sent to the server.

[0331] The server then adds the received image data to an analysis queue, where the image analysis module operates. Image preprocessing involves resizing, noise reduction, and color adjustment. Furthermore, a machine learning model (e.g., a trained CNN model) is used to analyze the image, identify food items, and label each one. The quantity of each food item is also estimated.

[0332] The server queries a food database (e.g., USDA Food Database) via its API to obtain detailed nutritional information (e.g., calories, protein, fat, carbohydrates, vitamins, minerals) for each identified food item. The server then generates a nutritional balance chart based on the nutritional information obtained and compares it with the user's recommended daily intake. This chart is created in a visually easy-to-understand format, such as a radar chart or bar graph, and sent to the device in JSON format.

[0333] The device uses a chart drawing library (e.g., Chart.js, D3.js) to display a nutritional balance chart to the user, allowing the user to visually understand their own nutritional balance.

[0334] Next, the user enters their health goals and chronic illness information into the app, and the data is sent from the device to the server. The server uses this information to update the user profile and generate meal suggestions that take into account the user's emotional data and health goals. For example, high-protein, low-calorie recipes or restaurant menus may be suggested.

[0335] Furthermore, when the user enters their desired budget amount using a form to input their budget, the device sends the budget information to the server. The server queries the product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within the budget. The food list is filtered based on the budget, nutritional balance, and emotional state and sent to the device and displayed to the user.

[0336] Finally, if the user enables the "Connect with Medical Institutions" option in the app settings, the device will send nutritional balance data and emotional data to the server with the user's permission. The server will then share this data with medical institutions through APIs after implementing security measures, enabling feedback and treatment plans from medical institutions.

[0337] For example, a user eats a hamburger steak set meal (rice, hamburger steak, and salad) for lunch and uploads a photo of it to the app. The device sends the captured image and emotional data to the server, which analyzes the image and identifies each food item. The server then obtains nutritional information, creates a nutritional balance chart, and sends it to the device. The user visually checks the chart and then sets a diet goal, which suggests a high-protein, low-calorie dinner menu based on their emotional state.

[0338] An example prompt using a generative AI model is as follows:

[0339] Describe a system that evaluates nutritional balance based on food images and emotion data taken by the user with a smartphone, and then suggests customized meals based on health goals. The following steps are performed in order: 1. Uploading images and emotion data. 2. Food identification using image analysis. 3. Obtaining nutritional information. 4. Charting nutritional balance. 5. Displaying the results to the user. 6. Generating customized meal suggestions. 7. Suggesting ingredients based on budget. 8. Data integration with medical institutions.

[0340] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0341] Step 1:

[0342] The user opens the application. The user takes a picture of food using the smartphone camera. The food image is obtained as input. The emotion recognition module in the device analyzes the user's face and estimates their current emotion. This results in emotion data. When the user presses the "upload" button, the image and emotion data are sent to the server. The output is the image data and emotion data sent to the server.

[0343] Step 2:

[0344] The server adds the received image data to the analysis queue. The input is the received image data. The server's image analysis module performs initial preprocessing of the image, resizing, denoising, and color adjustment. The server then analyzes the image using a machine learning model (e.g., TensorFlow or PyTorch). Food items are identified and labeled. The quantity of each food item is also estimated. The output is data on the identified food items and their quantities.

[0345] Step 3:

[0346] The server analyzes the received emotional data. The input is emotional data. The server determines the user's emotional state and uses this data to adjust the meal suggestions. The output is the analyzed emotional state data.

[0347] Step 4:

[0348] The server queries a food database (e.g., USDA food database) based on the label of each identified food item. As input, it has the food item label. It retrieves detailed nutritional information (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) for each food item in the database. The output is the retrieved nutritional information.

[0349] Step 5:

[0350] The server generates a nutritional balance chart by comparing the acquired nutritional information with the user's recommended daily intake. The inputs are the acquired nutritional information and the user's recommended daily intake. The chart is created in a visually easy-to-understand format such as a radar chart or bar graph. The output is the generated nutritional balance chart data.

[0351] Step 6:

[0352] The server sends the generated chart data in JSON format to the terminal. The input is the generated chart data. The terminal displays the chart to the user using a drawing library (e.g., Chart.js or D3.js). The output is a nutritional balance chart displayed to the user.

[0353] Step 7:

[0354] The user inputs and updates health goals and chronic illness information within the app. The input includes health goals and chronic illness information. The device sends this information and emotional data to the server. The server updates the user profile based on the user's health goals, chronic illness information, and emotional state, and generates appropriate meal suggestions. The output is the generated customized meal suggestions.

[0355] Step 8:

[0356] The user uses a budget input form to input the desired budget amount. The input is the budget amount. The device sends the budget information to the server. The server queries the product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within the budget. The server filters based on the budget, nutritional balance, and emotional state, and sends the selected food list to the device. The output is a list of foods within the budget displayed to the user.

[0357] Step 9:

[0358] The user enables the "Link with Medical Institutions" option in the app settings. The inputs include the user's desire to link, nutritional balance data, and emotional data. The device sends the nutritional balance data and emotional data to the server along with the user's permission. The server takes appropriate security measures and shares the user data through an API dedicated to medical institutions. The output is the user data shared with the medical institution.

[0359] (Application example 2)

[0360] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0361] Today's consumers are health-conscious and tend to seek nutritionally balanced meals. However, it is difficult to understand what nutrients are contained in everyday meals. It is also complicated to make appropriate meal suggestions based on the user's health goals and emotional state, or to make purchasing suggestions based on their budget. Furthermore, while managing daily eating habits and collaborating with medical institutions is becoming increasingly important, there is a problem in that a seamless method for doing so has not yet been established.

[0362] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: [means for the user to take and upload images of foods;] [means for the server to analyze the received images and identify the foods; and [means for the server to obtain nutritional information for the identified foods and chart their nutritional balance.] This makes it possible [for the user to easily understand the nutritional balance of their daily meals and receive meal suggestions suited to their health goals and emotional state].

[0363] "Means for users to take and upload images of food" refers to a function that allows users to take photos of food using a device such as a smartphone or camera and send the image data to a server.

[0364] The "means for analyzing the image received by the server and identifying the food item" refers to the function of processing the image data received by the server and identifying the food item in the image. This analysis uses machine learning and image recognition algorithms.

[0365] "Means for the server to obtain nutritional information for identified foods and chart the nutritional balance" refers to a function that obtains nutritional data corresponding to identified food items from a database and visually represents it in a format that is easy for users to understand.

[0366] "Means for displaying a nutritional balance chart to a user via a terminal" refers to a function for showing a nutritional balance chart to a user on a device such as a smartphone or computer.

[0367] "A means for users to input information about chronic illnesses and health goals, and for the server to make individually appropriate meal suggestions" is a function in which users input their own health condition and goals, and the server then suggests appropriate meal plans based on that information.

[0368] "Means for recognizing the user's emotional state and dynamically adjusting meal suggestions" is a function that grasps the user's emotional state from their facial expressions and behavior, and changes the meal suggestions based on the results.

[0369] The "means to support budget-based food purchases on online shopping sites" is a function that generates a list of foods that can be purchased based on the budget set by the user and helps the user purchase food based on that information.

[0370] "Means for linking nutritional balance information with medical institutions" is a function that allows users to share their nutritional data with medical institutions, enabling them to receive appropriate treatment and advice.

[0371] To implement this invention, the following system configuration and processing procedures are required. The system allows users to take and upload images of food, analyzes the nutritional balance based on the images, and makes dietary suggestions based on individual health goals and emotional state. It also supports food purchasing according to budget and connects with medical institutions for nutritional information. The main hardware and software used to realize this system are described below.

[0372] Hardware and software used

[0373] Hardware:

[0374] Smartphone (iOS or Android)

[0375] Server (general cloud services, e.g. AWS EC2)

[0376] software:

[0377] Smartphone app (Swift for iOS, Kotlin for Android)

[0378] Machine learning libraries (TensorFlow, PyTorch)

[0379] Nutrition Database (public food database)

[0380] Image processing library (OpenCV)

[0381] Emotion Recognition Library (general-purpose emotion recognition API)

[0382] Processing overview and examples

[0383] Image upload and emotion recognition

[0384] The user opens the smartphone app and takes a photo of the food. After the photo is taken, the smartphone's built-in emotion recognition module analyzes the user's face and estimates their current emotional state. When the user presses the "upload" button, the image and emotional data are sent to the server.

[0385] Image analysis and food identification

[0386] The server adds the received image data to the analysis queue and invokes the image analysis module. As an initial preprocessing step, the OpenCV library is used to resize, denoise, and adjust the color of the image. Then, machine learning libraries (TensorFlow and PyTorch) are used to analyze the image and identify food items. These identified food items are labeled and their quantities are estimated.

[0387] Obtaining nutritional data and charting balance

[0388] The server queries a nutrition database based on the identified food item labels to obtain detailed nutritional information for each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) and generates a nutritional balance chart based on this information, comparing it with the user's recommended daily intake. The chart can be in the form of a radar chart, bar graph, or other format.

[0389] Displaying the results

[0390] The server sends the generated chart data in JSON format to the smartphone. The smartphone app uses a visualization library to draw the chart and display it visually to the user. The user can then check the chart to understand their own nutritional balance.

[0391] Customized meal suggestions

[0392] Users enter their health goals and chronic illness information in the app and send it to the server, which then updates the user profile based on this information and emotional data and generates personalized meal suggestions, including recipes and menus for dining out.

[0393] Food suggestions according to your budget

[0394] The user enters their desired budget amount using a budget input form. The smartphone app sends the budget information to the server. The server queries product databases from convenience stores and supermarkets to generate a list of foods that can be purchased within the budget. The server filters the food items based on budget, nutritional balance, and emotional state, and sends the selected list to the smartphone app. The app displays this list to the user, allowing them to purchase the foods through an online shopping site.

[0395] Collaboration with medical institutions

[0396] The user enables the "Link with Medical Institutions" option in the app settings. The smartphone then sends nutritional balance data and emotional data to the server with the user's permission. The server then takes appropriate security measures and shares the user data through an API dedicated to medical institutions. Based on this data, the medical institution provides the user with appropriate feedback and treatment plans.

[0397] Specific examples

[0398] A user eats curry rice for lunch and uploads a photo of it to the app. The smartphone sends the captured image and emotional state to the server, which analyzes the image and identifies each food. Based on the results, nutritional information is obtained and a nutritional balance chart is created. The chart is then sent to the smartphone, where the user can visually check it.

[0399] Furthermore, if the user has a goal of dieting, they input that goal into the app. If the emotion engine detects that the user is under high stress, the server takes this into consideration and suggests foods that have a relaxing effect (e.g., a recipe for salad chicken and tofu). If the user sets a budget of 1,000 yen for the next day and wants to buy a healthy lunch, the server generates a list of foods that can be purchased within that budget and displays it on the smartphone app. This allows the user to purchase healthy foods on the online shopping site while staying within their budget.

[0400] Prompt Sentence Examples

[0401] "Identify the food in this image and provide its nutritional information."

[0402] "Based on this nutritional balance chart, please suggest foods that are suitable for my diet goals."

[0403] "Give me a list of foods that are suitable for me on my budget, taking into account my emotional state."

[0404] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0405] Step 1:

[0406] A user opens a smartphone app and takes a photo of food. The input is the image of the food taken with the camera, and the output is the image data saved on the user's smartphone. Specifically, the app activates the camera function, the user takes a photo of the food, and saves the image.

[0407] Step 2:

[0408] The device's built-in emotion recognition module analyzes the user's face and estimates their emotional state. The input is a user's facial image, and the output is estimated emotion data. Specifically, the app captures a facial image and uses the emotion recognition library to analyze emotions from the facial expression data.

[0409] Step 3:

[0410] When the user presses the "Upload" button, the image data and emotion data are sent to the server. The input is the image data and emotion data of the food, and the output is the data received by the server. Specifically, the app generates an HTTP request and sends the image data and emotion data to the server.

[0411] Step 4:

[0412] The server adds the received image data to the analysis queue and invokes the image analysis module. The input is the image data stored on the server, and the output is the image data ready for analysis. Specifically, the server adds the image data to the queue and schedules it for analysis in the next stage.

[0413] Step 5:

[0414] The server uses the OpenCV library to perform initial preprocessing such as image resizing, noise removal, and color adjustment. The input is image data ready for analysis, and the output is preprocessed image data. Specifically, it calls OpenCV functions to perform processing to improve the image quality.

[0415] Step 6:

[0416] The server uses a machine learning model (TensorFlow or PyTorch) to analyze the image and identify the food items. The input is preprocessed image data, and the output is the identified food items and their labels. Specifically, the model inputs the image and performs inference to obtain the identification results.

[0417] Step 7:

[0418] Based on the identification results, the server sends a query to a nutrition database to obtain detailed nutrition information for each food item. The input is the food item label, and the output is the nutrition information. The specific operation is to generate a database query to obtain the required nutrition data.

[0419] Step 8:

[0420] Based on the nutritional information acquired by the server, a nutritional balance chart is generated by comparing it with the recommended daily intake. The input is detailed nutritional information, and the output is the nutritional balance chart data. Specifically, the server runs an algorithm that compares the data and generates the chart.

[0421] Step 9:

[0422] The chart data generated by the server is sent to the terminal in JSON format. The input is the nutritional balance chart data, and the output is the chart data received by the terminal. Specifically, the chart data is sent as an HTTP response.

[0423] Step 10:

[0424] The terminal uses the visualization library to draw charts and display them visually to the user. The input is the received chart data, and the output is the chart displayed to the user. Specifically, it calls the library's drawing functions to display the chart on the screen.

[0425] Step 11:

[0426] The user enters health goals and chronic disease information in the app and sends it to the server. The input is the health goals and chronic disease information entered by the user, and the output is the health information stored on the server. Specifically, the app provides an input form and sends the input data to the server.

[0427] Step 12:

[0428] The server generates meal suggestions based on the user's health goals, chronic illness information, and emotional state, and sends them to the user. The input is user profile data and emotional data, and the output is customized meal suggestions. Specifically, the server runs a suggestion generation algorithm to suggest appropriate meal menus.

[0429] Step 13:

[0430] The user uses a budget input form to enter the desired budget amount and send it to the server. The input is the budget information entered by the user, and the output is the budget information saved on the server. Specifically, the app provides an input form and sends the budget data to the server.

[0431] Step 14:

[0432] The server queries the product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within a budget. The input is budget information and food identification data, and the output is a list of foods that can be purchased. Specifically, the server generates a database query and retrieves the appropriate food list.

[0433] Step 15:

[0434] The server filters the food items based on budget, nutritional balance, and emotional state, and sends the selected food list to the device. The input is the available food list and user profile data, and the output is the filtered food list. Specifically, the server runs the filtering algorithm to generate an appropriate list.

[0435] Step 16:

[0436] The terminal displays the filtered food list to the user and allows them to purchase the food through the online shopping site. The input is the filtered food list, and the output is the displayed list and a purchase link. The specific operation is to display the list on the screen and provide a purchase link.

[0437] Step 17:

[0438] The user enables the "Link with Medical Institutions" option in the app settings. The input is the user's setting selection, and the output is the setting change. Specifically, the user turns on the option on the app settings screen.

[0439] Step 18:

[0440] The terminal transmits the nutritional balance data and emotion data to the server with the user's permission. The input is the data with the user's permission, and the output is the nutritional balance data and emotion data stored on the server. Specifically, the terminal transmits the data to the server after confirming the user's permission.

[0441] Step 19:

[0442] The server takes appropriate security measures and then shares user data through an API dedicated to medical institutions. The input is the saved user data, and the output is the data sent to the medical institution. Specifically, data is shared through a secure API.

[0443] The above are the specific processing steps of the system that realizes the application example. Each step is important for realizing the function of assisting the user in managing their health.

[0444] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0445] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0446] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0447] [Second embodiment]

[0448] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0449] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0450] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0451] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0452] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0453] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0454] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0455] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0456] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0457] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0458] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0459] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0460] This invention is a system that allows users to take pictures of foods, chart their nutritional balance based on those pictures, and make dietary suggestions based on individual health goals. This system includes a means for users to take and upload food images, a means for a server to analyze the received images and identify the foods, a means for the server to obtain nutritional information for the identified foods and chart their nutritional balance, a means for displaying the nutritional balance chart to the user via a terminal, a means for the user to input information about chronic illnesses and health goals and for the server to make individually appropriate dietary suggestions, a means for supporting budget-based food purchasing at convenience stores and supermarkets, and a means for linking nutritional balance information with medical institutions.

[0461] Program processing

[0462] Image upload

[0463] A user opens the application and takes a photo of a food. The image is saved on the device and an upload button is displayed. When the user presses the upload button, the image is sent to the server.

[0464] Image analysis

[0465] The server adds the received image to the analysis queue. It invokes the image analysis module, which first preprocesses the image (e.g., resizes the image, removes noise, etc.). The server then uses a machine learning model to recognize food in the image. Specifically, it segments the image region, identifies each food item, and labels each one. At this stage, the type and quantity of food are estimated.

[0466] Acquiring food data

[0467] The server queries the food database based on the label of each identified food, and retrieves detailed nutritional information (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) for each food in the database.

[0468] Nutritional balance chart

[0469] The server aggregates the acquired nutritional information and creates a chart of nutritional balance based on the generated information. The nutritional balance chart is displayed in a visually easy-to-understand format (radar chart, bar graph, etc.).

[0470] Displaying the results

[0471] The server sends the chart data in JSON format to the device, which then visually displays the chart to the user using a drawing library. The user can refer to this chart and understand their diet at a glance.

[0472] Customized Offers

[0473] Users enter their health goals and chronic illness information within the app. The device sends this information to the server, which uses this information to update the user's profile and provide personalized dietary suggestions. These suggestions are then sent to the device and displayed to the user.

[0474] Food suggestions according to your budget

[0475] The user enters their desired budget amount using a budget input form. The device sends the budget information to the server. The server queries the product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within the budget. The server filters the list based on budget and nutritional balance and sends the selected list to the device. The device displays this list to the user.

[0476] Collaboration with medical institutions

[0477] The user enables the "Link with medical institutions" option in the app settings. The device sends nutritional balance data to the server with the user's permission. The server takes security measures and shares the user data through an API dedicated to medical institutions. Medical institutions use this data to provide the user with appropriate feedback and treatment plans.

[0478] Specific examples

[0479] A user eats a hamburger steak set meal (rice, hamburger steak, salad) for lunch and uploads a photo of it to the app. The device sends the captured image to the server, which analyzes the image and identifies each food item. The server obtains nutritional information based on the identification results and creates a chart of nutritional balance. The chart is then sent to the device and the user can visually confirm it. Furthermore, if the user has a diet goal, they input that goal into the app. The server takes this into consideration and suggests a low-calorie dinner menu (e.g., a recipe for salad and grilled chicken). If a user sets a budget of 1,000 yen for the next day and wants to buy a healthy lunch, the server will suggest convenience store items (e.g., sandwiches, salad, and tea) that can be purchased within that budget. Users can maintain a healthy diet by referring to the ingredient list when making purchases. Users can also share their nutritional balance information with medical institutions and engage in regular health management.

[0480] In this way, the system allows users to easily and efficiently manage their daily diet and receive appropriate support according to their individual health conditions.

[0481] The processing flow will be explained below.

[0482] Step 1:

[0483] A user opens the application and takes a photo of the food. After taking the photo, the user presses the "upload" button.

[0484] Step 2:

[0485] The terminal stores the captured image file in a temporary storage area within the application, and transmits the image data to the server.

[0486] Step 3:

[0487] The server adds the received image data to the analysis queue and invokes the image analysis module. As initial preprocessing, it performs image resizing, noise removal, color adjustment, etc.

[0488] Step 4:

[0489] The server uses machine learning models to analyze the image, identify food item regions, label each item, and estimate the quantity of each food item.

[0490] Step 5:

[0491] The server queries the food database based on the label of each recognized food item, and obtains detailed nutritional information for each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.).

[0492] Step 6:

[0493] The server compares the collected nutritional information with the user's recommended daily intake and generates a nutritional balance chart in a visually easy-to-understand format such as a radar chart or bar graph.

[0494] Step 7:

[0495] The server sends the generated chart data in JSON format to the terminal.

[0496] Step 8:

[0497] The device uses a drawing library to display the chart data received to the user, who can then check the chart to understand their own nutritional balance.

[0498] Step 9:

[0499] Users enter and update their health goals and chronic illness information within the app.

[0500] Step 10:

[0501] The terminal transmits the input information to the server.

[0502] Step 11:

[0503] The server updates the user's profile based on their health goals and chronic illnesses, and generates suitable meal suggestions, including recipes and menus for dining out.

[0504] Step 12:

[0505] The server sends meal suggestions to the terminal.

[0506] Step 13:

[0507] The device displays suggested recipes and meal menus to the user, who can then use them to select a meal.

[0508] Step 14:

[0509] The user inputs the budget for purchasing ingredients, for example, "1000 yen."

[0510] Step 15:

[0511] The terminal transmits the budget information to the server.

[0512] Step 16:

[0513] The server queries product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within a budget. The list is filtered to take nutritional balance into consideration.

[0514] Step 17:

[0515] The server transmits the generated food list to the terminal.

[0516] Step 18:

[0517] The terminal displays a food list to the user, and the user uses the list as a reference when purchasing ingredients.

[0518] Step 19:

[0519] The user enables the "Connect with healthcare providers" option in the app settings.

[0520] Step 20:

[0521] The terminal confirms the user's permission and transmits the permission information along with the nutritional balance data to the server.

[0522] Step 21:

[0523] The server takes appropriate security measures and shares user data through an API dedicated to medical institutions.

[0524] Step 22:

[0525] Medical institutions receive the shared data and provide appropriate feedback and treatment plans to users.

[0526] Example 1

[0527] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0528] Currently, there are many dietary management tools and applications on the market, but most of them have limitations in their ability to effectively understand the user's dietary content and nutritional balance and make personalized suggestions based on that information. They also lack the functionality to suggest ingredients based on the user's budget or to support health management in collaboration with medical institutions. There is a need for a system that can improve these points and enable users to manage their health more effectively.

[0529] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0530] In this invention, the server includes means for preprocessing the received images and identifying foods using a machine learning model, means for acquiring nutritional information for the identified foods and charting their nutritional balance, and means for linking the nutritional balance information with medical institutions. This enables users to easily manage their diet through food images, receive personalized dietary suggestions, and even manage their health consistently through collaboration with medical institutions.

[0531] "User" refers to an individual who uses the system to take pictures of food and manage their diet.

[0532] "Server" refers to a central processing unit for receiving and analyzing data sent by users and providing necessary information.

[0533] "Terminal" refers to a computing device (such as a smartphone, tablet, or PC) used by a user.

[0534] "Food image" refers to photographic data containing food that a user takes and uploads to the system.

[0535] "Preprocessing" refers to the initial processing such as resizing and noise removal that is performed before the server analyzes the image.

[0536] A "machine learning model" refers to an algorithm that learns patterns from data and identifies and classifies foods.

[0537] A "food database" refers to a collection of data that stores nutritional information about each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.).

[0538] "Charting nutritional balance" refers to presenting collected nutritional information in a visually easy-to-understand format (such as a radar chart or bar graph).

[0539] "Health goal" refers to a goal related to a user's health status (e.g., dieting, muscle building, managing a chronic illness, etc.).

[0540] "Meal suggestions" refers to recommendations for providing an individualized meal plan based on the user's health goals and chronic illness information.

[0541] "Support for food purchasing" refers to providing users with a list of foods that can be purchased within their budget, and assisting them in their purchasing activities at convenience stores and supermarkets.

[0542] "Collaboration with medical institutions" refers to a system in which users' nutritional balance information is shared with medical institutions and feedback and treatment plans are provided as needed.

[0543] "Drawing library" refers to a software component for visually displaying data (e.g., D3.js, Chart.js, Matplotlib, etc.).

[0544] The present invention is a system that allows a user to take images of food, chart the nutritional balance based on the images, and make dietary suggestions based on individual health goals. This system includes means for the user to take and upload images of food, means for a server to analyze the received images and identify the food, means for the server to obtain nutritional information for the identified foods and chart the nutritional balance, means for displaying the nutritional balance chart to the user via a terminal, means for the user to input health goals and chronic illness information and for the server to make individually appropriate dietary suggestions, means for supporting food purchasing according to budget, means for linking nutritional balance information with medical institutions, and means for pre-processing the captured images by resizing and removing noise.

[0545] Hardware and software used

[0546] To realize this system, the following hardware and software are used.

[0547] 1. Hardware

[0548] User device: smartphone, tablet, or computer.

[0549] Server: Server equipment equipped with a high-performance CPU and GPU.

[0550] 2. Software

[0551] Image analysis module: Uses Python and utilizes libraries such as OpenCV and TensorFlow.

[0552] Database: A relational database such as MySQL or PostgreSQL.

[0553] Drawing libraries: Visualization libraries such as D3.js, Chart.js, Matplotlib, etc.

[0554] API: Implements a RESTful API to communicate between the server and the terminal.

[0555] Example of operation

[0556] A specific example of the operation of the system of the present invention will be described below.

[0557] Specific examples

[0558] A user eats a hamburger steak set meal (rice, hamburger steak, salad) for lunch and uploads a photo of it to the app. The user takes a photo of the food using the app's camera function and presses the upload button, which sends the image to the server. The server preprocesses the received image (for example, resizes the image to 256x256 pixels and removes noise) and begins analysis using a machine learning model using TensorFlow. The server analyzes the image, identifies the foods - hamburger steak, rice, and salad - and labels each food.

[0559] The server then queries a MySQL database based on the identified food labels to obtain detailed nutritional information for each food (e.g., calories, protein, fat, carbohydrates, vitamins, and minerals). The obtained nutritional information is aggregated and a nutritional balance chart is created. The server sends this chart information in JSON format to the device, which then visually displays the chart using a drawing library such as D3.js. Users can refer to the chart to understand their diet at a glance.

[0560] Furthermore, when a user sets "diet" as a "health goal" in the app, the server will suggest corresponding low-calorie menus (e.g., salad and grilled chicken recipes). If a user sets a budget of 1,000 yen and wants to buy a healthy lunch, the server will generate a list of foods that can be purchased within that budget (e.g., sandwiches, salads, and tea) and send it to the device. By referring to the list of ingredients when making purchases, the user can maintain a healthy diet.

[0561] Prompt Sentence Examples

[0562] Example inputs to a generative AI model:

[0563] "Write a program that analyzes an image of a person eating oatmeal, banana, and yogurt for breakfast, along with the detailed ingredients, and then charts and displays the nutritional balance of this meal."

[0564] This system allows users to easily and efficiently manage their daily diet and receive appropriate support according to their individual health conditions.

[0565] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0566] Step 1:

[0567] The user takes a picture of a food item and presses the upload button in the application. The device saves the image to its internal storage and sends an upload request to the server. The input data is the food image taken by the user, and the output is the image sent to the server.

[0568] Specific behavior:

[0569] A user takes a photo of a hamburger steak set meal using the camera on their smartphone.

[0570] The device saves the image file (e.g., lunch.jpg) to its internal storage.

[0571] The user taps the upload button in the app and the image is sent to the server.

[0572] Step 2:

[0573] The server receives the uploaded images and adds them to the analysis queue. The server uses image analysis modules to preprocess the images and convert them into an analyzable format. The input data is the original uploaded image, and the output is the preprocessed image data.

[0574] Specific behavior:

[0575] The server resizes the image to 256x256 pixels and denoises it.

[0576] The preprocessed image data is fed into a machine learning model.

[0577] Step 3:

[0578] The server uses a machine learning model (e.g., TensorFlow) to analyze the preprocessed images and identify the foods. It identifies each food item in the image and assigns a label to each. The input data is the preprocessed image data, and the output is the label and estimated quantity of the identified food.

[0579] Specific behavior:

[0580] The server uses the YOLO model to detect food items (e.g., hamburger, rice, salad) in the image.

[0581] The server labels each food item and estimates the quantity (e.g., "hamburger": 1 piece, "rice": 1 bowl, "salad": 1 plate).

[0582] Step 4:

[0583] The server queries the food database based on the label of each identified food and retrieves detailed nutritional information for each food (calories, protein, fat, carbohydrates, vitamins, minerals, etc.). The input data is the label of the identified food, and the output is the retrieved nutritional information.

[0584] Specific behavior:

[0585] The server uses a SELECT query to retrieve nutritional information from the MySQL database based on the label "hamburger."

[0586] Similarly, obtain nutritional information for each food item (e.g., "Calories": 350kcal, "Protein": 25g).

[0587] Step 5:

[0588] The server aggregates the acquired nutritional information and generates chart data to visually represent nutritional balance. The input data is detailed nutritional information for each food, and the output is chart data.

[0589] Specific behavior:

[0590] A server aggregates the nutritional information of all foods.

[0591] The server uses the Matplotlib library to plot the nutritional balance as a radar chart.

[0592] Step 6:

[0593] The server generates chart data and sends it to the terminal in JSON format. The terminal uses a drawing library (e.g., D3.js) to display the chart to the user based on the data received. The input data is the chart data, and the output is a visual chart that is displayed to the user.

[0594] Specific behavior:

[0595] The server converts the chart data into JSON format and sends it to the terminal.

[0596] The device uses Chart.js to draw the nutritional balance in the form of a bar graph and displays it on the app's UI.

[0597] Step 7:

[0598] The user enters their health goals and chronic illness information into the app. The device sends this information to the server, which then makes personalized dietary suggestions. The input data is the user's health goals and chronic illness information, and the output is dietary suggestions.

[0599] Specific behavior:

[0600] The user enters "health goal" as "diet."

[0601] The server will then suggest a low-calorie option (e.g., salad and grilled chicken) based on that information.

[0602] Step 8:

[0603] The user uses a budget input form to input the desired budget amount. The device sends the budget information to the server, which then executes a query to generate a list of foods that can be purchased within the budget. The input data is the user's budget information, and the output is a list of foods that can be purchased within the budget.

[0604] Specific behavior:

[0605] The user sets the "budget" to "1,000 yen" within the app.

[0606] The server generates a list of foods that can be purchased for under 1,000 yen, suggesting sandwiches, salads, and tea.

[0607] Step 9:

[0608] The user enables the "Link with medical institutions" option in the app settings. Based on the user's permission, the device sends nutritional balance data to the server, and the server shares the user data through an API dedicated to medical institutions. The input data is the user's nutritional balance data and medical institution information, and the output is data shared with the medical institution.

[0609] Specific behavior:

[0610] The user enables the "Connect with Healthcare Providers" option.

[0611] The server encrypts the data and sends it to an HTTPS endpoint dedicated to the medical institution.

[0612] (Application example 1)

[0613] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0614] In modern society, there is a demand for appropriate nutritional management and dietary suggestions tailored to each individual's health condition. However, daily dietary management is complicated, and it is particularly difficult to confirm the nutritional balance of food when eating out or purchasing it. Furthermore, when purchasing at convenience stores or supermarkets, it is not easy to select ingredients that take nutritional balance and budget into consideration. Furthermore, there is a demand for more appropriate health management by linking nutritional information with medical institutions, but current systems are not able to adequately address this demand.

[0615] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0616] In this invention, the server includes means for a user to photograph and upload images of foods, means for the server to analyze the received images and identify the foods, means for the server to obtain nutritional information for the identified foods and create a nutritional balance chart, means for the server to display the nutritional balance chart to the user via a terminal, means for the user to input chronic illness information and health goals and for the server to make individually appropriate meal suggestions, means for supporting budget-based food purchasing at sales facilities, means for linking the nutritional balance information with medical institutions, means for analyzing photographed images of food shelves and obtaining nutritional information for each food item to display to the user, and means for suggesting foods that can be purchased within the user's budget based on the user's input data. This allows users to check nutritional information on the spot when purchasing food at a physical store, make healthy food choices within their budget, and even manage their health in cooperation with medical institutions.

[0617] "User" refers to an individual who uses the system to take pictures of food and receive nutritional balance and meal suggestions.

[0618] "Server" refers to a central computer system for analyzing food images, obtaining nutritional information, and displaying it to the user.

[0619] The "means for taking and uploading images of food" refers to the part that provides the function for users to take images of food using a smartphone or smart glasses and send those images to the system.

[0620] "Means for identifying food" refers to technology that analyzes the image received by the server and identifies the food in the image.

[0621] "Means for obtaining nutritional information and charting nutritional balance" refers to the function of collecting nutritional data for identified foods and displaying it in a visually easy-to-understand format.

[0622] "Means for displaying a nutritional balance chart to a user on a terminal" refers to a function that allows a user to see a nutritional balance chart on a terminal such as a smartphone or computer.

[0623] "A means for inputting health goals and for the server to make individually appropriate meal suggestions" refers to a function in which the user inputs information about chronic illnesses and health goals, and the server creates customized meal suggestions based on that information.

[0624] "Means to support food purchasing according to budget at sales facilities" refers to a function that suggests foods that can be purchased taking into account the user's budget and supports the purchase.

[0625] "Means for linking nutritional balance information with medical institutions" refers to a function for sharing a user's nutritional information with medical institutions and supporting health management.

[0626] "Means for analyzing images of food shelves, obtaining nutritional information for each food item, and displaying this information to the user" refers to a function that analyzes images of food shelves taken at a sales facility, obtains nutritional information for each food item, and provides this information to the user.

[0627] "Means for suggesting foods that can be purchased within a budget based on input data" refers to a function that creates and suggests a list of foods that can be purchased within a budget based on the user's input data (budget and health goals).

[0628] This invention is a system that allows a user to take an image of food, charts the nutritional balance based on the image, and makes meal suggestions based on individual health goals. The system includes the following means:

[0629] 1. Image uploading method: Users take pictures of the food shelves using their smartphones or smart glasses and upload them to the server from their devices. The uploaded images are stored on the server.

[0630] 2. Image analysis: The server analyzes the received images and identifies the food in the image. This image analysis is performed using a machine learning model (e.g., using TensorFlow). The server first preprocesses the images (resizes, removes noise, etc.), then identifies and labels the food.

[0631] 3. Nutritional information acquisition means: Based on the food identified from the analyzed image, the server acquires the detailed nutritional information of the food from a food database (e.g., an internally stored nutritional database), including information such as calories, protein, fat, carbohydrates, vitamins, and minerals.

[0632] 4. Nutritional balance charting: The server aggregates the nutritional balance based on the acquired nutritional information and displays it in a visually easy-to-understand format (for example, a radar chart or bar graph). This allows users to understand the nutritional balance of the foods they have consumed at a glance.

[0633] 5. Display of results: The charted nutritional balance is sent from the server to the device, which then uses a visualization library to render it and display it to the user, allowing the user to understand their own dietary habits.

[0634] 6. Customized Meal Suggestion: Users can input their health goals and chronic illness information within the application. This information is sent to the server, which updates the user's profile and provides personalized meal suggestions. These suggestions are sent to the device, providing the user with guidelines for maintaining a healthy diet.

[0635] 7. Budget-based food recommendation: The user inputs their desired budget amount into the application. The server connects to the database of sales facilities to generate a list of foods that can be purchased within the budget. It filters based on budget and nutritional balance and sends the selected food list to the terminal. The terminal displays this list to the user, who can receive assistance in purchasing healthy foods within their budget.

[0636] 8. Linking with medical institutions: When the user enables the "Linking with medical institutions" option, the device will send nutritional balance data to the server with the user's permission. The server will then share the user data through an API dedicated to medical institutions, after taking security measures. Medical institutions will use this data to provide the user with appropriate feedback and treatment plans.

[0637] For example, when a user takes a photo of a food shelf in a physical store and uploads it, the server analyzes the image, obtains nutritional information for each food item, and displays it to the user. Based on this information, the user can select foods that take into account nutritional balance and budget on the spot. Furthermore, by inputting the user's health goals and budget, the server can make optimal meal suggestions.

[0638] Additionally, examples of prompts that are useful as input to generative AI models include:

[0639] "A user takes a photo of a food shelf in a physical store and uploads it. Please explain how the system uses the image to create a nutritional balance chart and provide meal suggestions aligned with the user's health goals. Specifically, please explain in detail the process of image upload, image analysis, nutritional information acquisition, and meal suggestions. Please also include suggestions based on a budget."

[0640] In this way, the system allows users to easily and efficiently manage their daily diet and receive appropriate support according to their individual health conditions.

[0641] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0642] Step 1: Upload an image

[0643] A user takes a photo of the food shelf with a smartphone or smart glasses. The device saves the image and displays an upload button. When the user presses the upload button, the image is sent from the device to the server. The input is the image taken by the user, and the output is the image data sent to the server.

[0644] Step 2: Image analysis

[0645] The server adds the received image to the analysis queue. It invokes the image analysis module to first preprocess the image (resize, remove noise, etc.). The server then uses a generative AI model to recognize food in the image. This process involves segmenting the image area, identifying each food item, and labeling each one. The input is the image data sent to the server, and the output is the label information of the identified food and its location information.

[0646] Step 3: Obtain nutritional information

[0647] The server queries an internal food database based on the label of each identified food, and obtains detailed nutritional information (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) for each food in the database. The input is the label information of the identified food, and the output is the nutritional information.

[0648] Step 4: Nutritional Balance Chart

[0649] The server aggregates the acquired nutritional information and charts the nutritional balance in a visually easy-to-understand format (such as a radar chart or bar graph). The chart data is generated in JSON format. The input is the aggregated nutritional information, and the output is the JSON data of the nutritional balance chart.

[0650] Step 5: View the results

[0651] The server sends the chart data to the terminal, which uses a visualization library to draw the chart and display it visually to the user. The input is the JSON data of the chart, and the output is the nutritional balance chart displayed to the user.

[0652] Step 6: Customized Meal Suggestions

[0653] The user enters their health goals and chronic illness information within the application. The device sends this information to the server. The server updates the user's profile and generates personalized meal suggestions. The suggestions are sent to the device and displayed to the user. The input is the user's health goals and chronic illness information, and the output is personalized meal suggestions.

[0654] Step 7: Proposing ingredients according to your budget

[0655] The user enters their budget in the app. The device sends the budget information to the server. The server queries a database of sales establishments to generate a list of foods that can be purchased within the budget. After filtering, the server sends the selected list of ingredients to the device. The device displays this list to the user. The input is the user's budget information, and the output is a list of foods that the user can purchase.

[0656] Step 8: Collaborate with medical institutions

[0657] The user enables the "Link with medical institutions" option in the app settings. The device sends nutritional balance data to the server with the user's permission. The server takes security measures and shares the user data through an API dedicated to medical institutions. Based on this data, medical institutions provide the user with appropriate feedback and treatment plans. The input is the user's nutritional balance data, and the output is the data sent to the medical institution.

[0658] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0659] The present invention is a system that allows a user to take an image of food, charts the nutritional balance based on the image, and makes meal suggestions based on individual health goals and emotional state. This system includes means for a user to take and upload an image of food, means for a server to analyze the received image and identify the food, means for the server to obtain nutritional information for the identified food and chart the nutritional balance, means for displaying the nutritional balance chart to the user via a terminal, means for the user to input information about chronic illnesses and health goals and for the server to make individually appropriate meal suggestions, means for supporting budget-based food purchasing at convenience stores and supermarkets, means for linking nutritional balance information with medical institutions, and means for recognizing the user's emotions using an emotion engine.

[0660] Program processing

[0661] Image upload and emotion recognition

[0662] The user opens the application and takes a photo of the food. After taking the photo, the device's emotion recognition module analyzes the user's face and estimates their current emotion. When the user presses the "Upload" button, the image and emotion data are sent to the server.

[0663] Image analysis

[0664] The server adds the received image data to an analysis queue and invokes the image analysis module. Initial preprocessing includes image resizing, noise reduction, and color adjustment. The server then analyzes the image using a machine learning model to identify and label each food item. The quantity of each food item is also estimated.

[0665] Emotional Data Processing

[0666] The server analyzes the received emotional data to determine the user's emotional state, which is used as input data for dynamically adjusting the user's meal suggestions.

[0667] Acquiring food data

[0668] The server queries the food database based on the label of each identified food item, and retrieves detailed nutritional information (e.g., calories, protein, fat, carbohydrates, vitamins, minerals) for each food item in the database.

[0669] Nutritional balance chart

[0670] The server compares the nutritional information it obtains with the user's recommended daily intake and generates a nutritional balance chart in a visually easy-to-understand format, such as a radar chart or bar graph.

[0671] Displaying the results

[0672] The server sends the generated chart data in JSON format to the device. The device displays the chart to the user using a drawing library. The user can check the chart and understand their own nutritional balance.

[0673] Customized Offers

[0674] Users input and update their health goals and chronic illness information within the app. The device then sends this information and emotional data to the server. The server then updates the user profile based on the user's health goals, chronic illness information, and emotional state, and generates appropriate meal suggestions. These suggestions include recipes and menus for dining out.

[0675] Food suggestions according to your budget

[0676] The user enters their desired budget amount using a budget input form. The device sends the budget information to the server. The server queries convenience store and supermarket product databases to generate a list of foods that can be purchased within the budget. The server filters the list based on budget, nutritional balance, and emotional state, and sends the selected list to the device. The device displays this list to the user.

[0677] Collaboration with medical institutions

[0678] The user enables the "Link with Medical Institutions" option in the app settings. With the user's permission, the device sends nutritional balance data and emotional data to the server. The server takes appropriate security measures and shares the user data through an API dedicated to medical institutions. Medical institutions use this data to provide the user with appropriate feedback and treatment plans.

[0679] Specific examples

[0680] A user eats a hamburger steak set meal (rice, hamburger steak, salad) for lunch and uploads a photo of it to the app. The device sends the captured image to the server, which analyzes the image and identifies each food item. The server obtains nutritional information based on the identification results and creates a chart of the nutritional balance. The chart is then sent to the device and the user can visually check it.

[0681] Furthermore, if a user has a goal of losing weight and inputs that goal into the app, if the emotion engine detects that the user is feeling low, the server will take that into account and suggest a high-protein, low-calorie dinner menu (e.g., a recipe for salad and grilled chicken) to help them regain their energy.

[0682] If a user sets a budget of 1,000 yen for the next day and wants to buy a healthy lunch, the server will suggest convenience store items (e.g., sandwiches, salads, tea) that can be purchased within that budget. The user can purchase ingredients based on this list and maintain a healthy diet.

[0683] Users can also share their nutritional balance information and emotional data with medical institutions and receive regular health management advice from doctors. This system allows users to efficiently manage their daily diet and receive appropriate support tailored to their individual health and emotional states.

[0684] The processing flow will be explained below.

[0685] Step 1:

[0686] The user opens the application and takes a photo of the food. After taking the photo, the device's emotion engine analyzes the user's face in real time and estimates their current emotional state. The user then presses the "Upload" button.

[0687] Step 2:

[0688] The terminal stores the captured image file and emotion data in a temporary storage area within the application, and transmits the data to the server.

[0689] Step 3:

[0690] The server adds the received image data to the analysis queue and invokes the image analysis module. As initial preprocessing, it performs image resizing, noise removal, color adjustment, etc.

[0691] Step 4:

[0692] The server uses machine learning models to analyze the image, identify food item regions, label each item, and estimate the quantity of each food item.

[0693] Step 5:

[0694] The server queries the food database based on the label of each recognized food item, and obtains detailed nutritional information for each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.).

[0695] Step 6:

[0696] The server compares the collected nutritional information with the user's recommended daily intake and generates a nutritional balance chart in a visually easy-to-understand format such as a radar chart or bar graph.

[0697] Step 7:

[0698] The server analyzes the emotional data to determine the user's emotional state, which is used as input data for dynamically adjusting the user's meal suggestions.

[0699] Step 8:

[0700] The server sends the generated chart data and the user's emotional state in JSON format to the terminal.

[0701] Step 9:

[0702] The device uses a drawing library to display the chart data and emotion data received to the user, who can then check the chart to understand their own nutritional balance and emotional state.

[0703] Step 10:

[0704] Users enter and update their health goals and chronic illness information within the app, and the device sends this information and emotional data to the server.

[0705] Step 11:

[0706] The server updates the user's profile based on the user's health goals, chronic illness information, and emotional data, and generates suitable meal suggestions, including recipes and menus for dining out.

[0707] Step 12:

[0708] The server sends meal suggestions to the terminal.

[0709] Step 13:

[0710] The device displays suggested recipes and meal menus to the user, who then uses them to select a meal.

[0711] Step 14:

[0712] The user inputs the budget for purchasing ingredients, for example, "1000 yen."

[0713] Step 15:

[0714] The terminal transmits the budget information to the server.

[0715] Step 16:

[0716] The server queries convenience store and supermarket product databases to generate a list of food items that can be purchased within a budget. The food list is filtered based on nutritional balance and emotional state.

[0717] Step 17:

[0718] The server transmits the generated food list to the terminal.

[0719] Step 18:

[0720] The terminal displays a food list to the user, and the user uses the list as a reference when purchasing ingredients.

[0721] Step 19:

[0722] The user enables the "Connect with healthcare providers" option in the app settings.

[0723] Step 20:

[0724] The terminal confirms the user's permission and transmits the permission information along with the nutritional balance data and emotion data to the server.

[0725] Step 21:

[0726] The server takes appropriate security measures and shares user data through an API dedicated to medical institutions.

[0727] Step 22:

[0728] Medical institutions receive the shared data and provide appropriate feedback and treatment plans to users.

[0729] Example 2

[0730] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0731] Conventional dietary management systems make it difficult for users to understand the nutritional balance of foods and receive dietary suggestions tailored to their health goals. They also lack the ability to provide dietary suggestions based on emotional state or budget, and lack the ability to connect with medical institutions, making comprehensive health management difficult.

[0732] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: [means for an emotion recognition module of the terminal to analyze the user's face and estimate the emotion;] [means for the server to add received images to an analysis queue, and for the image analysis module to preprocess the images and identify foods using a machine learning model;] [means for the server to obtain nutritional information of the identified foods from a database and chart their nutritional balance;] [means for the server to select and provide foods according to a budget entered by the user; and] [means for linking nutritional balance information and emotion data with a medical institution. This allows the user's food selection to be optimized according to their emotional state and budget, and also enables health management in collaboration with a medical institution.

[0733] A "user" is someone who uses the system to take and upload images of food and receive nutritional balance and meal suggestions.

[0734] "Terminal" refers to a device operated by a user, such as a smartphone or tablet, and includes an emotion recognition module and a chart drawing library.

[0735] The "server" is the central computer of the system, and is a device that performs processes such as image analysis, database queries, meal suggestion generation, and medical institution collaboration.

[0736] An "emotion recognition module" is software or hardware built into a device that has the function of analyzing a user's face and estimating their emotional state.

[0737] "Image Analysis Module" means software within the server that pre-processes received images and uses machine learning models to identify food products.

[0738] A "machine learning model" is an algorithm that makes predictions and classifications based on data, and is a type of artificial intelligence used in image analysis modules.

[0739] A "food database" is a data store containing detailed nutritional information for various foods, to which a server can send queries to retrieve the required data.

[0740] A "nutritional balance chart" is a diagram that visually shows the user's nutritional intake status, and is displayed in the form of a radar chart or bar graph.

[0741] A "visualization library" is a software library for drawing charts and graphs, used on a device to visually display data in JSON format.

[0742] "Health goals" are goals that users set within the app based on their individual health status and lifestyle goals, and are taken into consideration when making meal suggestions.

[0743] The "budget" is a price range set by the user when purchasing ingredients, and is a guideline used to generate the food list provided by the server.

[0744] "Medical institution" refers to a hospital, clinic, medical office, etc., and is a facility that can share users' nutritional balance information and emotion data by linking with a server.

[0745] "Meal suggestions" refers to suggestions such as recipes and dining out menus generated by the server based on the user's health goals, emotional state, and budget.

[0746] The present invention is a system that allows users to take pictures of food, charts the nutritional balance based on the pictures, and suggests meals based on individual health goals and emotional state. Specific methods for implementing this system are described below.

[0747] The system includes devices such as smartphones and tablets, a server, a food database, and related software modules, including emotion recognition modules, image analysis modules, machine learning models (e.g., TensorFlow, PyTorch), and visualization libraries (e.g., Chart.js, D3.js).

[0748] First, the user opens the application and takes a picture of the food using the device's camera. An emotion recognition module (e.g., Face++ API) on the device analyzes the user's face and estimates their current emotion. This emotion data is later used to customize meal suggestions. After taking the picture, the user presses the "Upload" button, and the image and emotion data are sent to the server.

[0749] The server then adds the received image data to an analysis queue, where the image analysis module operates. Image preprocessing involves resizing, noise reduction, and color adjustment. Furthermore, a machine learning model (e.g., a trained CNN model) is used to analyze the image, identify food items, and label each one. The quantity of each food item is also estimated.

[0750] The server queries a food database (e.g., USDA Food Database) via its API to obtain detailed nutritional information (e.g., calories, protein, fat, carbohydrates, vitamins, minerals) for each identified food item. The server then generates a nutritional balance chart based on the nutritional information obtained and compares it with the user's recommended daily intake. This chart is created in a visually easy-to-understand format, such as a radar chart or bar graph, and sent to the device in JSON format.

[0751] The device uses a chart drawing library (e.g., Chart.js, D3.js) to display a nutritional balance chart to the user, allowing the user to visually understand their own nutritional balance.

[0752] Next, the user enters their health goals and chronic illness information into the app, and the data is sent from the device to the server. The server uses this information to update the user profile and generate meal suggestions that take into account the user's emotional data and health goals. For example, high-protein, low-calorie recipes or restaurant menus may be suggested.

[0753] Furthermore, when the user enters their desired budget amount using a form to input their budget, the device sends the budget information to the server. The server queries the product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within the budget. The food list is filtered based on the budget, nutritional balance, and emotional state and sent to the device and displayed to the user.

[0754] Finally, if the user enables the "Connect with Medical Institutions" option in the app settings, the device will send nutritional balance data and emotional data to the server with the user's permission. The server will then share this data with medical institutions through APIs after implementing security measures, enabling feedback and treatment plans from medical institutions.

[0755] For example, a user eats a hamburger steak set meal (rice, hamburger steak, and salad) for lunch and uploads a photo of it to the app. The device sends the captured image and emotional data to the server, which analyzes the image and identifies each food item. The server then obtains nutritional information, creates a nutritional balance chart, and sends it to the device. The user visually checks the chart and then sets a diet goal, which suggests a high-protein, low-calorie dinner menu based on their emotional state.

[0756] An example prompt using a generative AI model is as follows:

[0757] Describe a system that evaluates nutritional balance based on food images and emotion data taken by the user with a smartphone, and then suggests customized meals based on health goals. The following steps are performed in order: 1. Uploading images and emotion data. 2. Food identification using image analysis. 3. Obtaining nutritional information. 4. Charting nutritional balance. 5. Displaying the results to the user. 6. Generating customized meal suggestions. 7. Suggesting ingredients based on budget. 8. Data integration with medical institutions.

[0758] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0759] Step 1:

[0760] The user opens the application. The user takes a picture of food using the smartphone camera. The food image is obtained as input. The emotion recognition module in the device analyzes the user's face and estimates their current emotion. This results in emotion data. When the user presses the "upload" button, the image and emotion data are sent to the server. The output is the image data and emotion data sent to the server.

[0761] Step 2:

[0762] The server adds the received image data to the analysis queue. The input is the received image data. The server's image analysis module performs initial preprocessing of the image, resizing, denoising, and color adjustment. The server then analyzes the image using a machine learning model (e.g., TensorFlow or PyTorch). Food items are identified and labeled. The quantity of each food item is also estimated. The output is data on the identified food items and their quantities.

[0763] Step 3:

[0764] The server analyzes the received emotional data. The input is emotional data. The server determines the user's emotional state and uses this data to adjust the meal suggestions. The output is the analyzed emotional state data.

[0765] Step 4:

[0766] The server queries a food database (e.g., USDA food database) based on the label of each identified food item. As input, it has the food item label. It retrieves detailed nutritional information (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) for each food item in the database. The output is the retrieved nutritional information.

[0767] Step 5:

[0768] The server generates a nutritional balance chart by comparing the acquired nutritional information with the user's recommended daily intake. The inputs are the acquired nutritional information and the user's recommended daily intake. The chart is created in a visually easy-to-understand format such as a radar chart or bar graph. The output is the generated nutritional balance chart data.

[0769] Step 6:

[0770] The server sends the generated chart data in JSON format to the terminal. The input is the generated chart data. The terminal displays the chart to the user using a drawing library (e.g., Chart.js or D3.js). The output is a nutritional balance chart displayed to the user.

[0771] Step 7:

[0772] The user inputs and updates health goals and chronic illness information within the app. The input includes health goals and chronic illness information. The device sends this information and emotional data to the server. The server updates the user profile based on the user's health goals, chronic illness information, and emotional state, and generates appropriate meal suggestions. The output is the generated customized meal suggestions.

[0773] Step 8:

[0774] The user uses a budget input form to input the desired budget amount. The input is the budget amount. The device sends the budget information to the server. The server queries the product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within the budget. The server filters based on the budget, nutritional balance, and emotional state, and sends the selected food list to the device. The output is a list of foods within the budget displayed to the user.

[0775] Step 9:

[0776] The user enables the "Link with Medical Institutions" option in the app settings. The inputs include the user's desire to link, nutritional balance data, and emotional data. The device sends the nutritional balance data and emotional data to the server along with the user's permission. The server takes appropriate security measures and shares the user data through an API dedicated to medical institutions. The output is the user data shared with the medical institution.

[0777] (Application example 2)

[0778] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0779] Today's consumers are health-conscious and tend to seek nutritionally balanced meals. However, it is difficult to understand what nutrients are contained in everyday meals. It is also complicated to make appropriate meal suggestions based on the user's health goals and emotional state, or to make purchasing suggestions based on their budget. Furthermore, while managing daily eating habits and collaborating with medical institutions is becoming increasingly important, there is a problem in that a seamless method for doing so has not yet been established.

[0780] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: [means for the user to take and upload images of foods;] [means for the server to analyze the received images and identify the foods; and [means for the server to obtain nutritional information for the identified foods and chart their nutritional balance.] This makes it possible [for the user to easily understand the nutritional balance of their daily meals and receive meal suggestions suited to their health goals and emotional state].

[0781] "Means for users to take and upload images of food" refers to a function that allows users to take photos of food using a device such as a smartphone or camera and send the image data to a server.

[0782] The "means for analyzing the image received by the server and identifying the food item" refers to the function of processing the image data received by the server and identifying the food item in the image. This analysis uses machine learning and image recognition algorithms.

[0783] "Means for the server to obtain nutritional information for identified foods and chart the nutritional balance" refers to a function that obtains nutritional data corresponding to identified food items from a database and visually represents it in a format that is easy for users to understand.

[0784] "Means for displaying a nutritional balance chart to a user via a terminal" refers to a function for showing a nutritional balance chart to a user on a device such as a smartphone or computer.

[0785] "A means for users to input information about chronic illnesses and health goals, and for the server to make individually appropriate meal suggestions" is a function in which users input their own health condition and goals, and the server then suggests appropriate meal plans based on that information.

[0786] "Means for recognizing the user's emotional state and dynamically adjusting meal suggestions" is a function that grasps the user's emotional state from their facial expressions and behavior, and changes the meal suggestions based on the results.

[0787] The "means to support budget-based food purchases on online shopping sites" is a function that generates a list of foods that can be purchased based on the budget set by the user and helps the user purchase food based on that information.

[0788] "Means for linking nutritional balance information with medical institutions" is a function that allows users to share their nutritional data with medical institutions, enabling them to receive appropriate treatment and advice.

[0789] To implement this invention, the following system configuration and processing procedures are required. The system allows users to take and upload images of food, analyzes the nutritional balance based on the images, and makes dietary suggestions based on individual health goals and emotional state. It also supports food purchasing according to budget and connects with medical institutions for nutritional information. The main hardware and software used to realize this system are described below.

[0790] Hardware and software used

[0791] Hardware:

[0792] Smartphone (iOS or Android)

[0793] Server (general cloud services, e.g. AWS EC2)

[0794] software:

[0795] Smartphone app (Swift for iOS, Kotlin for Android)

[0796] Machine learning libraries (TensorFlow, PyTorch)

[0797] Nutrition Database (public food database)

[0798] Image processing library (OpenCV)

[0799] Emotion Recognition Library (general-purpose emotion recognition API)

[0800] Processing overview and examples

[0801] Image upload and emotion recognition

[0802] The user opens the smartphone app and takes a photo of the food. After the photo is taken, the smartphone's built-in emotion recognition module analyzes the user's face and estimates their current emotional state. When the user presses the "upload" button, the image and emotional data are sent to the server.

[0803] Image analysis and food identification

[0804] The server adds the received image data to the analysis queue and invokes the image analysis module. As an initial preprocessing step, the OpenCV library is used to resize, denoise, and adjust the color of the image. Then, machine learning libraries (TensorFlow and PyTorch) are used to analyze the image and identify food items. These identified food items are labeled and their quantities are estimated.

[0805] Obtaining nutritional data and charting balance

[0806] The server queries a nutrition database based on the identified food item labels to obtain detailed nutritional information for each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) and generates a nutritional balance chart based on this information, comparing it with the user's recommended daily intake. The chart can be in the form of a radar chart, bar graph, or other format.

[0807] Displaying the results

[0808] The server sends the generated chart data in JSON format to the smartphone. The smartphone app uses a visualization library to draw the chart and display it visually to the user. The user can then check the chart to understand their own nutritional balance.

[0809] Customized meal suggestions

[0810] Users enter their health goals and chronic illness information in the app and send it to the server, which then updates the user profile based on this information and emotional data and generates personalized meal suggestions, including recipes and menus for dining out.

[0811] Food suggestions according to your budget

[0812] The user enters their desired budget amount using a budget input form. The smartphone app sends the budget information to the server. The server queries product databases from convenience stores and supermarkets to generate a list of foods that can be purchased within the budget. The server filters the food items based on budget, nutritional balance, and emotional state, and sends the selected list to the smartphone app. The app displays this list to the user, allowing them to purchase the foods through an online shopping site.

[0813] Collaboration with medical institutions

[0814] The user enables the "Link with Medical Institutions" option in the app settings. The smartphone then sends nutritional balance data and emotional data to the server with the user's permission. The server then takes appropriate security measures and shares the user data through an API dedicated to medical institutions. Based on this data, the medical institution provides the user with appropriate feedback and treatment plans.

[0815] Specific examples

[0816] A user eats curry rice for lunch and uploads a photo of it to the app. The smartphone sends the captured image and emotional state to the server, which analyzes the image and identifies each food. Based on the results, nutritional information is obtained and a nutritional balance chart is created. The chart is then sent to the smartphone, where the user can visually check it.

[0817] Furthermore, if the user has a goal of dieting, they input that goal into the app. If the emotion engine detects that the user is under high stress, the server takes this into consideration and suggests foods that have a relaxing effect (e.g., a recipe for salad chicken and tofu). If the user sets a budget of 1,000 yen for the next day and wants to buy a healthy lunch, the server generates a list of foods that can be purchased within that budget and displays it on the smartphone app. This allows the user to purchase healthy foods on the online shopping site while staying within their budget.

[0818] Prompt Sentence Examples

[0819] "Identify the food in this image and provide its nutritional information."

[0820] "Based on this nutritional balance chart, please suggest foods that are suitable for my diet goals."

[0821] "Give me a list of foods that are suitable for me on my budget, taking into account my emotional state."

[0822] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0823] Step 1:

[0824] A user opens a smartphone app and takes a photo of food. The input is the image of the food taken with the camera, and the output is the image data saved on the user's smartphone. Specifically, the app activates the camera function, the user takes a photo of the food, and saves the image.

[0825] Step 2:

[0826] The device's built-in emotion recognition module analyzes the user's face and estimates their emotional state. The input is a user's facial image, and the output is estimated emotion data. Specifically, the app captures a facial image and uses the emotion recognition library to analyze emotions from the facial expression data.

[0827] Step 3:

[0828] When the user presses the "Upload" button, the image data and emotion data are sent to the server. The input is the image data and emotion data of the food, and the output is the data received by the server. Specifically, the app generates an HTTP request and sends the image data and emotion data to the server.

[0829] Step 4:

[0830] The server adds the received image data to the analysis queue and invokes the image analysis module. The input is the image data stored on the server, and the output is the image data ready for analysis. Specifically, the server adds the image data to the queue and schedules it for analysis in the next stage.

[0831] Step 5:

[0832] The server uses the OpenCV library to perform initial preprocessing such as image resizing, noise removal, and color adjustment. The input is image data ready for analysis, and the output is preprocessed image data. Specifically, it calls OpenCV functions to perform processing to improve the image quality.

[0833] Step 6:

[0834] The server uses a machine learning model (TensorFlow or PyTorch) to analyze the image and identify the food items. The input is preprocessed image data, and the output is the identified food items and their labels. Specifically, the model inputs the image and performs inference to obtain the identification results.

[0835] Step 7:

[0836] Based on the identification results, the server sends a query to a nutrition database to obtain detailed nutrition information for each food item. The input is the food item label, and the output is the nutrition information. The specific operation is to generate a database query to obtain the required nutrition data.

[0837] Step 8:

[0838] Based on the nutritional information acquired by the server, a nutritional balance chart is generated by comparing it with the recommended daily intake. The input is detailed nutritional information, and the output is the nutritional balance chart data. Specifically, the server runs an algorithm that compares the data and generates the chart.

[0839] Step 9:

[0840] The chart data generated by the server is sent to the terminal in JSON format. The input is the nutritional balance chart data, and the output is the chart data received by the terminal. Specifically, the chart data is sent as an HTTP response.

[0841] Step 10:

[0842] The terminal uses the visualization library to draw charts and display them visually to the user. The input is the received chart data, and the output is the chart displayed to the user. Specifically, it calls the library's drawing functions to display the chart on the screen.

[0843] Step 11:

[0844] The user enters health goals and chronic disease information in the app and sends it to the server. The input is the health goals and chronic disease information entered by the user, and the output is the health information stored on the server. Specifically, the app provides an input form and sends the input data to the server.

[0845] Step 12:

[0846] The server generates meal suggestions based on the user's health goals, chronic illness information, and emotional state, and sends them to the user. The input is user profile data and emotional data, and the output is customized meal suggestions. Specifically, the server runs a suggestion generation algorithm to suggest appropriate meal menus.

[0847] Step 13:

[0848] The user uses a budget input form to enter the desired budget amount and send it to the server. The input is the budget information entered by the user, and the output is the budget information saved on the server. Specifically, the app provides an input form and sends the budget data to the server.

[0849] Step 14:

[0850] The server queries the product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within a budget. The input is budget information and food identification data, and the output is a list of foods that can be purchased. Specifically, the server generates a database query and retrieves the appropriate food list.

[0851] Step 15:

[0852] The server filters the food items based on budget, nutritional balance, and emotional state, and sends the selected food list to the device. The input is the available food list and user profile data, and the output is the filtered food list. Specifically, the server runs the filtering algorithm to generate an appropriate list.

[0853] Step 16:

[0854] The terminal displays the filtered food list to the user and allows them to purchase the food through the online shopping site. The input is the filtered food list, and the output is the displayed list and a purchase link. The specific operation is to display the list on the screen and provide a purchase link.

[0855] Step 17:

[0856] The user enables the "Link with Medical Institutions" option in the app settings. The input is the user's setting selection, and the output is the setting change. Specifically, the user turns on the option on the app settings screen.

[0857] Step 18:

[0858] The terminal transmits the nutritional balance data and emotion data to the server with the user's permission. The input is the data with the user's permission, and the output is the nutritional balance data and emotion data stored on the server. Specifically, the terminal transmits the data to the server after confirming the user's permission.

[0859] Step 19:

[0860] The server takes appropriate security measures and then shares user data through an API dedicated to medical institutions. The input is the saved user data, and the output is the data sent to the medical institution. Specifically, data is shared through a secure API.

[0861] The above are the specific processing steps of the system that realizes the application example. Each step is important for realizing the function of assisting the user in managing their health.

[0862] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0863] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0864] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0865] [Third embodiment]

[0866] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0867] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0868] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0869] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0870] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0871] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0872] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0873] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0874] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0875] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0876] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0877] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0878] This invention is a system that allows users to take pictures of foods, chart their nutritional balance based on those pictures, and make dietary suggestions based on individual health goals. This system includes a means for users to take and upload food images, a means for a server to analyze the received images and identify the foods, a means for the server to obtain nutritional information for the identified foods and chart their nutritional balance, a means for displaying the nutritional balance chart to the user via a terminal, a means for the user to input information about chronic illnesses and health goals and for the server to make individually appropriate dietary suggestions, a means for supporting budget-based food purchasing at convenience stores and supermarkets, and a means for linking nutritional balance information with medical institutions.

[0879] Program processing

[0880] Image upload

[0881] A user opens the application and takes a photo of a food. The image is saved on the device and an upload button is displayed. When the user presses the upload button, the image is sent to the server.

[0882] Image analysis

[0883] The server adds the received image to the analysis queue. It invokes the image analysis module, which first preprocesses the image (e.g., resizes the image, removes noise, etc.). The server then uses a machine learning model to recognize food in the image. Specifically, it segments the image region, identifies each food item, and labels each one. At this stage, the type and quantity of food are estimated.

[0884] Acquiring food data

[0885] The server queries the food database based on the label of each identified food, and retrieves detailed nutritional information (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) for each food in the database.

[0886] Nutritional balance chart

[0887] The server aggregates the acquired nutritional information and creates a chart of nutritional balance based on the generated information. The nutritional balance chart is displayed in a visually easy-to-understand format (radar chart, bar graph, etc.).

[0888] Displaying the results

[0889] The server sends the chart data in JSON format to the device, which then visually displays the chart to the user using a drawing library. The user can refer to this chart and understand their diet at a glance.

[0890] Customized Offers

[0891] Users enter their health goals and chronic illness information within the app. The device sends this information to the server, which uses this information to update the user's profile and provide personalized dietary suggestions. These suggestions are then sent to the device and displayed to the user.

[0892] Food suggestions according to your budget

[0893] The user enters their desired budget amount using a budget input form. The device sends the budget information to the server. The server queries the product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within the budget. The server filters the list based on budget and nutritional balance and sends the selected list to the device. The device displays this list to the user.

[0894] Collaboration with medical institutions

[0895] The user enables the "Link with medical institutions" option in the app settings. The device sends nutritional balance data to the server with the user's permission. The server takes security measures and shares the user data through an API dedicated to medical institutions. Medical institutions use this data to provide the user with appropriate feedback and treatment plans.

[0896] Specific examples

[0897] A user eats a hamburger steak set meal (rice, hamburger steak, salad) for lunch and uploads a photo of it to the app. The device sends the captured image to the server, which analyzes the image and identifies each food item. The server obtains nutritional information based on the identification results and creates a chart of nutritional balance. The chart is then sent to the device and the user can visually confirm it. Furthermore, if the user has a diet goal, they input that goal into the app. The server takes this into consideration and suggests a low-calorie dinner menu (e.g., a recipe for salad and grilled chicken). If a user sets a budget of 1,000 yen for the next day and wants to buy a healthy lunch, the server will suggest convenience store items (e.g., sandwiches, salad, and tea) that can be purchased within that budget. Users can maintain a healthy diet by referring to the ingredient list when making purchases. Users can also share their nutritional balance information with medical institutions and engage in regular health management.

[0898] In this way, the system allows users to easily and efficiently manage their daily diet and receive appropriate support according to their individual health conditions.

[0899] The processing flow will be explained below.

[0900] Step 1:

[0901] A user opens the application and takes a photo of the food. After taking the photo, the user presses the "upload" button.

[0902] Step 2:

[0903] The terminal stores the captured image file in a temporary storage area within the application, and transmits the image data to the server.

[0904] Step 3:

[0905] The server adds the received image data to the analysis queue and invokes the image analysis module. As initial preprocessing, it performs image resizing, noise removal, color adjustment, etc.

[0906] Step 4:

[0907] The server uses machine learning models to analyze the image, identify food item regions, label each item, and estimate the quantity of each food item.

[0908] Step 5:

[0909] The server queries the food database based on the label of each recognized food item, and obtains detailed nutritional information for each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.).

[0910] Step 6:

[0911] The server compares the collected nutritional information with the user's recommended daily intake and generates a nutritional balance chart in a visually easy-to-understand format such as a radar chart or bar graph.

[0912] Step 7:

[0913] The server sends the generated chart data in JSON format to the terminal.

[0914] Step 8:

[0915] The device uses a drawing library to display the chart data received to the user, who can then check the chart to understand their own nutritional balance.

[0916] Step 9:

[0917] Users enter and update their health goals and chronic illness information within the app.

[0918] Step 10:

[0919] The terminal transmits the input information to the server.

[0920] Step 11:

[0921] The server updates the user's profile based on their health goals and chronic illnesses, and generates suitable meal suggestions, including recipes and menus for dining out.

[0922] Step 12:

[0923] The server sends meal suggestions to the terminal.

[0924] Step 13:

[0925] The device displays suggested recipes and meal menus to the user, who can then use them to select a meal.

[0926] Step 14:

[0927] The user inputs the budget for purchasing ingredients, for example, "1000 yen."

[0928] Step 15:

[0929] The terminal transmits the budget information to the server.

[0930] Step 16:

[0931] The server queries product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within a budget. The list is filtered to take nutritional balance into consideration.

[0932] Step 17:

[0933] The server transmits the generated food list to the terminal.

[0934] Step 18:

[0935] The terminal displays a food list to the user, and the user uses the list as a reference when purchasing ingredients.

[0936] Step 19:

[0937] The user enables the "Connect with healthcare providers" option in the app settings.

[0938] Step 20:

[0939] The terminal confirms the user's permission and transmits the permission information along with the nutritional balance data to the server.

[0940] Step 21:

[0941] The server takes appropriate security measures and shares user data through an API dedicated to medical institutions.

[0942] Step 22:

[0943] Medical institutions receive the shared data and provide appropriate feedback and treatment plans to users.

[0944] Example 1

[0945] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0946] Currently, there are many dietary management tools and applications on the market, but most of them have limitations in their ability to effectively understand the user's dietary content and nutritional balance and make personalized suggestions based on that information. They also lack the functionality to suggest ingredients based on the user's budget or to support health management in collaboration with medical institutions. There is a need for a system that can improve these points and enable users to manage their health more effectively.

[0947] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0948] In this invention, the server includes means for preprocessing the received images and identifying foods using a machine learning model, means for acquiring nutritional information for the identified foods and charting their nutritional balance, and means for linking the nutritional balance information with medical institutions. This enables users to easily manage their diet through food images, receive personalized dietary suggestions, and even manage their health consistently through collaboration with medical institutions.

[0949] "User" refers to an individual who uses the system to take pictures of food and manage their diet.

[0950] "Server" refers to a central processing unit for receiving and analyzing data sent by users and providing necessary information.

[0951] "Terminal" refers to a computing device (such as a smartphone, tablet, or PC) used by a user.

[0952] "Food image" refers to photographic data containing food that a user takes and uploads to the system.

[0953] "Preprocessing" refers to the initial processing such as resizing and noise removal that is performed before the server analyzes the image.

[0954] A "machine learning model" refers to an algorithm that learns patterns from data and identifies and classifies foods.

[0955] A "food database" refers to a collection of data that stores nutritional information about each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.).

[0956] "Charting nutritional balance" refers to presenting collected nutritional information in a visually easy-to-understand format (such as a radar chart or bar graph).

[0957] "Health goal" refers to a goal related to a user's health status (e.g., dieting, muscle building, managing a chronic illness, etc.).

[0958] "Meal suggestions" refers to recommendations for providing an individualized meal plan based on the user's health goals and chronic illness information.

[0959] "Support for food purchasing" refers to providing users with a list of foods that can be purchased within their budget, and assisting them in their purchasing activities at convenience stores and supermarkets.

[0960] "Collaboration with medical institutions" refers to a system in which users' nutritional balance information is shared with medical institutions and feedback and treatment plans are provided as needed.

[0961] "Drawing library" refers to a software component for visually displaying data (e.g., D3.js, Chart.js, Matplotlib, etc.).

[0962] The present invention is a system that allows a user to take images of food, chart the nutritional balance based on the images, and make dietary suggestions based on individual health goals. This system includes means for the user to take and upload images of food, means for a server to analyze the received images and identify the food, means for the server to obtain nutritional information for the identified foods and chart the nutritional balance, means for displaying the nutritional balance chart to the user via a terminal, means for the user to input health goals and chronic illness information and for the server to make individually appropriate dietary suggestions, means for supporting food purchasing according to budget, means for linking nutritional balance information with medical institutions, and means for pre-processing the captured images by resizing and removing noise.

[0963] Hardware and software used

[0964] To realize this system, the following hardware and software are used.

[0965] 1. Hardware

[0966] User device: smartphone, tablet, or computer.

[0967] Server: Server equipment equipped with a high-performance CPU and GPU.

[0968] 2. Software

[0969] Image analysis module: Uses Python and utilizes libraries such as OpenCV and TensorFlow.

[0970] Database: A relational database such as MySQL or PostgreSQL.

[0971] Drawing libraries: Visualization libraries such as D3.js, Chart.js, Matplotlib, etc.

[0972] API: Implements a RESTful API to communicate between the server and the terminal.

[0973] Example of operation

[0974] A specific example of the operation of the system of the present invention will be described below.

[0975] Specific examples

[0976] A user eats a hamburger steak set meal (rice, hamburger steak, salad) for lunch and uploads a photo of it to the app. The user takes a photo of the food using the app's camera function and presses the upload button, which sends the image to the server. The server preprocesses the received image (for example, resizes the image to 256x256 pixels and removes noise) and begins analysis using a machine learning model using TensorFlow. The server analyzes the image, identifies the foods - hamburger steak, rice, and salad - and labels each food.

[0977] The server then queries a MySQL database based on the identified food labels to obtain detailed nutritional information for each food (e.g., calories, protein, fat, carbohydrates, vitamins, and minerals). The obtained nutritional information is aggregated and a nutritional balance chart is created. The server sends this chart information in JSON format to the device, which then visually displays the chart using a drawing library such as D3.js. Users can refer to the chart to understand their diet at a glance.

[0978] Furthermore, when a user sets "diet" as a "health goal" in the app, the server will suggest corresponding low-calorie menus (e.g., salad and grilled chicken recipes). If a user sets a budget of 1,000 yen and wants to buy a healthy lunch, the server will generate a list of foods that can be purchased within that budget (e.g., sandwiches, salads, and tea) and send it to the device. By referring to the list of ingredients when making purchases, the user can maintain a healthy diet.

[0979] Prompt Sentence Examples

[0980] Example inputs to a generative AI model:

[0981] "Write a program that analyzes an image of a person eating oatmeal, banana, and yogurt for breakfast, along with the detailed ingredients, and then charts and displays the nutritional balance of this meal."

[0982] This system allows users to easily and efficiently manage their daily diet and receive appropriate support according to their individual health conditions.

[0983] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0984] Step 1:

[0985] The user takes a picture of a food item and presses the upload button in the application. The device saves the image to its internal storage and sends an upload request to the server. The input data is the food image taken by the user, and the output is the image sent to the server.

[0986] Specific behavior:

[0987] A user takes a photo of a hamburger steak set meal using the camera on their smartphone.

[0988] The device saves the image file (e.g., lunch.jpg) to its internal storage.

[0989] The user taps the upload button in the app and the image is sent to the server.

[0990] Step 2:

[0991] The server receives the uploaded images and adds them to the analysis queue. The server uses image analysis modules to preprocess the images and convert them into an analyzable format. The input data is the original uploaded image, and the output is the preprocessed image data.

[0992] Specific behavior:

[0993] The server resizes the image to 256x256 pixels and denoises it.

[0994] The preprocessed image data is fed into a machine learning model.

[0995] Step 3:

[0996] The server uses a machine learning model (e.g., TensorFlow) to analyze the preprocessed images and identify the foods. It identifies each food item in the image and assigns a label to each. The input data is the preprocessed image data, and the output is the label and estimated quantity of the identified food.

[0997] Specific behavior:

[0998] The server uses the YOLO model to detect food items (e.g., hamburger, rice, salad) in the image.

[0999] The server labels each food item and estimates the quantity (e.g., "hamburger": 1 piece, "rice": 1 bowl, "salad": 1 plate).

[1000] Step 4:

[1001] The server queries the food database based on the label of each identified food and retrieves detailed nutritional information for each food (calories, protein, fat, carbohydrates, vitamins, minerals, etc.). The input data is the label of the identified food, and the output is the retrieved nutritional information.

[1002] Specific behavior:

[1003] The server uses a SELECT query to retrieve nutritional information from the MySQL database based on the label "hamburger."

[1004] Similarly, obtain nutritional information for each food item (e.g., "Calories": 350kcal, "Protein": 25g).

[1005] Step 5:

[1006] The server aggregates the acquired nutritional information and generates chart data to visually represent nutritional balance. The input data is detailed nutritional information for each food, and the output is chart data.

[1007] Specific behavior:

[1008] A server aggregates the nutritional information of all foods.

[1009] The server uses the Matplotlib library to plot the nutritional balance as a radar chart.

[1010] Step 6:

[1011] The server generates chart data and sends it to the terminal in JSON format. The terminal uses a drawing library (e.g., D3.js) to display the chart to the user based on the data received. The input data is the chart data, and the output is a visual chart that is displayed to the user.

[1012] Specific behavior:

[1013] The server converts the chart data into JSON format and sends it to the terminal.

[1014] The device uses Chart.js to draw the nutritional balance in the form of a bar graph and displays it on the app's UI.

[1015] Step 7:

[1016] The user enters their health goals and chronic illness information into the app. The device sends this information to the server, which then makes personalized dietary suggestions. The input data is the user's health goals and chronic illness information, and the output is dietary suggestions.

[1017] Specific behavior:

[1018] The user enters "health goal" as "diet."

[1019] The server will then suggest a low-calorie option (e.g., salad and grilled chicken) based on that information.

[1020] Step 8:

[1021] The user uses a budget input form to input the desired budget amount. The device sends the budget information to the server, which then executes a query to generate a list of foods that can be purchased within the budget. The input data is the user's budget information, and the output is a list of foods that can be purchased within the budget.

[1022] Specific behavior:

[1023] The user sets the "budget" to "1,000 yen" within the app.

[1024] The server generates a list of foods that can be purchased for under 1,000 yen, suggesting sandwiches, salads, and tea.

[1025] Step 9:

[1026] The user enables the "Link with medical institutions" option in the app settings. Based on the user's permission, the device sends nutritional balance data to the server, and the server shares the user data through an API dedicated to medical institutions. The input data is the user's nutritional balance data and medical institution information, and the output is data shared with the medical institution.

[1027] Specific behavior:

[1028] The user enables the "Connect with Healthcare Providers" option.

[1029] The server encrypts the data and sends it to an HTTPS endpoint dedicated to the medical institution.

[1030] (Application example 1)

[1031] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1032] In modern society, there is a demand for appropriate nutritional management and dietary suggestions tailored to each individual's health condition. However, daily dietary management is complicated, and it is particularly difficult to confirm the nutritional balance of food when eating out or purchasing it. Furthermore, when purchasing at convenience stores or supermarkets, it is not easy to select ingredients that take nutritional balance and budget into consideration. Furthermore, there is a demand for more appropriate health management by linking nutritional information with medical institutions, but current systems are not able to adequately address this demand.

[1033] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1034] In this invention, the server includes means for a user to photograph and upload images of foods, means for the server to analyze the received images and identify the foods, means for the server to obtain nutritional information for the identified foods and create a nutritional balance chart, means for the server to display the nutritional balance chart to the user via a terminal, means for the user to input chronic illness information and health goals and for the server to make individually appropriate meal suggestions, means for supporting budget-based food purchasing at sales facilities, means for linking the nutritional balance information with medical institutions, means for analyzing photographed images of food shelves and obtaining nutritional information for each food item to display to the user, and means for suggesting foods that can be purchased within the user's budget based on the user's input data. This allows users to check nutritional information on the spot when purchasing food at a physical store, make healthy food choices within their budget, and even manage their health in cooperation with medical institutions.

[1035] "User" refers to an individual who uses the system to take pictures of food and receive nutritional balance and meal suggestions.

[1036] "Server" refers to a central computer system for analyzing food images, obtaining nutritional information, and displaying it to the user.

[1037] The "means for taking and uploading images of food" refers to the part that provides the function for users to take images of food using a smartphone or smart glasses and send those images to the system.

[1038] "Means for identifying food" refers to technology that analyzes the image received by the server and identifies the food in the image.

[1039] "Means for obtaining nutritional information and charting nutritional balance" refers to the function of collecting nutritional data for identified foods and displaying it in a visually easy-to-understand format.

[1040] "Means for displaying a nutritional balance chart to a user on a terminal" refers to a function that allows a user to see a nutritional balance chart on a terminal such as a smartphone or computer.

[1041] "A means for inputting health goals and for the server to make individually appropriate meal suggestions" refers to a function in which the user inputs information about chronic illnesses and health goals, and the server creates customized meal suggestions based on that information.

[1042] "Means to support food purchasing according to budget at sales facilities" refers to a function that suggests foods that can be purchased taking into account the user's budget and supports the purchase.

[1043] "Means for linking nutritional balance information with medical institutions" refers to a function for sharing a user's nutritional information with medical institutions and supporting health management.

[1044] "Means for analyzing images of food shelves, obtaining nutritional information for each food item, and displaying this information to the user" refers to a function that analyzes images of food shelves taken at a sales facility, obtains nutritional information for each food item, and provides this information to the user.

[1045] "Means for suggesting foods that can be purchased within a budget based on input data" refers to a function that creates and suggests a list of foods that can be purchased within a budget based on the user's input data (budget and health goals).

[1046] This invention is a system that allows a user to take an image of food, charts the nutritional balance based on the image, and makes meal suggestions based on individual health goals. The system includes the following means:

[1047] 1. Image uploading method: Users take pictures of the food shelves using their smartphones or smart glasses and upload them to the server from their devices. The uploaded images are stored on the server.

[1048] 2. Image analysis: The server analyzes the received images and identifies the food in the image. This image analysis is performed using a machine learning model (e.g., using TensorFlow). The server first preprocesses the images (resizes, removes noise, etc.), then identifies and labels the food.

[1049] 3. Nutritional information acquisition means: Based on the food identified from the analyzed image, the server acquires the detailed nutritional information of the food from a food database (e.g., an internally stored nutritional database), including information such as calories, protein, fat, carbohydrates, vitamins, and minerals.

[1050] 4. Nutritional balance charting: The server aggregates the nutritional balance based on the acquired nutritional information and displays it in a visually easy-to-understand format (for example, a radar chart or bar graph). This allows users to understand the nutritional balance of the foods they have consumed at a glance.

[1051] 5. Display of results: The charted nutritional balance is sent from the server to the device, which then uses a visualization library to render it and display it to the user, allowing the user to understand their own dietary habits.

[1052] 6. Customized Meal Suggestion: Users can input their health goals and chronic illness information within the application. This information is sent to the server, which updates the user's profile and provides personalized meal suggestions. These suggestions are sent to the device, providing the user with guidelines for maintaining a healthy diet.

[1053] 7. Budget-based food recommendation: The user inputs their desired budget amount into the application. The server connects to the database of sales facilities to generate a list of foods that can be purchased within the budget. It filters based on budget and nutritional balance and sends the selected food list to the terminal. The terminal displays this list to the user, who can receive assistance in purchasing healthy foods within their budget.

[1054] 8. Linking with medical institutions: When the user enables the "Linking with medical institutions" option, the device will send nutritional balance data to the server with the user's permission. The server will then share the user data through an API dedicated to medical institutions, after taking security measures. Medical institutions will use this data to provide the user with appropriate feedback and treatment plans.

[1055] For example, when a user takes a photo of a food shelf in a physical store and uploads it, the server analyzes the image, obtains nutritional information for each food item, and displays it to the user. Based on this information, the user can select foods that take into account nutritional balance and budget on the spot. Furthermore, by inputting the user's health goals and budget, the server can make optimal meal suggestions.

[1056] Additionally, examples of prompts that are useful as input to generative AI models include:

[1057] "A user takes a photo of a food shelf in a physical store and uploads it. Please explain how the system uses the image to create a nutritional balance chart and provide meal suggestions aligned with the user's health goals. Specifically, please explain in detail the process of image upload, image analysis, nutritional information acquisition, and meal suggestions. Please also include suggestions based on a budget."

[1058] In this way, the system allows users to easily and efficiently manage their daily diet and receive appropriate support according to their individual health conditions.

[1059] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1060] Step 1: Upload an image

[1061] A user takes a photo of the food shelf with a smartphone or smart glasses. The device saves the image and displays an upload button. When the user presses the upload button, the image is sent from the device to the server. The input is the image taken by the user, and the output is the image data sent to the server.

[1062] Step 2: Image analysis

[1063] The server adds the received image to the analysis queue. It invokes the image analysis module to first preprocess the image (resize, remove noise, etc.). The server then uses a generative AI model to recognize food in the image. This process involves segmenting the image area, identifying each food item, and labeling each one. The input is the image data sent to the server, and the output is the label information of the identified food and its location information.

[1064] Step 3: Obtain nutritional information

[1065] The server queries an internal food database based on the label of each identified food, and obtains detailed nutritional information (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) for each food in the database. The input is the label information of the identified food, and the output is the nutritional information.

[1066] Step 4: Nutritional Balance Chart

[1067] The server aggregates the acquired nutritional information and charts the nutritional balance in a visually easy-to-understand format (such as a radar chart or bar graph). The chart data is generated in JSON format. The input is the aggregated nutritional information, and the output is the JSON data of the nutritional balance chart.

[1068] Step 5: View the results

[1069] The server sends the chart data to the terminal, which uses a visualization library to draw the chart and display it visually to the user. The input is the JSON data of the chart, and the output is the nutritional balance chart displayed to the user.

[1070] Step 6: Customized Meal Suggestions

[1071] The user enters their health goals and chronic illness information within the application. The device sends this information to the server. The server updates the user's profile and generates personalized meal suggestions. The suggestions are sent to the device and displayed to the user. The input is the user's health goals and chronic illness information, and the output is personalized meal suggestions.

[1072] Step 7: Proposing ingredients according to your budget

[1073] The user enters their budget in the app. The device sends the budget information to the server. The server queries a database of sales establishments to generate a list of foods that can be purchased within the budget. After filtering, the server sends the selected list of ingredients to the device. The device displays this list to the user. The input is the user's budget information, and the output is a list of foods that the user can purchase.

[1074] Step 8: Collaborate with medical institutions

[1075] The user enables the "Link with medical institutions" option in the app settings. The device sends nutritional balance data to the server with the user's permission. The server takes security measures and shares the user data through an API dedicated to medical institutions. Based on this data, medical institutions provide the user with appropriate feedback and treatment plans. The input is the user's nutritional balance data, and the output is the data sent to the medical institution.

[1076] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1077] The present invention is a system that allows a user to take an image of food, charts the nutritional balance based on the image, and makes meal suggestions based on individual health goals and emotional state. This system includes means for a user to take and upload an image of food, means for a server to analyze the received image and identify the food, means for the server to obtain nutritional information for the identified food and chart the nutritional balance, means for displaying the nutritional balance chart to the user via a terminal, means for the user to input information about chronic illnesses and health goals and for the server to make individually appropriate meal suggestions, means for supporting budget-based food purchasing at convenience stores and supermarkets, means for linking nutritional balance information with medical institutions, and means for recognizing the user's emotions using an emotion engine.

[1078] Program processing

[1079] Image upload and emotion recognition

[1080] The user opens the application and takes a photo of the food. After taking the photo, the device's emotion recognition module analyzes the user's face and estimates their current emotion. When the user presses the "Upload" button, the image and emotion data are sent to the server.

[1081] Image analysis

[1082] The server adds the received image data to an analysis queue and invokes the image analysis module. Initial preprocessing includes image resizing, noise reduction, and color adjustment. The server then analyzes the image using a machine learning model to identify and label each food item. The quantity of each food item is also estimated.

[1083] Emotional Data Processing

[1084] The server analyzes the received emotional data to determine the user's emotional state, which is used as input data for dynamically adjusting the user's meal suggestions.

[1085] Acquiring food data

[1086] The server queries the food database based on the label of each identified food item, and retrieves detailed nutritional information (e.g., calories, protein, fat, carbohydrates, vitamins, minerals) for each food item in the database.

[1087] Nutritional balance chart

[1088] The server compares the nutritional information it obtains with the user's recommended daily intake and generates a nutritional balance chart in a visually easy-to-understand format, such as a radar chart or bar graph.

[1089] Displaying the results

[1090] The server sends the generated chart data in JSON format to the device. The device displays the chart to the user using a drawing library. The user can check the chart and understand their own nutritional balance.

[1091] Customized Offers

[1092] Users input and update their health goals and chronic illness information within the app. The device then sends this information and emotional data to the server. The server then updates the user profile based on the user's health goals, chronic illness information, and emotional state, and generates appropriate meal suggestions. These suggestions include recipes and menus for dining out.

[1093] Food suggestions according to your budget

[1094] The user enters their desired budget amount using a budget input form. The device sends the budget information to the server. The server queries convenience store and supermarket product databases to generate a list of foods that can be purchased within the budget. The server filters the list based on budget, nutritional balance, and emotional state, and sends the selected list to the device. The device displays this list to the user.

[1095] Collaboration with medical institutions

[1096] The user enables the "Link with Medical Institutions" option in the app settings. With the user's permission, the device sends nutritional balance data and emotional data to the server. The server takes appropriate security measures and shares the user data through an API dedicated to medical institutions. Medical institutions use this data to provide the user with appropriate feedback and treatment plans.

[1097] Specific examples

[1098] A user eats a hamburger steak set meal (rice, hamburger steak, salad) for lunch and uploads a photo of it to the app. The device sends the captured image to the server, which analyzes the image and identifies each food item. The server obtains nutritional information based on the identification results and creates a chart of the nutritional balance. The chart is then sent to the device and the user can visually check it.

[1099] Furthermore, if a user has a goal of losing weight and inputs that goal into the app, if the emotion engine detects that the user is feeling low, the server will take that into account and suggest a high-protein, low-calorie dinner menu (e.g., a recipe for salad and grilled chicken) to help them regain their energy.

[1100] If a user sets a budget of 1,000 yen for the next day and wants to buy a healthy lunch, the server will suggest convenience store items (e.g., sandwiches, salads, tea) that can be purchased within that budget. The user can purchase ingredients based on this list and maintain a healthy diet.

[1101] Users can also share their nutritional balance information and emotional data with medical institutions and receive regular health management advice from doctors. This system allows users to efficiently manage their daily diet and receive appropriate support tailored to their individual health and emotional states.

[1102] The processing flow will be explained below.

[1103] Step 1:

[1104] The user opens the application and takes a photo of the food. After taking the photo, the device's emotion engine analyzes the user's face in real time and estimates their current emotional state. The user then presses the "Upload" button.

[1105] Step 2:

[1106] The terminal stores the captured image file and emotion data in a temporary storage area within the application, and transmits the data to the server.

[1107] Step 3:

[1108] The server adds the received image data to the analysis queue and invokes the image analysis module. As initial preprocessing, it performs image resizing, noise removal, color adjustment, etc.

[1109] Step 4:

[1110] The server uses machine learning models to analyze the image, identify food item regions, label each item, and estimate the quantity of each food item.

[1111] Step 5:

[1112] The server queries the food database based on the label of each recognized food item, and obtains detailed nutritional information for each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.).

[1113] Step 6:

[1114] The server compares the collected nutritional information with the user's recommended daily intake and generates a nutritional balance chart in a visually easy-to-understand format such as a radar chart or bar graph.

[1115] Step 7:

[1116] The server analyzes the emotional data to determine the user's emotional state, which is used as input data for dynamically adjusting the user's meal suggestions.

[1117] Step 8:

[1118] The server sends the generated chart data and the user's emotional state in JSON format to the terminal.

[1119] Step 9:

[1120] The device uses a drawing library to display the chart data and emotion data received to the user, who can then check the chart to understand their own nutritional balance and emotional state.

[1121] Step 10:

[1122] Users enter and update their health goals and chronic illness information within the app, and the device sends this information and emotional data to the server.

[1123] Step 11:

[1124] The server updates the user's profile based on the user's health goals, chronic illness information, and emotional data, and generates suitable meal suggestions, including recipes and menus for dining out.

[1125] Step 12:

[1126] The server sends meal suggestions to the terminal.

[1127] Step 13:

[1128] The device displays suggested recipes and meal menus to the user, who then uses them to select a meal.

[1129] Step 14:

[1130] The user inputs the budget for purchasing ingredients, for example, "1000 yen."

[1131] Step 15:

[1132] The terminal transmits the budget information to the server.

[1133] Step 16:

[1134] The server queries convenience store and supermarket product databases to generate a list of food items that can be purchased within a budget. The food list is filtered based on nutritional balance and emotional state.

[1135] Step 17:

[1136] The server transmits the generated food list to the terminal.

[1137] Step 18:

[1138] The terminal displays a food list to the user, and the user uses the list as a reference when purchasing ingredients.

[1139] Step 19:

[1140] The user enables the "Connect with healthcare providers" option in the app settings.

[1141] Step 20:

[1142] The terminal confirms the user's permission and transmits the permission information along with the nutritional balance data and emotion data to the server.

[1143] Step 21:

[1144] The server takes appropriate security measures and shares user data through an API dedicated to medical institutions.

[1145] Step 22:

[1146] Medical institutions receive the shared data and provide appropriate feedback and treatment plans to users.

[1147] Example 2

[1148] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1149] Conventional dietary management systems make it difficult for users to understand the nutritional balance of foods and receive dietary suggestions tailored to their health goals. They also lack the ability to provide dietary suggestions based on emotional state or budget, and lack the ability to connect with medical institutions, making comprehensive health management difficult.

[1150] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: [means for an emotion recognition module of the terminal to analyze the user's face and estimate the emotion;] [means for the server to add received images to an analysis queue, and for the image analysis module to preprocess the images and identify foods using a machine learning model;] [means for the server to obtain nutritional information of the identified foods from a database and chart their nutritional balance;] [means for the server to select and provide foods according to a budget entered by the user; and] [means for linking nutritional balance information and emotion data with a medical institution. This allows the user's food selection to be optimized according to their emotional state and budget, and also enables health management in collaboration with a medical institution.

[1151] A "user" is someone who uses the system to take and upload images of food and receive nutritional balance and meal suggestions.

[1152] "Terminal" refers to a device operated by a user, such as a smartphone or tablet, and includes an emotion recognition module and a chart drawing library.

[1153] The "server" is the central computer of the system, and is a device that performs processes such as image analysis, database queries, meal suggestion generation, and medical institution collaboration.

[1154] An "emotion recognition module" is software or hardware built into a device that has the function of analyzing a user's face and estimating their emotional state.

[1155] "Image Analysis Module" means software within the server that pre-processes received images and uses machine learning models to identify food products.

[1156] A "machine learning model" is an algorithm that makes predictions and classifications based on data, and is a type of artificial intelligence used in image analysis modules.

[1157] A "food database" is a data store containing detailed nutritional information for various foods, to which a server can send queries to retrieve the required data.

[1158] A "nutritional balance chart" is a diagram that visually shows the user's nutritional intake status, and is displayed in the form of a radar chart or bar graph.

[1159] A "visualization library" is a software library for drawing charts and graphs, used on a device to visually display data in JSON format.

[1160] "Health goals" are goals that users set within the app based on their individual health status and lifestyle goals, and are taken into consideration when making meal suggestions.

[1161] The "budget" is a price range set by the user when purchasing ingredients, and is a guideline used to generate the food list provided by the server.

[1162] "Medical institution" refers to a hospital, clinic, medical office, etc., and is a facility that can share users' nutritional balance information and emotion data by linking with a server.

[1163] "Meal suggestions" refers to suggestions such as recipes and dining out menus generated by the server based on the user's health goals, emotional state, and budget.

[1164] The present invention is a system that allows users to take pictures of food, charts the nutritional balance based on the pictures, and suggests meals based on individual health goals and emotional state. Specific methods for implementing this system are described below.

[1165] The system includes devices such as smartphones and tablets, a server, a food database, and related software modules, including emotion recognition modules, image analysis modules, machine learning models (e.g., TensorFlow, PyTorch), and visualization libraries (e.g., Chart.js, D3.js).

[1166] First, the user opens the application and takes a picture of the food using the device's camera. An emotion recognition module (e.g., Face++ API) on the device analyzes the user's face and estimates their current emotion. This emotion data is later used to customize meal suggestions. After taking the picture, the user presses the "Upload" button, and the image and emotion data are sent to the server.

[1167] The server then adds the received image data to an analysis queue, where the image analysis module operates. Image preprocessing involves resizing, noise reduction, and color adjustment. Furthermore, a machine learning model (e.g., a trained CNN model) is used to analyze the image, identify food items, and label each one. The quantity of each food item is also estimated.

[1168] The server queries a food database (e.g., USDA Food Database) via its API to obtain detailed nutritional information (e.g., calories, protein, fat, carbohydrates, vitamins, minerals) for each identified food item. The server then generates a nutritional balance chart based on the nutritional information obtained and compares it with the user's recommended daily intake. This chart is created in a visually easy-to-understand format, such as a radar chart or bar graph, and sent to the device in JSON format.

[1169] The device uses a chart drawing library (e.g., Chart.js, D3.js) to display a nutritional balance chart to the user, allowing the user to visually understand their own nutritional balance.

[1170] Next, the user enters their health goals and chronic illness information into the app, and the data is sent from the device to the server. The server uses this information to update the user profile and generate meal suggestions that take into account the user's emotional data and health goals. For example, high-protein, low-calorie recipes or restaurant menus may be suggested.

[1171] Furthermore, when the user enters their desired budget amount using a form to input their budget, the device sends the budget information to the server. The server queries the product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within the budget. The food list is filtered based on the budget, nutritional balance, and emotional state and sent to the device and displayed to the user.

[1172] Finally, if the user enables the "Connect with Medical Institutions" option in the app settings, the device will send nutritional balance data and emotional data to the server with the user's permission. The server will then share this data with medical institutions through APIs after implementing security measures, enabling feedback and treatment plans from medical institutions.

[1173] For example, a user eats a hamburger steak set meal (rice, hamburger steak, and salad) for lunch and uploads a photo of it to the app. The device sends the captured image and emotional data to the server, which analyzes the image and identifies each food item. The server then obtains nutritional information, creates a nutritional balance chart, and sends it to the device. The user visually checks the chart and then sets a diet goal, which suggests a high-protein, low-calorie dinner menu based on their emotional state.

[1174] An example prompt using a generative AI model is as follows:

[1175] Describe a system that evaluates nutritional balance based on food images and emotion data taken by the user with a smartphone, and then suggests customized meals based on health goals. The following steps are performed in order: 1. Uploading images and emotion data. 2. Food identification using image analysis. 3. Obtaining nutritional information. 4. Charting nutritional balance. 5. Displaying the results to the user. 6. Generating customized meal suggestions. 7. Suggesting ingredients based on budget. 8. Data integration with medical institutions.

[1176] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1177] Step 1:

[1178] The user opens the application. The user takes a picture of food using the smartphone camera. The food image is obtained as input. The emotion recognition module in the device analyzes the user's face and estimates their current emotion. This results in emotion data. When the user presses the "upload" button, the image and emotion data are sent to the server. The output is the image data and emotion data sent to the server.

[1179] Step 2:

[1180] The server adds the received image data to the analysis queue. The input is the received image data. The server's image analysis module performs initial preprocessing of the image, resizing, denoising, and color adjustment. The server then analyzes the image using a machine learning model (e.g., TensorFlow or PyTorch). Food items are identified and labeled. The quantity of each food item is also estimated. The output is data on the identified food items and their quantities.

[1181] Step 3:

[1182] The server analyzes the received emotional data. The input is emotional data. The server determines the user's emotional state and uses this data to adjust the meal suggestions. The output is the analyzed emotional state data.

[1183] Step 4:

[1184] The server queries a food database (e.g., USDA food database) based on the label of each identified food item. As input, it has the food item label. It retrieves detailed nutritional information (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) for each food item in the database. The output is the retrieved nutritional information.

[1185] Step 5:

[1186] The server generates a nutritional balance chart by comparing the acquired nutritional information with the user's recommended daily intake. The inputs are the acquired nutritional information and the user's recommended daily intake. The chart is created in a visually easy-to-understand format such as a radar chart or bar graph. The output is the generated nutritional balance chart data.

[1187] Step 6:

[1188] The server sends the generated chart data in JSON format to the terminal. The input is the generated chart data. The terminal displays the chart to the user using a drawing library (e.g., Chart.js or D3.js). The output is a nutritional balance chart displayed to the user.

[1189] Step 7:

[1190] The user inputs and updates health goals and chronic illness information within the app. The input includes health goals and chronic illness information. The device sends this information and emotional data to the server. The server updates the user profile based on the user's health goals, chronic illness information, and emotional state, and generates appropriate meal suggestions. The output is the generated customized meal suggestions.

[1191] Step 8:

[1192] The user uses a budget input form to input the desired budget amount. The input is the budget amount. The device sends the budget information to the server. The server queries the product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within the budget. The server filters based on the budget, nutritional balance, and emotional state, and sends the selected food list to the device. The output is a list of foods within the budget displayed to the user.

[1193] Step 9:

[1194] The user enables the "Link with Medical Institutions" option in the app settings. The inputs include the user's desire to link, nutritional balance data, and emotional data. The device sends the nutritional balance data and emotional data to the server along with the user's permission. The server takes appropriate security measures and shares the user data through an API dedicated to medical institutions. The output is the user data shared with the medical institution.

[1195] (Application example 2)

[1196] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1197] Today's consumers are health-conscious and tend to seek nutritionally balanced meals. However, it is difficult to understand what nutrients are contained in everyday meals. It is also complicated to make appropriate meal suggestions based on the user's health goals and emotional state, or to make purchasing suggestions based on their budget. Furthermore, while managing daily eating habits and collaborating with medical institutions is becoming increasingly important, there is a problem in that a seamless method for doing so has not yet been established.

[1198] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes: [means for the user to take and upload images of foods;] [means for the server to analyze the received images and identify the foods; and [means for the server to obtain nutritional information for the identified foods and chart their nutritional balance.] This makes it possible [for the user to easily understand the nutritional balance of their daily meals and receive meal suggestions suited to their health goals and emotional state].

[1199] "Means for users to take and upload images of food" refers to a function that allows users to take photos of food using a device such as a smartphone or camera and send the image data to a server.

[1200] The "means for analyzing the image received by the server and identifying the food item" refers to the function of processing the image data received by the server and identifying the food item in the image. This analysis uses machine learning and image recognition algorithms.

[1201] "Means for the server to obtain nutritional information for identified foods and chart the nutritional balance" refers to a function that obtains nutritional data corresponding to identified food items from a database and visually represents it in a format that is easy for users to understand.

[1202] "Means for displaying a nutritional balance chart to a user via a terminal" refers to a function for showing a nutritional balance chart to a user on a device such as a smartphone or computer.

[1203] "A means for users to input information about chronic illnesses and health goals, and for the server to make individually appropriate meal suggestions" is a function in which users input their own health condition and goals, and the server then suggests appropriate meal plans based on that information.

[1204] "Means for recognizing the user's emotional state and dynamically adjusting meal suggestions" is a function that grasps the user's emotional state from their facial expressions and behavior, and changes the meal suggestions based on the results.

[1205] The "means to support budget-based food purchases on online shopping sites" is a function that generates a list of foods that can be purchased based on the budget set by the user and helps the user purchase food based on that information.

[1206] "Means for linking nutritional balance information with medical institutions" is a function that allows users to share their nutritional data with medical institutions, enabling them to receive appropriate treatment and advice.

[1207] To implement this invention, the following system configuration and processing procedures are required. The system allows users to take and upload images of food, analyzes the nutritional balance based on the images, and makes dietary suggestions based on individual health goals and emotional state. It also supports food purchasing according to budget and connects with medical institutions for nutritional information. The main hardware and software used to realize this system are described below.

[1208] Hardware and software used

[1209] Hardware:

[1210] Smartphone (iOS or Android)

[1211] Server (general cloud services, e.g. AWS EC2)

[1212] software:

[1213] Smartphone app (Swift for iOS, Kotlin for Android)

[1214] Machine learning libraries (TensorFlow, PyTorch)

[1215] Nutrition Database (public food database)

[1216] Image processing library (OpenCV)

[1217] Emotion Recognition Library (general-purpose emotion recognition API)

[1218] Processing overview and examples

[1219] Image upload and emotion recognition

[1220] The user opens the smartphone app and takes a photo of the food. After the photo is taken, the smartphone's built-in emotion recognition module analyzes the user's face and estimates their current emotional state. When the user presses the "upload" button, the image and emotional data are sent to the server.

[1221] Image analysis and food identification

[1222] The server adds the received image data to the analysis queue and invokes the image analysis module. As an initial preprocessing step, the OpenCV library is used to resize, denoise, and adjust the color of the image. Then, machine learning libraries (TensorFlow and PyTorch) are used to analyze the image and identify food items. These identified food items are labeled and their quantities are estimated.

[1223] Obtaining nutritional data and charting balance

[1224] The server queries a nutrition database based on the identified food item labels to obtain detailed nutritional information for each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) and generates a nutritional balance chart based on this information, comparing it with the user's recommended daily intake. The chart can be in the form of a radar chart, bar graph, or other format.

[1225] Displaying the results

[1226] The server sends the generated chart data in JSON format to the smartphone. The smartphone app uses a visualization library to draw the chart and display it visually to the user. The user can then check the chart to understand their own nutritional balance.

[1227] Customized meal suggestions

[1228] Users enter their health goals and chronic illness information in the app and send it to the server, which then updates the user profile based on this information and emotional data and generates personalized meal suggestions, including recipes and menus for dining out.

[1229] Food suggestions according to your budget

[1230] The user enters their desired budget amount using a budget input form. The smartphone app sends the budget information to the server. The server queries product databases from convenience stores and supermarkets to generate a list of foods that can be purchased within the budget. The server filters the food items based on budget, nutritional balance, and emotional state, and sends the selected list to the smartphone app. The app displays this list to the user, allowing them to purchase the foods through an online shopping site.

[1231] Collaboration with medical institutions

[1232] The user enables the "Link with Medical Institutions" option in the app settings. The smartphone then sends nutritional balance data and emotional data to the server with the user's permission. The server then takes appropriate security measures and shares the user data through an API dedicated to medical institutions. Based on this data, the medical institution provides the user with appropriate feedback and treatment plans.

[1233] Specific examples

[1234] A user eats curry rice for lunch and uploads a photo of it to the app. The smartphone sends the captured image and emotional state to the server, which analyzes the image and identifies each food. Based on the results, nutritional information is obtained and a nutritional balance chart is created. The chart is then sent to the smartphone, where the user can visually check it.

[1235] Furthermore, if the user has a goal of dieting, they input that goal into the app. If the emotion engine detects that the user is under high stress, the server takes this into consideration and suggests foods that have a relaxing effect (e.g., a recipe for salad chicken and tofu). If the user sets a budget of 1,000 yen for the next day and wants to buy a healthy lunch, the server generates a list of foods that can be purchased within that budget and displays it on the smartphone app. This allows the user to purchase healthy foods on the online shopping site while staying within their budget.

[1236] Prompt Sentence Examples

[1237] "Identify the food in this image and provide its nutritional information."

[1238] "Based on this nutritional balance chart, please suggest foods that are suitable for my diet goals."

[1239] "Give me a list of foods that are suitable for me on my budget, taking into account my emotional state."

[1240] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1241] Step 1:

[1242] A user opens a smartphone app and takes a photo of food. The input is the image of the food taken with the camera, and the output is the image data saved on the user's smartphone. Specifically, the app activates the camera function, the user takes a photo of the food, and saves the image.

[1243] Step 2:

[1244] The device's built-in emotion recognition module analyzes the user's face and estimates their emotional state. The input is a user's facial image, and the output is estimated emotion data. Specifically, the app captures a facial image and uses the emotion recognition library to analyze emotions from the facial expression data.

[1245] Step 3:

[1246] When the user presses the "Upload" button, the image data and emotion data are sent to the server. The input is the image data and emotion data of the food, and the output is the data received by the server. Specifically, the app generates an HTTP request and sends the image data and emotion data to the server.

[1247] Step 4:

[1248] The server adds the received image data to the analysis queue and invokes the image analysis module. The input is the image data stored on the server, and the output is the image data ready for analysis. Specifically, the server adds the image data to the queue and schedules it for analysis in the next stage.

[1249] Step 5:

[1250] The server uses the OpenCV library to perform initial preprocessing such as image resizing, noise removal, and color adjustment. The input is image data ready for analysis, and the output is preprocessed image data. Specifically, it calls OpenCV functions to perform processing to improve the image quality.

[1251] Step 6:

[1252] The server uses a machine learning model (TensorFlow or PyTorch) to analyze the image and identify the food items. The input is preprocessed image data, and the output is the identified food items and their labels. Specifically, the model inputs the image and performs inference to obtain the identification results.

[1253] Step 7:

[1254] Based on the identification results, the server sends a query to a nutrition database to obtain detailed nutrition information for each food item. The input is the food item label, and the output is the nutrition information. The specific operation is to generate a database query to obtain the required nutrition data.

[1255] Step 8:

[1256] Based on the nutritional information acquired by the server, a nutritional balance chart is generated by comparing it with the recommended daily intake. The input is detailed nutritional information, and the output is the nutritional balance chart data. Specifically, the server runs an algorithm that compares the data and generates the chart.

[1257] Step 9:

[1258] The chart data generated by the server is sent to the terminal in JSON format. The input is the nutritional balance chart data, and the output is the chart data received by the terminal. Specifically, the chart data is sent as an HTTP response.

[1259] Step 10:

[1260] The terminal uses the visualization library to draw charts and display them visually to the user. The input is the received chart data, and the output is the chart displayed to the user. Specifically, it calls the library's drawing functions to display the chart on the screen.

[1261] Step 11:

[1262] The user enters health goals and chronic disease information in the app and sends it to the server. The input is the health goals and chronic disease information entered by the user, and the output is the health information stored on the server. Specifically, the app provides an input form and sends the input data to the server.

[1263] Step 12:

[1264] The server generates meal suggestions based on the user's health goals, chronic illness information, and emotional state, and sends them to the user. The input is user profile data and emotional data, and the output is customized meal suggestions. Specifically, the server runs a suggestion generation algorithm to suggest appropriate meal menus.

[1265] Step 13:

[1266] The user uses a budget input form to enter the desired budget amount and send it to the server. The input is the budget information entered by the user, and the output is the budget information saved on the server. Specifically, the app provides an input form and sends the budget data to the server.

[1267] Step 14:

[1268] The server queries the product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within a budget. The input is budget information and food identification data, and the output is a list of foods that can be purchased. Specifically, the server generates a database query and retrieves the appropriate food list.

[1269] Step 15:

[1270] The server filters the food items based on budget, nutritional balance, and emotional state, and sends the selected food list to the device. The input is the available food list and user profile data, and the output is the filtered food list. Specifically, the server runs the filtering algorithm to generate an appropriate list.

[1271] Step 16:

[1272] The terminal displays the filtered food list to the user and allows them to purchase the food through the online shopping site. The input is the filtered food list, and the output is the displayed list and a purchase link. The specific operation is to display the list on the screen and provide a purchase link.

[1273] Step 17:

[1274] The user enables the "Link with Medical Institutions" option in the app settings. The input is the user's setting selection, and the output is the setting change. Specifically, the user turns on the option on the app settings screen.

[1275] Step 18:

[1276] The terminal transmits the nutritional balance data and emotion data to the server with the user's permission. The input is the data with the user's permission, and the output is the nutritional balance data and emotion data stored on the server. Specifically, the terminal transmits the data to the server after confirming the user's permission.

[1277] Step 19:

[1278] The server takes appropriate security measures and then shares user data through an API dedicated to medical institutions. The input is the saved user data, and the output is the data sent to the medical institution. Specifically, data is shared through a secure API.

[1279] The above are the specific processing steps of the system that realizes the application example. Each step is important for realizing the function of assisting the user in managing their health.

[1280] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1281] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1282] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1283] [Fourth embodiment]

[1284] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1285] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1286] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1287] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1288] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1289] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1290] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1291] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1292] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1293] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1294] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1295] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1296] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1297] This invention is a system that allows users to take pictures of foods, chart their nutritional balance based on those pictures, and make dietary suggestions based on individual health goals. This system includes a means for users to take and upload food images, a means for a server to analyze the received images and identify the foods, a means for the server to obtain nutritional information for the identified foods and chart their nutritional balance, a means for displaying the nutritional balance chart to the user via a terminal, a means for the user to input information about chronic illnesses and health goals and for the server to make individually appropriate dietary suggestions, a means for supporting budget-based food purchasing at convenience stores and supermarkets, and a means for linking nutritional balance information with medical institutions.

[1298] Program processing

[1299] Image upload

[1300] A user opens the application and takes a photo of a food. The image is saved on the device and an upload button is displayed. When the user presses the upload button, the image is sent to the server.

[1301] Image analysis

[1302] The server adds the received image to the analysis queue. It invokes the image analysis module, which first preprocesses the image (e.g., resizes the image, removes noise, etc.). The server then uses a machine learning model to recognize food in the image. Specifically, it segments the image region, identifies each food item, and labels each one. At this stage, the type and quantity of food are estimated.

[1303] Acquiring food data

[1304] The server queries the food database based on the label of each identified food, and retrieves detailed nutritional information (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) for each food in the database.

[1305] Nutritional balance chart

[1306] The server aggregates the acquired nutritional information and creates a chart of nutritional balance based on the generated information. The nutritional balance chart is displayed in a visually easy-to-understand format (radar chart, bar graph, etc.).

[1307] Displaying the results

[1308] The server sends the chart data in JSON format to the device, which then visually displays the chart to the user using a drawing library. The user can refer to this chart and understand their diet at a glance.

[1309] Customized Offers

[1310] Users enter their health goals and chronic illness information within the app. The device sends this information to the server, which uses this information to update the user's profile and provide personalized dietary suggestions. These suggestions are then sent to the device and displayed to the user.

[1311] Food suggestions according to your budget

[1312] The user enters their desired budget amount using a budget input form. The device sends the budget information to the server. The server queries the product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within the budget. The server filters the list based on budget and nutritional balance and sends the selected list to the device. The device displays this list to the user.

[1313] Collaboration with medical institutions

[1314] The user enables the "Link with medical institutions" option in the app settings. The device sends nutritional balance data to the server with the user's permission. The server takes security measures and shares the user data through an API dedicated to medical institutions. Medical institutions use this data to provide the user with appropriate feedback and treatment plans.

[1315] Specific examples

[1316] A user eats a hamburger steak set meal (rice, hamburger steak, salad) for lunch and uploads a photo of it to the app. The device sends the captured image to the server, which analyzes the image and identifies each food item. The server obtains nutritional information based on the identification results and creates a chart of nutritional balance. The chart is then sent to the device and the user can visually confirm it. Furthermore, if the user has a diet goal, they input that goal into the app. The server takes this into consideration and suggests a low-calorie dinner menu (e.g., a recipe for salad and grilled chicken). If a user sets a budget of 1,000 yen for the next day and wants to buy a healthy lunch, the server will suggest convenience store items (e.g., sandwiches, salad, and tea) that can be purchased within that budget. Users can maintain a healthy diet by referring to the ingredient list when making purchases. Users can also share their nutritional balance information with medical institutions and engage in regular health management.

[1317] In this way, the system allows users to easily and efficiently manage their daily diet and receive appropriate support according to their individual health conditions.

[1318] The processing flow will be explained below.

[1319] Step 1:

[1320] A user opens the application and takes a photo of the food. After taking the photo, the user presses the "upload" button.

[1321] Step 2:

[1322] The terminal stores the captured image file in a temporary storage area within the application, and transmits the image data to the server.

[1323] Step 3:

[1324] The server adds the received image data to the analysis queue and invokes the image analysis module. As initial preprocessing, it performs image resizing, noise removal, color adjustment, etc.

[1325] Step 4:

[1326] The server uses machine learning models to analyze the image, identify food item regions, label each item, and estimate the quantity of each food item.

[1327] Step 5:

[1328] The server queries the food database based on the label of each recognized food item, and obtains detailed nutritional information for each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.).

[1329] Step 6:

[1330] The server compares the collected nutritional information with the user's recommended daily intake and generates a nutritional balance chart in a visually easy-to-understand format such as a radar chart or bar graph.

[1331] Step 7:

[1332] The server sends the generated chart data in JSON format to the terminal.

[1333] Step 8:

[1334] The device uses a drawing library to display the chart data received to the user, who can then check the chart to understand their own nutritional balance.

[1335] Step 9:

[1336] Users enter and update their health goals and chronic illness information within the app.

[1337] Step 10:

[1338] The terminal transmits the input information to the server.

[1339] Step 11:

[1340] The server updates the user's profile based on their health goals and chronic illnesses, and generates suitable meal suggestions, including recipes and menus for dining out.

[1341] Step 12:

[1342] The server sends meal suggestions to the terminal.

[1343] Step 13:

[1344] The device displays suggested recipes and meal menus to the user, who can then use them to select a meal.

[1345] Step 14:

[1346] The user inputs the budget for purchasing ingredients, for example, "1000 yen."

[1347] Step 15:

[1348] The terminal transmits the budget information to the server.

[1349] Step 16:

[1350] The server queries product databases of convenience stores and supermarkets to generate a list of foods that can be purchased within a budget. The list is filtered to take nutritional balance into consideration.

[1351] Step 17:

[1352] The server transmits the generated food list to the terminal.

[1353] Step 18:

[1354] The terminal displays a food list to the user, and the user uses the list as a reference when purchasing ingredients.

[1355] Step 19:

[1356] The user enables the "Connect with healthcare providers" option in the app settings.

[1357] Step 20:

[1358] The terminal confirms the user's permission and transmits the permission information along with the nutritional balance data to the server.

[1359] Step 21:

[1360] The server takes appropriate security measures and shares user data through an API dedicated to medical institutions.

[1361] Step 22:

[1362] Medical institutions receive the shared data and provide appropriate feedback and treatment plans to users.

[1363] Example 1

[1364] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1365] Currently, there are many dietary management tools and applications on the market, but most of them have limitations in their ability to effectively understand the user's dietary content and nutritional balance and make personalized suggestions based on that information. They also lack the functionality to suggest ingredients based on the user's budget or to support health management in collaboration with medical institutions. There is a need for a system that can improve these points and enable users to manage their health more effectively.

[1366] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1367] In this invention, the server includes means for preprocessing the received images and identifying foods using a machine learning model, means for acquiring nutritional information for the identified foods and charting their nutritional balance, and means for linking the nutritional balance information with medical institutions. This enables users to easily manage their diet through food images, receive personalized dietary suggestions, and even manage their health consistently through collaboration with medical institutions.

[1368] "User" refers to an individual who uses the system to take pictures of food and manage their diet.

[1369] "Server" refers to a central processing unit for receiving and analyzing data sent by users and providing necessary information.

[1370] "Terminal" refers to a computing device (such as a smartphone, tablet, or PC) used by a user.

[1371] "Food image" refers to photographic data containing food that a user takes and uploads to the system.

[1372] "Preprocessing" refers to the initial processing such as resizing and noise removal that is performed before the server analyzes the image.

[1373] A "machine learning model" refers to an algorithm that learns patterns from data and identifies and classifies foods.

[1374] A "food database" refers to a collection of data that stores nutritional information about each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.).

[1375] "Charting nutritional balance" refers to presenting collected nutritional information in a visually easy-to-understand format (such as a radar chart or bar graph).

[1376] "Health goal" refers to a goal related to a user's health status (e.g., dieting, muscle building, managing a chronic illness, etc.).

[1377] "Meal suggestions" refers to recommendations for providing an individualized meal plan based on the user's health goals and chronic illness information.

[1378] "Support for food purchasing" refers to providing users with a list of foods that can be purchased within their budget, and assisting them in their purchasing activities at convenience stores and supermarkets.

[1379] "Collaboration with medical institutions" refers to a system in which users' nutritional balance information is shared with medical institutions and feedback and treatment plans are provided as needed.

[1380] "Drawing library" refers to a software component for visually displaying data (e.g., D3.js, Chart.js, Matplotlib, etc.).

[1381] The present invention is a system that allows a user to take images of food, chart the nutritional balance based on the images, and make dietary suggestions based on individual health goals. This system includes means for the user to take and upload images of food, means for a server to analyze the received images and identify the food, means for the server to obtain nutritional information for the identified foods and chart the nutritional balance, means for displaying the nutritional balance chart to the user via a terminal, means for the user to input health goals and chronic illness information and for the server to make individually appropriate dietary suggestions, means for supporting food purchasing according to budget, means for linking nutritional balance information with medical institutions, and means for pre-processing the captured images by resizing and removing noise.

[1382] Hardware and software used

[1383] To realize this system, the following hardware and software are used.

[1384] 1. Hardware

[1385] User device: smartphone, tablet, or computer.

[1386] Server: Server equipment equipped with a high-performance CPU and GPU.

[1387] 2. Software

[1388] Image analysis module: Uses Python and utilizes libraries such as OpenCV and TensorFlow.

[1389] Database: A relational database such as MySQL or PostgreSQL.

[1390] Drawing libraries: Visualization libraries such as D3.js, Chart.js, Matplotlib, etc.

[1391] API: Implements a RESTful API to communicate between the server and the terminal.

[1392] Example of operation

[1393] A specific example of the operation of the system of the present invention will be described below.

[1394] Specific examples

[1395] A user eats a hamburger steak set meal (rice, hamburger steak, salad) for lunch and uploads a photo of it to the app. The user takes a photo of the food using the app's camera function and presses the upload button, which sends the image to the server. The server preprocesses the received image (for example, resizes the image to 256x256 pixels and removes noise) and begins analysis using a machine learning model using TensorFlow. The server analyzes the image, identifies the foods - hamburger steak, rice, and salad - and labels each food.

[1396] The server then queries a MySQL database based on the identified food labels to obtain detailed nutritional information for each food (e.g., calories, protein, fat, carbohydrates, vitamins, and minerals). The obtained nutritional information is aggregated and a nutritional balance chart is created. The server sends this chart information in JSON format to the device, which then visually displays the chart using a drawing library such as D3.js. Users can refer to the chart to understand their diet at a glance.

[1397] Furthermore, when a user sets "diet" as a "health goal" in the app, the server will suggest corresponding low-calorie menus (e.g., salad and grilled chicken recipes). If a user sets a budget of 1,000 yen and wants to buy a healthy lunch, the server will generate a list of foods that can be purchased within that budget (e.g., sandwiches, salads, and tea) and send it to the device. By referring to the list of ingredients when making purchases, the user can maintain a healthy diet.

[1398] Prompt Sentence Examples

[1399] Example inputs to a generative AI model:

[1400] "Write a program that analyzes an image of a person eating oatmeal, banana, and yogurt for breakfast, along with the detailed ingredients, and then charts and displays the nutritional balance of this meal."

[1401] This system allows users to easily and efficiently manage their daily diet and receive appropriate support according to their individual health conditions.

[1402] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1403] Step 1:

[1404] The user takes a picture of a food item and presses the upload button in the application. The device saves the image to its internal storage and sends an upload request to the server. The input data is the food image taken by the user, and the output is the image sent to the server.

[1405] Specific behavior:

[1406] A user takes a photo of a hamburger steak set meal using the camera on their smartphone.

[1407] The device saves the image file (e.g., lunch.jpg) to its internal storage.

[1408] The user taps the upload button in the app and the image is sent to the server.

[1409] Step 2:

[1410] The server receives the uploaded images and adds them to the analysis queue. The server uses image analysis modules to preprocess the images and convert them into an analyzable format. The input data is the original uploaded image, and the output is the preprocessed image data.

[1411] Specific behavior:

[1412] The server resizes the image to 256x256 pixels and denoises it.

[1413] The preprocessed image data is fed into a machine learning model.

[1414] Step 3:

[1415] The server uses a machine learning model (e.g., TensorFlow) to analyze the preprocessed images and identify the foods. It identifies each food item in the image and assigns a label to each. The input data is the preprocessed image data, and the output is the label and estimated quantity of the identified food.

[1416] Specific behavior:

[1417] The server uses the YOLO model to detect food items (e.g., hamburger, rice, salad) in the image.

[1418] The server labels each food item and estimates the quantity (e.g., "hamburger": 1 piece, "rice": 1 bowl, "salad": 1 plate).

[1419] Step 4:

[1420] The server queries the food database based on the label of each identified food and retrieves detailed nutritional information for each food (calories, protein, fat, carbohydrates, vitamins, minerals, etc.). The input data is the label of the identified food, and the output is the retrieved nutritional information.

[1421] Specific behavior:

[1422] The server uses a SELECT query to retrieve nutritional information from the MySQL database based on the label "hamburger."

[1423] Similarly, obtain nutritional information for each food item (e.g., "Calories": 350kcal, "Protein": 25g).

[1424] Step 5:

[1425] The server aggregates the acquired nutritional information and generates chart data to visually represent nutritional balance. The input data is detailed nutritional information for each food, and the output is chart data.

[1426] Specific behavior:

[1427] A server aggregates the nutritional information of all foods.

[1428] The server uses the Matplotlib library to plot the nutritional balance as a radar chart.

[1429] Step 6:

[1430] The server generates chart data and sends it to the terminal in JSON format. The terminal uses a drawing library (e.g., D3.js) to display the chart to the user based on the data received. The input data is the chart data, and the output is a visual chart that is displayed to the user.

[1431] Specific behavior:

[1432] The server converts the chart data into JSON format and sends it to the terminal.

[1433] The device uses Chart.js to draw the nutritional balance in the form of a bar graph and displays it on the app's UI.

[1434] Step 7:

[1435] The user enters their health goals and chronic illness information into the app. The device sends this information to the server, which then makes personalized dietary suggestions. The input data is the user's health goals and chronic illness information, and the output is dietary suggestions.

[1436] Specific behavior:

[1437] The user enters "health goal" as "diet."

[1438] The server will then suggest a low-calorie option (e.g., salad and grilled chicken) based on that information.

[1439] Step 8:

[1440] The user uses a budget input form to input the desired budget amount. The device sends the budget information to the server, which then executes a query to generate a list of foods that can be purchased within the budget. The input data is the user's budget information, and the output is a list of foods that can be purchased within the budget.

[1441] Specific behavior:

[1442] The user sets the "budget" to "1,000 yen" within the app.

[1443] The server generates a list of foods that can be purchased for under 1,000 yen, suggesting sandwiches, salads, and tea.

[1444] Step 9:

[1445] The user enables the "Link with medical institutions" option in the app settings. Based on the user's permission, the device sends nutritional balance data to the server, and the server shares the user data through an API dedicated to medical institutions. The input data is the user's nutritional balance data and medical institution information, and the output is data shared with the medical institution.

[1446] Specific behavior:

[1447] The user enables the "Connect with Healthcare Providers" option.

[1448] The server encrypts the data and sends it to an HTTPS endpoint dedicated to the medical institution.

[1449] (Application example 1)

[1450] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1451] In modern society, there is a demand for appropriate nutritional management and dietary suggestions tailored to each individual's health condition. However, daily dietary management is complicated, and it is particularly difficult to confirm the nutritional balance of food when eating out or purchasing it. Furthermore, when purchasing at convenience stores or supermarkets, it is not easy to select ingredients that take nutritional balance and budget into consideration. Furthermore, there is a demand for more appropriate health management by linking nutritional information with medical institutions, but current systems are not able to adequately address this demand.

[1452] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1453] In this invention, the server includes means for a user to photograph and upload images of foods, means for the server to analyze the received images and identify the foods, means for the server to obtain nutritional information for the identified foods and create a nutritional balance chart, means for the server to display the nutritional balance chart to the user via a terminal, means for the user to input chronic illness information and health goals and for the server to make individually appropriate meal suggestions, means for supporting budget-based food purchasing at sales facilities, means for linking the nutritional balance information with medical institutions, means for analyzing photographed images of food shelves and obtaining nutritional information for each food item to display to the user, and means for suggesting foods that can be purchased within the user's budget based on the user's input data. This allows users to check nutritional information on the spot when purchasing food at a physical store, make healthy food choices within their budget, and even manage their health in cooperation with medical institutions.

[1454] "User" refers to an individual who uses the system to take pictures of food and receive nutritional balance and meal suggestions.

[1455] "Server" refers to a central computer system for analyzing food images, obtaining nutritional information, and displaying it to the user.

[1456] The "means for taking and uploading images of food" refers to the part that provides the function for users to take images of food using a smartphone or smart glasses and send those images to the system.

[1457] "Means for identifying food" refers to technology that analyzes the image received by the server and identifies the food in the image.

[1458] "Means for obtaining nutritional information and charting nutritional balance" refers to the function of collecting nutritional data for identified foods and displaying it in a visually easy-to-understand format.

[1459] "Means for displaying a nutritional balance chart to a user on a terminal" refers to a function that allows a user to see a nutritional balance chart on a terminal such as a smartphone or computer.

[1460] "A means for inputting health goals and for the server to make individually appropriate meal suggestions" refers to a function in which the user inputs information about chronic illnesses and health goals, and the server creates customized meal suggestions based on that information.

[1461] "Means to support food purchasing according to budget at sales facilities" refers to a function that suggests foods that can be purchased taking into account the user's budget and supports the purchase.

[1462] "Means for linking nutritional balance information with medical institutions" refers to a function for sharing a user's nutritional information with medical institutions and supporting health management.

[1463] "Means for analyzing images of food shelves, obtaining nutritional information for each food item, and displaying this information to the user" refers to a function that analyzes images of food shelves taken at a sales facility, obtains nutritional information for each food item, and provides this information to the user.

[1464] "Means for suggesting foods that can be purchased within a budget based on input data" refers to a function that creates and suggests a list of foods that can be purchased within a budget based on the user's input data (budget and health goals).

[1465] This invention is a system that allows a user to take an image of food, charts the nutritional balance based on the image, and makes meal suggestions based on individual health goals. The system includes the following means:

[1466] 1. Image uploading method: Users take pictures of the food shelves using their smartphones or smart glasses and upload them to the server from their devices. The uploaded images are stored on the server.

[1467] 2. Image analysis: The server analyzes the received images and identifies the food in the image. This image analysis is performed using a machine learning model (e.g., using TensorFlow). The server first preprocesses the images (resizes, removes noise, etc.), then identifies and labels the food.

[1468] 3. Nutritional information acquisition means: Based on the food identified from the analyzed image, the server acquires the detailed nutritional information of the food from a food database (e.g., an internally stored nutritional database), including information such as calories, protein, fat, carbohydrates, vitamins, and minerals.

[1469] 4. Nutritional balance charting: The server aggregates the nutritional balance based on the acquired nutritional information and displays it in a visually easy-to-understand format (for example, a radar chart or bar graph). This allows users to understand the nutritional balance of the foods they have consumed at a glance.

[1470] 5. Display of results: The charted nutritional balance is sent from the server to the device, which then uses a visualization library to render it and display it to the user, allowing the user to understand their own dietary habits.

[1471] 6. Customized Meal Suggestion: Users can input their health goals and chronic illness information within the application. This information is sent to the server, which updates the user's profile and provides personalized meal suggestions. These suggestions are sent to the device, providing the user with guidelines for maintaining a healthy diet.

[1472] 7. Budget-based food recommendation: The user inputs their desired budget amount into the application. The server connects to the database of sales facilities to generate a list of foods that can be purchased within the budget. It filters based on budget and nutritional balance and sends the selected food list to the terminal. The terminal displays this list to the user, who can receive assistance in purchasing healthy foods within their budget.

[1473] 8. Linking with medical institutions: When the user enables the "Linking with medical institutions" option, the device will send nutritional balance data to the server with the user's permission. The server will then share the user data through an API dedicated to medical institutions, after taking security measures. Medical institutions will use this data to provide the user with appropriate feedback and treatment plans.

[1474] For example, when a user takes a photo of a food shelf in a physical store and uploads it, the server analyzes the image, obtains nutritional information for each food item, and displays it to the user. Based on this information, the user can select foods that take into account nutritional balance and budget on the spot. Furthermore, by inputting the user's health goals and budget, the server can make optimal meal suggestions.

[1475] Additionally, examples of prompts that are useful as input to generative AI models include:

[1476] "A user takes a photo of a food shelf in a physical store and uploads it. Please explain how the system uses the image to create a nutritional balance chart and provide meal suggestions aligned with the user's health goals. Specifically, please explain in detail the process of image upload, image analysis, nutritional information acquisition, and meal suggestions. Please also include suggestions based on a budget."

[1477] In this way, the system allows users to easily and efficiently manage their daily diet and receive appropriate support according to their individual health conditions.

[1478] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1479] Step 1: Upload an image

[1480] A user takes a photo of the food shelf with a smartphone or smart glasses. The device saves the image and displays an upload button. When the user presses the upload button, the image is sent from the device to the server. The input is the image taken by the user, and the output is the image data sent to the server.

[1481] Step 2: Image analysis

[1482] The server adds the received image to the analysis queue. It invokes the image analysis module to first preprocess the image (resize, remove noise, etc.). The server then uses a generative AI model to recognize food in the image. This process involves segmenting the image area, identifying each food item, and labeling each one. The input is the image data sent to the server, and the output is the label information of the identified food and its location information.

[1483] Step 3: Obtain nutritional information

[1484] The server queries an internal food database based on the label of each identified food, and obtains detailed nutritional information (calories, protein, fat, carbohydrates, vitamins, minerals, etc.) for each food in the database. The input is the label information of the identified food, and the output is the nutritional information.

[1485] Step 4: Nutritional Balance Chart

[1486] The server aggregates the acquired nutritional information and charts the nutritional balance in a visually easy-to-understand format (such as a radar chart or bar graph). The chart data is generated in JSON format. The input is the aggregated nutritional information, and the output is the JSON data of the nutritional balance chart.

[1487] Step 5: View the results

[1488] The server sends the chart data to the terminal, which uses a visualization library to draw the chart and display it visually to the user. The input is the JSON data of the chart, and the output is the nutritional balance chart displayed to the user.

[1489] Step 6: Customized Meal Suggestions

[1490] The user enters their health goals and chronic illness information within the application. The device sends this information to the server. The server updates the user's profile and generates personalized meal suggestions. The suggestions are sent to the device and displayed to the user. The input is the user's health goals and chronic illness information, and the output is personalized meal suggestions.

[1491] Step 7: Proposing ingredients according to your budget

[1492] The user enters their budget in the app. The device sends the budget information to the server. The server queries a database of sales establishments to generate a list of foods that can be purchased within the budget. After filtering, the server sends the selected list of ingredients to the device. The device displays this list to the user. The input is the user's budget information, and the output is a list of foods that the user can purchase.

[1493] Step 8: Collaborate with medical institutions

[1494] The user enables the "Link with medical institutions" option in the app settings. The device sends nutritional balance data to the server with the user's permission. The server takes security measures and shares the user data through an API dedicated to medical institutions. Based on this data, medical institutions provide the user with appropriate feedback and treatment plans. The input is the user's nutritional balance data, and the output is the data sent to the medical institution.

[1495] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1496] The present invention is a system that allows a user to take an image of food, charts the nutritional balance based on the image, and makes meal suggestions based on individual health goals and emotional state. This system includes means for a user to take and upload an image of food, means for a server to analyze the received image and identify the food, means for the server to obtain nutritional information for the identified food and chart the nutritional balance, means for displaying the nutritional balance chart to the user via a terminal, means for the user to input information about chronic illnesses and health goals and for the server to make individually appropriate meal suggestions, means for supporting budget-based food purchasing at convenience stores and supermarkets, means for linking nutritional balance information with medical institutions, and means for recognizing the user's emotions using an emotion engine.

[1497] Program processing

[1498] Image upload and emotion recognition

[1499] The user opens the application and takes a photo of the food. After taking the photo, the device's emotion recognition module analyzes the user's face and estimates their current emotion. When the user presses the "Upload" button, the image and emotion data are sent to the server.

[1500] Image analysis

[1501] The server adds the received image data to an analysis queue and invokes the image analysis module. Initial preprocessing includes image resizing, noise reduction, and color adjustment. The server then analyzes the image using a machine learning model to identify and label each food item. The quantity of each food item is also estimated.

[1502] Emotional Data Processing

[1503] The server analyzes the received emotional data to determine the user's emotional state, which is used as input data for dynamically adjusting the user's meal suggestions.

[1504] Acquiring food data

[1505] The server queries the food database based on the label of each identified food item, and retrieves detailed nutritional information (e.g., calories, protein, fat, carbohydrates, vitamins, minerals) for each food item in the database.

[1506] Nutritional balance chart

[1507] The server compares the nutritional information it obtains with the user's recommended daily intake and generates a nutritional balance chart in a visually easy-to-understand format, such as a radar chart or bar graph.

[1508] Displaying the results

[1509] The server sends the generated chart data in JSON format to the device. The device displays the chart to the user using a drawing library. The user can check the chart and understand their own nutritional balance.

[1510] Customized Offers

[1511] Users input and update their health goals and chronic illness information within the app. The device then sends this information and emotional data to the server. The server then updates the user profile based on the user's health goals, chronic illness information, and emotional state, and generates appropriate meal suggestions. These suggestions include recipes and menus for dining out.

[1512] Food suggestions according to your budget

[1513] The user enters their desired budget amount using a budget input form. The device sends the budget information to the server. The server queries convenience store and supermarket product databases to generate a list of foods that can be purchased within the budget. The server filters the list based on budget, nutritional balance, and emotional state, and sends the selected list to the device. The device displays this list to the user.

[1514] Collaboration with medical institutions

[1515] The user enables the "Link with Medical Institutions" option in the app settings. With the user's permission, the device sends nutritional balance data and emotional data to the server. The server takes appropriate security measures and shares the user data through an API dedicated to medical institutions. Medical institutions use this data to provide the user with appropriate feedback and treatment plans.

[1516] Specific examples

[1517] A user eats a hamburger steak set meal (rice, hamburger steak, salad) for lunch and uploads a photo of it to the app. The device sends the captured image to the server, which analyzes the image and identifies each food item. The server obtains nutritional information based on the identification results and creates a chart of the nutritional balance. The chart is then sent to the device and the user can visually check it.

[1518] Furthermore, if a user has a goal of losing weight and inputs that goal into the app, if the emotion engine detects that the user is feeling low, the server will take that into account and suggest a high-protein, low-calorie dinner menu (e.g., a recipe for salad and grilled chicken) to help them regain their energy.

[1519] If a user sets a budget of 1,000 yen for the next day and wants to buy a healthy lunch, the server will suggest convenience store items (e.g., sandwiches, salads, tea) that can be purchased within that budget. The user can purchase ingredients based on this list and maintain a healthy diet.

[1520] Users can also share their nutritional balance information and emotional data with medical institutions and receive regular health management advice from doctors. This system allows users to efficiently manage their daily diet and receive appropriate support tailored to their individual health and emotional states.

[1521] The processing flow will be explained below.

[1522] Step 1:

[1523] The user opens the application and takes a photo of the food. After taking the photo, the device's emotion engine analyzes the user's face in real time and estimates their current emotional state. The user then presses the "Upload" button.

[1524] Step 2:

[1525] The terminal stores the captured image file and emotion data in a temporary storage area within the application, and transmits the data to the server.

[1526] Step 3:

[1527] The server adds the received image data to the analysis queue and invokes the image analysis module. As initial preprocessing, it performs image resizing, noise removal, color adjustment, etc.

[1528] Step 4:

[1529] The server uses machine learning models to analyze the image, identify food item regions, label each item, and estimate the quantity of each food item.

[1530] Step 5:

[1531] The server queries the food database based on the label of each recognized food item, and obtains detailed nutritional information for each food item (calories, protein, fat, carbohydrates, vitamins, minerals, etc.).

[1532] Step 6:

[1533] The server compares the collected nutritional information with the user's recommended daily intake and generates a nutritional balance chart in a visually easy-to-understand format such as a radar chart or bar graph.

[1534] Step 7:

[1535] The server analyzes the emotional data to determine the user's emotional state, which is used as input data for dynamically adjusting the user's meal suggestions.

[1536] Step 8:

[1537] The server sends the generated chart data and the user's emotional state in JSON format to the terminal.

[1538] Step 9:

[1539] The device uses a drawing library to display the chart data and emotion data received to the user, who can then check the chart to understand their own nutritional balance and emotional state.

[1540] Step 10:

[1541] Users enter and update their health goals and chronic illness information within the app, and the device sends this information and emotional data to the server.

[1542] Step 11:

[1543] The server updates the user's profile based on the user's health goals, chronic illness information, and emotional data, and generates suitable meal suggestions, including recipes and menus for dining out.

[1544] Step 12:

[1545] The server sends meal suggestions to the terminal.

[1546] Step 13:

[1547] The device displays suggested recipes and meal menus to the user, who then uses them to select a meal.

[1548] Step 14:

[1549] The user inputs the budget for purchasing ingredients, for example, "1000 yen."

[1550] Step 15:

[1551] The terminal transmits the budget information to the server.

[1552] Step 16:

[1553] The server queries convenience store and supermarket product databases to generate a list of food items that can be purchased within a budget. The food list is filtered based on nutritional balance and emotional state.

[1554] Step 17:

[1555] The server transmits the generated food list to the terminal.

[1556] Step 18:

[1557] The terminal displays a food list to the user, and the user uses the list as a reference when purchasing ingredients.

[1558] Step 19:

[1559] The user enables the "Connect with healthcare providers" option in the app settings.

[1560] Step 20:

[1561] The terminal confirms the user's permission and transmits the permission information along with the nutritional balance data and emotion data to the server.

[1562] Step 21:

[1563] The server takes appropriate security measures and shares user data through an API dedicated to medical institutions.

[1564] Step 22:

[1565] Medical institutions receive the shared data and provide appropriate feedback and treatment plans to users.

[1566] Example 2

[1567] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1568] Conventional dietary management systems make it difficult for users to understand the nutritional balance of foods and receive dietary suggestions tailored to their health goals. They also lack the ability to provide dietary suggestions based on emotional state or budget, and lack the ability to connect with medical institutions, making comprehensive health management difficult.

[1569] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: [means for an emotion recognition module of the terminal to analyze the user's face and estimate the emotion;] [means for the server to add received images to an analysis queue, and for the image analysis module to preprocess the images and identify foods using a machine learning model;] [means for the server to obtain nutritional information of the identified foods from a database and chart their nutritional balance;] [means for the server to select and provide foods according to a budget entered by the user; and] [means for linking nutritional balance information and emotion data with a medical institution. This allows the user's food selection to be optimized according to their emotional state and budget, and also enables health management in collaboration with a medical institution.

[1570] A "user" is someone who uses the system to take and upload images of food and receive nutritional balance and meal suggestions.

[1571] "Terminal" refers to a device operated by a user, such as a smartphone or tablet, and includes an emotion recognition module and a chart drawing library.

[1572] The "server" is the central computer of the system, and is a device that performs processes such as image analysis, database queries, meal suggestion generation, and medical institution collaboration.

[1573] An "emotion recognition module" is software or hardware built into a device that has the function of analyzing a user's face and estimating their emotional state.

[1574] "Image Analysis Module" means software within the server that pre-processes received images and uses machine learning models to identify food products.

[1575] A "machine learning model" is an algorithm that makes predictions and classifications based on data, and is a type of artificial intelligence used in image analysis modules.

[1576] A "food database" is a data store containing detailed nutritional information for various foods, to which a server can send queries to retrieve the required data.

[1577] A "nutritional balance chart" is a diagram that visually shows the user's nutritional intake status, and is displayed in the form of a radar chart or bar graph.

[1578] A "visualization library" is a software library for drawing charts and graphs, used on a device to visually display data in JSON format.

[1579] "Health goals" are goals that users set within the app based on their individual health status and lifestyle goals, and are taken into consideration when making meal suggestions.

[1580] The "budget" is a price range set by the user when purchasing ingredients, and is a guideline used to generate the food list provided by the server.

[1581] "Medical institution" refers to a hospital, clinic, medical office, etc., and is a facility that can share users' nutritional balance information and emotion data by linking with a server.

[1582] "Meal suggestions" refers to suggestions such as recipes and dining out menus generated by the server based on the user's health goals, emotional state, and budget.

[1583] The present invention is a system that allows users to take pictures of food, charts the nutritional balance based on the pictures, and suggests meals based on individual health goals and emotional state. Specific methods for implementing this system are described below.

[1584] The system includes devices such as smartphones and tablets, a server, a food database, and related software modules, including emotion recognition modules, image analysis modules, machine learning models (e.g., TensorFlow, PyTorch), and visualization libraries (e.g., Chart.js, D3.js).

[1585] First, the user opens the application and takes a picture of the food using the device's camera. An emotion recognition module (e.g., Face++ API) on the device analyzes the user's face and estimates their current emotion. This emotion data is later used to customize meal suggestions. After taking the picture, the user presses the "Upload" button, and the image and emotion data are sent to the server.

[1586] The server then adds the received image data to an analysis queue, where the image analysis module operates. Image preprocessing involves resizing, noise reduction, and color adjustment. Furthermore, a machine learning model (e.g., a trained CNN model) is used to analyze the image, identify food items, and label each one. The quantity of each food item is also estimated.

[1587] The server queries a food database (e.g., USDA Food Database) via its API to obtain detailed nutritional information (e.g., calories, protein, fat, carbohydrates, vitamins, minerals) for each identified food item. The server then generates a nutritional balance chart based on the nutritional information obtained and compares it with the user's recommended daily intake. This chart is created in a visually easy-to-understand format, such as a radar chart or bar graph, and sent to the device in JSON format.

[1588] The device uses a chart drawing library (e.g., Chart.js, D3.js) to display a nutritional balance chart to the user, allowing the user to visually understand their own nutritional balance.

[1589] Next, the user enters their health goals and chronic illness information into the app, and the data is sent from the device to the server. The server uses this information to update the user profile and generate meal suggestions that take into account the user's emotional data and health goals. For example, high-protein, low-calorie recipes or restaurant menus may be suggested.

[1590] Furthermore, when the user enters their desired budg...

Claims

1. a means for a user to take and upload an image of a food item; means for analyzing the image received by the server and identifying the food; a means for the server to obtain nutritional information of the identified food and chart the nutritional balance; means for displaying a nutritional balance chart to a user via a terminal; A means for users to input information about their chronic illnesses and health goals, and for the server to make personalized dietary suggestions; A means to support food purchases at convenience stores and supermarkets according to budget, A system that includes a means of linking nutritional balance information with medical institutions.

2. 10. The system of claim 1, wherein the server includes means for obtaining nutritional information from a food database and aggregating the collected information.

3. The system of claim 1 , wherein the terminal includes means for rendering the chart using a visualization library and visually displaying it to the user.

Images