System

Abstract

A system is provided.SOLUTION: A system comprising: means for acquiring an image of a meal; means for analyzing the acquired image and identifying a type of each food in the image; means for acquiring a nutritional value of the identified food from a database and calculating a total nutritional value; means for graphically displaying the calculated nutritional value; and means for identifying a missing nutrient compared to a daily nutrient requirement and suggesting a food to be additionally consumed.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] While consuming a balanced diet containing essential nutrients is important for maintaining good health, accurately understanding and managing the nutritional value of one's own meals is both tedious and difficult. Pregnant and dieting users, in particular, need to pay special attention to the nutrients they should consume, creating a demand for easy and effective nutritional management methods. Furthermore, effectively utilizing the ingredients in the refrigerator and planning balanced meals is also a difficult task. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system that acquires images of dishes, analyzes them to identify the type of food, and calculates and displays the nutritional value of each food. Specifically, the system recognizes the type of food from the acquired image, retrieves the corresponding nutritional value information from a database, calculates the total nutritional value, and displays the results in a graph. It also has a function that identifies nutrients that are lacking compared to the user's daily nutrient requirements and suggests specific foods that should be consumed additionally. It can also acquire images of the refrigerator, identify the foods contained therein through image analysis, and suggest nutritionally balanced menus based on that information. It also has a function that records the user's nutritional intake and weight daily and displays them in calendar format, and provides specific nutritional management plans for pregnant users and users on diets. This allows users to easily manage their daily meals and maintain a healthy diet.

[0006] "Food images" refer to visual data of food or dishes photographed by users using devices such as smartphones or digital cameras.

[0007] "Means for acquiring images" refers to a device or method for inputting image data of a dish into the system using the camera function or upload function provided on the terminal.

[0008] "Means for analyzing the image and identifying each food type in the image" refers to an algorithm or process that uses computer-based image recognition technology on the captured image to automatically identify each individual food or dish type contained in the image.

[0009] "Means for retrieving the nutritional value of identified foods from a database" refers to a system or method for searching and retrieving nutritional value information of foods identified by image recognition from a pre-installed database.

[0010] "Means for calculating total nutritional value" refers to a calculation algorithm or process that calculates the total calories and total amount of each nutrient based on the nutritional value of each food obtained.

[0011] "Means for displaying calculated nutritional values ​​in a graph" refers to a method or system for presenting calculated nutritional value data to a user in the form of a bar graph, pie chart, or the like, in order to make the data visually easy to understand.

[0012] "Means for identifying nutrients that are lacking compared to the daily required nutrients and suggesting additional foods to be consumed" refers to a method or system for comparing the user's recommended daily intake with their current nutritional intake, calculating the amount of nutrients that are lacking, and recommending specific foods or dishes to supplement them.

[0013] "Refrigerator image" refers to digital image data captured by a user to visually record the food in the refrigerator.

[0014] "Means for acquiring images of the refrigerator" refers to a device or method that allows a user to take a picture of the food in the refrigerator with a camera and input the image into the system.

[0015] "Means for identifying food in the refrigerator" refers to image recognition technology that analyzes images of the refrigerator to automatically identify the individual foods and ingredients contained within.

[0016] "Means for suggesting nutritionally balanced menus based on identified foods" refers to a system or method for suggesting nutritionally balanced meal menus and recipes to users based on recognized ingredients.

[0017] "Means for recording a user's daily nutritional intake status and weight and displaying it in calendar format" refers to a system that records daily nutritional intake data and weight changes and visually presents them in calendar format so that they can be checked on a daily basis.

[0018] "Means for providing specific nutritional management plans for users who are pregnant or dieting and making nutritional suggestions based on those plans" refers to methods and systems for creating nutritional management plans tailored to specific health conditions or goals and supporting appropriate nutritional intake based on those plans. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] System Overview

[0041] The present invention is a system that acquires images of dishes, analyzes the images to identify the type of food, and calculates and displays the nutritional values ​​of each food. It also has the function of acquiring images of a refrigerator, identifying the foods in the refrigerator, and proposing menus based on the identified foods. A specific embodiment of this system is described below.

[0042] Main functions and program processing

[0043] 1. Acquiring and Preprocessing Images of Dishes

[0044] The user takes a photo of the food with their smartphone and launches the dedicated app. The device receives the photo, performs basic pre-processing, and then sends it to the server.

[0045] 2. Image Recognition

[0046] The server uses an image recognition model (e.g., convolutional neural network) to analyze the received images. First, it identifies the type of food in the image, and then uses that information to retrieve the nutritional value of each food from a database. Through this process, it can identify foods such as "salad," "chicken steak," and "rice" that appear in the photo.

[0047] 3. Calculating and displaying nutritional values

[0048] The server calculates the total calories and the total of each nutrient based on the acquired nutritional information, and visualizes the results in a graph format (e.g., bar graph, pie chart) and sends it to the device, allowing the user to easily check it.

[0049] 4. Nutrition Suggestions

[0050] The server compares the user's daily nutrient requirements with their current intake and identifies any nutrient deficiencies. It also suggests specific foods to supplement the deficiencies. For example, it generates a notification such as, "You need 20 grams more dietary fiber. Try adding brown rice or mushrooms." The device displays this notification to the user.

[0051] 5. Refrigerator image acquisition and food ingredient recognition

[0052] The user takes a photo of the inside of their refrigerator and uploads the image using a dedicated app. The device receives the image and sends it to a server. The server uses image recognition technology to identify the food in the refrigerator. Based on this, the system suggests nutritionally balanced meals.

[0053] 6. Daily Records and Management

[0054] The server records the user's daily nutritional intake and weight and displays the data in a calendar format, allowing users to easily understand their health status. It also provides special nutritional management plans for pregnant and dieting users, and suggests nutrients based on those plans.

[0055] Specific examples

[0056] Food image processing

[0057] 1. The user takes a photo of their lunch (salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[0058] 2. The server receives the image and uses image recognition algorithms to identify salad, chicken steak, and rice.

[0059] 3. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[0060] 4. The server graphs the calculation results and sends them to the terminal for display to the user.

[0061] 5. The server identifies the nutrients that are lacking (e.g., dietary fiber) and suggests specific foods (e.g., brown rice).

[0062] Checking the ingredients in the refrigerator and suggesting menus

[0063] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[0064] 2. The server receives the images and uses image recognition algorithms to identify the food in the refrigerator.

[0065] 3. The server will suggest a nutritionally balanced menu (e.g., tomato and cheese salad) based on the identified foods.

[0066] Daily records and health management

[0067] 1. The server records daily nutritional intake and weight and displays them in a calendar format.

[0068] 2. The server provides special nutritional management plans for users who are pregnant or on a diet and makes suggestions based on them.

[0069] This allows users to easily manage their nutritional intake and efficiently maintain their health.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The user takes a picture of the food with their smartphone and launches a dedicated app.

[0073] Step 2:

[0074] The device receives the captured image and performs preprocessing on the image data (e.g., image resizing, format conversion).

[0075] Step 3:

[0076] The terminal transmits the image data after the preprocessing to the server.

[0077] Step 4:

[0078] The server analyzes the received image data and applies an image recognition algorithm (e.g., convolutional neural network) to identify each food item in the image.

[0079] Step 5:

[0080] Based on the identified food information, the server retrieves nutritional data (e.g., calories, protein, fat, carbohydrates) for each food from the database.

[0081] Step 6:

[0082] The server calculates the overall nutritional value of each food item based on the nutritional data it has acquired, by adding up the calories and nutrients of each food item.

[0083] Step 7:

[0084] The server then graphs the results, generating visual bar and pie charts of calories, protein, fat, and carbohydrates, for example.

[0085] Step 8:

[0086] The server sends the graphed nutritional data to the terminal, which displays the data to the user.

[0087] Step 9:

[0088] The server compares your current intake with your daily nutrient needs and identifies any nutrient deficiencies. Specifically, it compares your current intake with the recommended intake.

[0089] Step 10:

[0090] The server will suggest specific foods to supplement the missing nutrients, generating a notification such as "You are missing 20 grams of dietary fiber. Add brown rice or mushrooms."

[0091] Step 11:

[0092] The server generates a notification and sends it to the terminal, which displays it to the user.

[0093] Step 12:

[0094] The user takes a picture of the refrigerator and uploads it to the server using a dedicated app.

[0095] Step 13:

[0096] The server receives the image of the refrigerator and applies image recognition algorithms to identify the food items inside the refrigerator.

[0097] Step 14:

[0098] The server will then suggest nutritionally balanced meals based on the identified food data, such as a tomato and cheese salad and a lettuce and chicken sandwich.

[0099] Step 15:

[0100] The server sends the proposed menu information to the terminal, which displays this information to the user.

[0101] Step 16:

[0102] The server records the user's nutritional intake and weight on a daily basis and generates data to be displayed in calendar format.

[0103] Step 17:

[0104] The server sends the generated daily record data to the terminal, which displays it to the user in a calendar format.

[0105] Step 18:

[0106] The server creates a special nutritional management plan for users who are pregnant or dieting, and makes specific nutritional recommendations based on that plan.

[0107] Step 19:

[0108] The server sends recommendations based on a special nutritional management plan to the terminal, which displays them to the user.

[0109] Example 1

[0110] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0111] Conventional nutrition management systems require users to manually input food data, which is time-consuming and often lacks accuracy. Furthermore, they are unable to efficiently manage ingredients and suggest menus for users, making it difficult to plan nutritionally balanced meals. Furthermore, recording daily nutritional intake and weight is cumbersome, making it difficult to provide effective nutritional advice to specific users, such as those who are pregnant or on a diet.

[0112] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0113] In this invention, the server includes a means for preprocessing food images, a means for identifying the type of food in the image using a convolutional neural network, and a means for retrieving nutritional values ​​from a database and calculating the total nutritional value. This allows a user to simply take a photo of a dish, and the system automatically calculates the nutritional value of each food and suggests missing nutrients and foods that should be consumed in addition, enabling efficient and accurate nutritional management. Furthermore, the system also includes functions for preprocessing refrigerator images, identifying foods in the refrigerator using image recognition technology, and suggesting menus based on the identified foods, as well as a function for recording the user's nutritional intake and weight daily and displaying them in calendar format, thereby enabling comprehensive dietary and health management.

[0114] "Means for acquiring images of food" refers to the function that allows a user to take a photo of food using an input device such as a smartphone or camera and import it into the system.

[0115] "Means for preprocessing acquired images" refers to processing such as noise removal, resolution adjustment, and color correction performed on received images to improve the accuracy of image analysis.

[0116] The "means for transmitting preprocessed images to a server" refers to a function for transferring image data for which preprocessing has been completed to a server via the Internet.

[0117] "Means for identifying each food type in an image using a convolutional neural network" refers to the ability to use a convolutional neural network, a type of deep learning technology, to identify each food object in a preprocessed image and classify its type.

[0118] "Means for retrieving the nutritional value of identified foods from a database and calculating the total nutritional value" refers to a function for retrieving nutritional value information for each identified food from a database and calculating the total calories and the total value of each nutrient.

[0119] "Means for displaying calculated nutritional values ​​in a graph" refers to a function that visualizes and displays the calculated nutritional values ​​in the form of a bar graph, pie chart, etc., so that the user can easily understand them intuitively.

[0120] "Means to identify nutrients that are lacking compared to the daily required nutrients and suggest additional foods that should be taken" refers to a function that compares the user's daily nutritional intake standard with their current intake status, identifies nutrients that are lacking, and suggests specific foods to make up for the deficiency.

[0121] "Means for acquiring images of the refrigerator" refers to a function that allows a user to take a photo of the inside of the refrigerator using an input device such as a smartphone or camera and import it into the system.

[0122] "Means for identifying food items in a refrigerator using image recognition technology" refers to the use of deep learning or other image recognition algorithms to identify each food object in a photograph of the refrigerator and determine its type and quantity.

[0123] "Means for suggesting nutritionally balanced menus based on identified foods" refers to a function that suggests nutritionally balanced meal menus to the user based on the foods present in the refrigerator.

[0124] "Means for recording the user's nutritional intake and weight on a daily basis and displaying it in calendar format" refers to a function that records the user's daily nutrition intake and weight fluctuations and visually displays the data in calendar format.

[0125] "A means of providing specific nutritional management plans for users who are pregnant or on a diet and making nutritional suggestions based on those plans" refers to the function of creating appropriate nutritional management plans for users with specific health conditions or goals, and making specific nutritional supplement suggestions based on those plans.

[0126] System Overview

[0127] This system captures images of food and the contents of a refrigerator, identifies the type of food through image analysis, and calculates and displays its nutritional value. The server uses image recognition technology to allow users to easily manage their diet and nutrition via their smartphone. It also supports health management by recording the user's daily nutritional intake and weight and visualizing them in a calendar format.

[0128] Main functions and program processing

[0129] Acquiring and preprocessing food images

[0130] A user takes a photo of a dish using a smartphone and launches a dedicated app. The device receives the captured image and performs preprocessing such as noise reduction, resolution adjustment, and color correction. This preprocessing is performed using image analysis software (e.g., OpenCV). The preprocessed image is then sent to the server using a secure protocol (e.g., HTTPS).

[0131] Image Recognition

[0132] The server receives the image and uses a convolutional neural network (e.g., using TensorFlow or PyTorch) to identify the type of food in the image. This identifies foods such as "salad," "chicken steak," and "rice." The server then retrieves the nutritional value information for each food from a database (e.g., an SQL database).

[0133] Nutritional Value Calculation and Display

[0134] The server calculates the total calories and the total of each nutrient based on the acquired nutritional information. These calculation results are converted into graphs (bar graphs, pie charts, etc.) using a visualization tool (e.g., Matplotlib), and then sent to the terminal to be displayed intuitively to the user.

[0135] Nutrition Suggestions

[0136] The server compares the user's daily nutrient needs with their current intake. It then identifies nutrient deficiencies and suggests specific foods to fill the gaps. For example, it generates a notification such as, "You need 20 more grams of dietary fiber. Try adding brown rice or mushrooms." The device then displays this notification to the user.

[0137] Refrigerator image acquisition and food ingredient recognition

[0138] The user takes a photo of the inside of the refrigerator with their smartphone and uploads the image using a dedicated app. The device performs preprocessing and sends the image to the server. The server uses an image recognition algorithm (e.g., YOLOv4 or EfficientDet) to identify the foods in the refrigerator and uses that information to suggest nutritionally balanced meals.

[0139] Daily records and management

[0140] The server records the user's daily nutritional intake and weight and stores this data in a database. A view displaying the recorded data in a calendar format is generated and sent to the device. This allows the user to grasp their own health information at a glance. It also provides special nutritional management plans (e.g., for pregnancy or dieting) and suggests nutrients that are suitable for the user.

[0141] Specific examples

[0142] A concrete example of food image processing

[0143] 1. The user takes a photo of their lunch (salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[0144] 2. The server receives the image and uses an image recognition algorithm (e.g., TensorFlow's convolutional neural network) to identify salad, chicken steak, and rice.

[0145] 3. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[0146] 4. The server graphs the calculation results and sends them to the terminal for display to the user.

[0147] 5. The server identifies the nutrients that are lacking (e.g., dietary fiber) and suggests specific foods (e.g., brown rice).

[0148] Checking the ingredients in the refrigerator and suggesting a menu

[0149] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[0150] 2. The server receives the images and uses an image recognition algorithm (e.g., YOLOv4) to identify the food in the refrigerator.

[0151] 3. The server will suggest a nutritionally balanced menu (e.g., tomato and cheese salad) based on the identified foods.

[0152] Examples of daily records and health management

[0153] 1. The server records daily nutritional intake and weight and displays them in a calendar format.

[0154] 2. The server provides special nutritional management plans for users who are pregnant or on a diet and makes suggestions based on them.

[0155] By combining all of the above functions, users can easily and efficiently manage their nutritional intake and health. This system will be an important tool that contributes to raising consumer health awareness.

[0156] Example prompts to be input to the generative AI model

[0157] "Analyze this food photo to identify the types of foods it contains and their nutritional value."

[0158] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0159] Step 1: The user takes a picture of the food and launches the dedicated app.

[0160] The user takes a photo of the food using the camera function of their smartphone, opens the dedicated app, selects the image, and presses a button to start processing. The input of this step is the image taken by the camera, and the output is the image imported into the dedicated app.

[0161] Step 2: The device preprocesses the image.

[0162] The device performs preprocessing on the captured image, such as noise removal, resolution adjustment, and color correction, to improve the accuracy of image analysis. Specifically, it uses image processing libraries such as OpenCV. The input is the image taken by the user, and the output is a clear, preprocessed image.

[0163] Step 3: The device sends the preprocessed image to the server.

[0164] The device sends the preprocessed image data to the server using the HTTPS protocol. Data is encrypted during transmission to ensure security. The input is the preprocessed image, and the output is the image sent to the server.

[0165] Step 4: The server receives the image and applies the convolutional neural network.

[0166] The server stores the received images and begins analyzing them using TensorFlow or PyTorch, identifying each food type using a convolutional neural network. The input for this step is the image sent to the server, and the output is a list of each identified food object.

[0167] Step 5: The server retrieves the nutritional value of the food from the database.

[0168] For each food item identified as a result of the analysis, the server retrieves nutritional information from a database (e.g., an SQL database). The input is a list of identified foods, and the output is the nutritional information for each food item.

[0169] Step 6: The server calculates the total nutritional value.

[0170] The server calculates the total calories and the total of each nutrient based on the nutritional value information of each food. The input is the nutritional value information of each food, and the output is the total nutritional value.

[0171] Step 7: The server graphs the calculation results and sends them to the terminal.

[0172] The server converts the calculation results into graphs (such as bar graphs or pie charts) using a visualization tool (e.g., Matplotlib) and sends them to the terminal. The terminal receives them and displays them to the user. The input is the total nutritional value, and the output is the graphed data.

[0173] Step 8: Your server will provide you with nutrients you are lacking and suggestions for adding more.

[0174] The server compares the user's daily nutrient needs with their current intake to identify any nutrient deficiencies, and then suggests specific foods to supplement the deficiencies. The input is the user's intake and nutritional criteria, and the output is a notification of the recommendations.

[0175] Step 9: The user takes a picture of the inside of the refrigerator and uploads the image.

[0176] The user takes a picture of the inside of the refrigerator with their smartphone and uploads the image to a dedicated app. The input is the image of the inside of the refrigerator, and the output is the image uploaded to the app.

[0177] Step 10: The device sends the image of the refrigerator to the server.

[0178] The device performs preprocessing and then sends the image to the server. The input is the preprocessed image and the output is the image sent to the server.

[0179] Step 11: The server applies image recognition algorithms to identify the food in the refrigerator.

[0180] The server uses image recognition techniques such as YOLOv4 and EfficientDet to identify the foods in the refrigerator. The input is an image of the refrigerator sent to the server, and the output is a list of identified foods.

[0181] Step 12: The server suggests a menu.

[0182] The server creates a nutritionally balanced menu based on the identified food information and proposes it to the user. The input is a list of foods in the refrigerator, and the output is a proposed menu.

[0183] Step 13: The server records the user's nutritional intake and weight and displays them in a calendar format.

[0184] The server records the user's daily nutritional intake and weight and displays the data in a calendar format. The input is the user's intake information and weight data, and the output is a calendar display.

[0185] Step 14: The server provides a special nutrition plan.

[0186] The server provides special nutritional management plans for pregnant and dieting users and makes specific nutrition recommendations based on those plans. The input is each user's specific conditions, and the output is a notification of recommendations based on the plan.

[0187] (Application example 1)

[0188] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0189] Currently, it is important to eat meals that are conscious of health management and nutritional balance, but it is not easy to actually understand the contents of one's daily diet and manage it appropriately. In addition, when using delivery services, there are few ways to understand the nutritional value of the food, making it difficult for customers to make healthy choices. Furthermore, it is time-consuming to plan a menu that effectively uses the ingredients in the refrigerator. To solve these issues, a system that provides more detailed nutritional information and makes healthy meal suggestions is needed.

[0190] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0191] In this invention, the server includes means for acquiring images of dishes, means for analyzing the acquired images and identifying the type of each food in the images, means for acquiring the nutritional value of the identified foods from a database and calculating the total nutritional value, means for displaying the calculated nutritional value in a graph, means for identifying nutrients that are lacking by comparing them with the daily nutrient requirement and suggesting additional foods that should be consumed, means for a user to upload an image of a dish when ordering delivery and display the nutritional value information in real time, means for automatically calculating nutritional information for menus provided by the delivery company and suggesting it to the customer, means for recognizing the contents of the customer's refrigerator and suggesting healthy menus based on that, and means for managing the user's daily nutritional intake and suggesting an optimal meal plan.

[0192] This allows users to use information about ingredients in their refrigerators at home when ordering delivery to plan nutritionally balanced meals, enabling them to make healthy meal choices. It also allows for efficient and effective daily nutrition management.

[0193] "Cuisine" refers to food that has been prepared by cooking and that has been modified to improve its nutritional value or palatability.

[0194] "Capturing an image" refers to the process of capturing visual information as digital data using a device such as a camera or scanner.

[0195] "Analyzing images" refers to the process of applying pattern recognition and machine learning algorithms to captured image data to extract and identify specific information.

[0196] "Identifying the type of food" refers to classifying objects recognized from acquired image data into specific food categories.

[0197] "Retrieving nutritional values ​​from a database" means retrieving the specific calculations and nutritional components of a food from a pre-stored information source.

[0198] "Calculating total nutritional value" refers to the act of adding up the individual nutritional components of a specified food and calculating their sum.

[0199] "Displaying in a graph" means presenting calculated numerical information or data in the form of a bar graph, pie chart, or the like to make it visually easier to understand.

[0200] "Nutrient identification" means diagnosing whether you have a deficiency or excess of certain key components in your daily diet or nutrition plan.

[0201] "Suggesting foods to consume" refers to the act of recommending specific foods or ingredients to supplement missing nutrients.

[0202] "Delivery order" refers to the act of a user making a request to have food delivered by a delivery company.

[0203] "Recognizing the contents of a refrigerator" means performing image analysis to identify the types and quantities of food and ingredients stored in the refrigerator.

[0204] "Suggesting a menu" means creating and presenting a meal plan that takes nutritional balance into consideration and combines multiple dishes and foods.

[0205] "Managing daily nutritional intake" refers to a user recording how much nutrients they consume each day and adjusting their health and eating habits based on that information.

[0206] "Proposing the optimal meal plan" means designing and recommending the best meal content based on an individual's nutritional status and lifestyle.

[0207] To implement the present invention, the system operates as follows.

[0208] First, the user takes a photo of the food with their smartphone and launches the dedicated app. The device receives the photo, performs basic preprocessing, and then sends it to the server. This preprocessing uses an image processing library such as OpenCV.

[0209] The server then uses a pre-trained image recognition model (e.g., a TensorFlow / Keras convolutional neural network) to analyze the received image. The image recognition model identifies the type of food and uses that information to retrieve the nutritional values ​​of each food item from a database. At this stage, the necessary nutritional value data is obtained based on the information extracted from the image, such as "salad" or "chicken steak."

[0210] The server then calculates the total calories and the total of each nutrient based on the acquired nutritional information. The calculation results are visualized in a graph format (e.g., bar graph, pie chart) and sent to the device. This allows the user to check the nutritional value of the food they have photographed at a glance.

[0211] The server also compares the user's daily nutrient needs with their current intake to identify any nutrient deficiencies. It also has the ability to suggest specific foods to fill the gaps. For example, it generates a notification that says, "You need 20 more grams of dietary fiber. Try adding brown rice or mushrooms." and displays it on the device.

[0212] Additionally, the system allows users to take a photo of the inside of their refrigerator and upload the image using a dedicated app. The device receives the image and sends it to a server, which uses image recognition technology to identify the foods in the refrigerator. Based on the results, the system suggests nutritionally balanced meals.

[0213] The server also has a function to record the user's daily nutritional intake and weight and display them in a calendar format, allowing users to easily understand their health status. It also provides special nutritional management plans for users who are pregnant or on a diet, and makes suggestions based on those plans.

[0214] When using a delivery service, users can upload a photo of the food they want to order and view its nutritional information in real time. The system also includes a function to automatically calculate the nutritional information of the menu items offered by the delivery service and provide suggestions to customers. For example, users can be provided with nutritional information for delivery foods such as pizza and burgers.

[0215] Specific examples

[0216] 1. A user uploads a photo of a pizza.

[0217] 2. The server analyzes the image, identifies the pizza's ingredients, and retrieves its nutritional information from a database.

[0218] 3. The server calculates the nutritional value and displays it in a graph.

[0219] 4. The server will identify any nutrients that are lacking and make suggestions such as salads.

[0220] Example prompt sentence:

[0221] Upload an image of the food you want to order and we'll display nutritional information in real time and suggest meals that take into account the ingredients in your refrigerator.

[0222] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0223] Step 1:

[0224] The user takes a photo of the food with their smartphone and launches a dedicated app.

[0225] Input: Food image taken with a smartphone

[0226] Specific operation: The user takes a photo of the food and presses the "Upload image" button on the dedicated app.

[0227] Step 2:

[0228] The device receives the captured photo, performs some basic pre-processing, and then sends it to the server.

[0229] Input: Photographed food image

[0230] Data processing: Using the OpenCV library, preprocessing such as image resizing and noise reduction is performed.

[0231] Specific operation: After processing the image, the device uploads it to the server.

[0232] Step 3:

[0233] The server uses a pre-trained image recognition model to analyze the images it receives.

[0234] Input: Preprocessed food images

[0235] Data computation: A convolutional neural network (CNN) using TensorFlow / Keras is used to identify the type of food in the image.

[0236] What it does: The server runs a CNN model to classify the food in the image.

[0237] Step 4:

[0238] The server retrieves the nutritional values ​​of the identified foods from the database and calculates the total nutritional value.

[0239] Input: List of identified foods

[0240] Data acquisition: Obtain the nutritional value information of each food from the nutritional value database.

[0241] Data calculation: The nutritional value of each food is added up to calculate the total calories and the total of each nutrient.

[0242] Specific operation: The server issues a database query and calculates nutritional values ​​based on the retrieved data.

[0243] Step 5:

[0244] The server generates data that displays the calculated nutritional values ​​in a graph and sends it to the terminal.

[0245] Input: Totaled nutritional data

[0246] Data manipulation: Formatting data into visually understandable formats such as bar graphs and pie charts.

[0247] Specific operation: The server uses a graph generation tool to create visual data and transfers it to the terminal.

[0248] Step 6:

[0249] The terminal displays the graph to the user.

[0250] Input: Graph data sent from the server

[0251] What it does: A dedicated app displays graphs and provides visual feedback to the user.

[0252] Step 7:

[0253] The server compares the user's daily nutrient requirements with their current intake and identifies any nutrients that are lacking.

[0254] Input: User's nutritional intake data and nutrient requirements

[0255] Data calculation: Calculates nutrient deficiencies based on current intake.

[0256] What it does: The server analyzes current nutritional intake data and identifies nutrients that are lacking.

[0257] Step 8:

[0258] The server suggests specific foods to supplement the missing nutrients and sends a notification to the device.

[0259] Input: Missing nutrient data

[0260] Data generation: Generate a list of foods to supplement missing nutrients.

[0261] What happens: The server selects food suggestions and generates a notification with that information.

[0262] Step 9:

[0263] Users take a photo of the inside of their refrigerator and upload the image using a dedicated app.

[0264] Input: Photo of the inside of a refrigerator

[0265] What happens: The user takes a photo of the refrigerator and presses the upload button in the app.

[0266] Step 10:

[0267] The device receives a photo of the inside of the refrigerator and sends it to the server.

[0268] Input: Photo of the inside of a refrigerator

[0269] Specific operation: The device receives the photo, preprocesses it, and uploads it to the server.

[0270] Step 11:

[0271] The server uses image recognition technology to identify food in the refrigerator and suggest nutritionally balanced meals.

[0272] Input: Photo of the inside of a refrigerator

[0273] Data Computing: Using image recognition algorithms to classify and identify food items in the refrigerator.

[0274] Specific operation: The server performs image analysis and generates a menu based on the food information in the refrigerator.

[0275] Step 12:

[0276] The server records the user's daily nutritional intake and weight and displays them in calendar format.

[0277] Input: User nutritional intake and weight data

[0278] Data processing: Visually formatting data in a calendar format.

[0279] Specific operation: The server organizes daily data and provides it to the device as a calendar display.

[0280] Step 13:

[0281] The server provides special nutritional management plans for pregnant or dieting users and suggests nutrients based on the plans.

[0282] Input: User's special health status data (e.g., pregnant, dieting)

[0283] Data calculation: Create a special nutritional management plan and generate nutritional recommendations based on it.

[0284] Specific operation: The server creates a special plan and sends a notification containing the proposal to the device.

[0285] Example prompt sentence:

[0286] Upload an image of the food you want to order and we'll display nutritional information in real time and suggest meals that take into account the ingredients in your refrigerator.

[0287] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0288] System Overview

[0289] The present invention combines a system that acquires images of dishes, analyzes the images to identify the type of food, calculates and displays the nutritional value, and an emotion engine that recognizes the user's emotions. This system not only provides nutritional management but also makes meal suggestions based on the user's emotional state. Specific embodiments of this system are described below.

[0290] Main functions and program processing

[0291] 1. Acquiring and Preprocessing Images of Dishes

[0292] The user takes a photo of the food with their smartphone and launches the dedicated app. The device receives the image, preprocesses it (e.g., resizes the image, converts the format), and then sends it to the server.

[0293] 2. Image Recognition

[0294] The server uses an image recognition algorithm (e.g., convolutional neural network) to analyze the received images. It identifies the type of food in the image and uses that information to retrieve the nutritional value of each food from a database. Through this process, it can identify foods such as "salad," "chicken steak," and "rice" that appear in the photo.

[0295] 3. Calculating and displaying nutritional values

[0296] The server calculates the total calories and the total of each nutrient based on the acquired nutritional information, and visualizes the results in a graph format (e.g., bar graph, pie chart) and sends it to the device, allowing the user to easily check it.

[0297] 4. Nutrition Suggestions

[0298] The server compares the user's daily nutrient requirements with their current intake and identifies any nutrient deficiencies. It also suggests specific foods to supplement the deficiencies. For example, it generates a notification saying, "You need 20 grams more dietary fiber. Try adding brown rice or mushrooms." The device displays this notification to the user.

[0299] 5. Emotion Recognition by Emotion Engine

[0300] The server uses an emotion recognition engine to analyze the user's emotions, specifically identifying emotions such as joy, excitement, and satisfaction based on the food image and user input.

[0301] 6. Adjusting offers based on emotions

[0302] The server may tailor suggested additional foods or meal options based on the identified emotional state, for example, if the user is expressing a feeling of joy, more colorful dishes may be suggested to enhance enjoyment.

[0303] 7. Refrigerator image acquisition and food ingredient recognition

[0304] The user takes a photo of the inside of their refrigerator and uploads the image to a server using a dedicated app. The server then uses image recognition technology to identify the food in the refrigerator. Based on this information, the system suggests nutritionally balanced meals.

[0305] 8. Daily Records and Management

[0306] The server records the user's daily nutritional intake and weight and displays the data in a calendar format, allowing users to easily understand their health status. It also provides special nutritional management plans for pregnant and dieting users, and suggests nutrients based on those plans.

[0307] Specific examples

[0308] Food image processing

[0309] 1. The user takes a photo of their lunch (salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[0310] 2. The server receives the image and uses image recognition algorithms to identify salad, chicken steak, and rice.

[0311] 3. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[0312] 4. The server graphs the calculation results and sends them to the terminal for display to the user.

[0313] 5. The server identifies the nutrients that are lacking (e.g., dietary fiber) and suggests specific foods (e.g., brown rice).

[0314] 6. The server uses an emotion engine to analyze the user's emotions from the photos and input data and adjust the suggestions.

[0315] Checking the ingredients in the refrigerator and suggesting menus

[0316] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[0317] 2. The server receives the images and uses image recognition algorithms to identify the food in the refrigerator.

[0318] 3. The server will suggest a nutritionally balanced menu (e.g., tomato and cheese salad) based on the identified foods.

[0319] 4. The server analyzes the user's emotions through an emotion recognition engine and adjusts the menu accordingly.

[0320] Daily records and health management

[0321] 1. The server records daily nutritional intake and weight and displays them in a calendar format.

[0322] 2. The server provides special nutritional management plans for users who are pregnant or on a diet and makes suggestions based on them.

[0323] 3. The server tailors daily meal and activity suggestions based on the user's emotional state.

[0324] This allows users to not only manage their nutrition, but also receive meal suggestions based on their emotional state, allowing them to enjoy a richer diet.

[0325] The processing flow will be explained below.

[0326] Step 1:

[0327] The user takes a picture of the food with their smartphone and launches a dedicated app.

[0328] Step 2:

[0329] The device receives the captured image and performs preprocessing on the image data (e.g., image resizing, format conversion).

[0330] Step 3:

[0331] The terminal transmits the preprocessed image data to the server.

[0332] Step 4:

[0333] The server analyzes the received image data and uses an image recognition algorithm (e.g., convolutional neural network) to identify each food item in the image.

[0334] Step 5:

[0335] Based on the identified food information, the server retrieves nutritional data (e.g., calories, protein, fat, carbohydrates) for each food from the database.

[0336] Step 6:

[0337] The server calculates the overall nutritional value of each food based on the nutritional data of each food item, adding up the calories and nutrients of each food item.

[0338] Step 7:

[0339] The server then graphs the results, generating visual bar and pie charts of calories, protein, fat, and carbohydrates.

[0340] Step 8:

[0341] The server sends the graphed nutritional data to the terminal, which displays it to the user.

[0342] Step 9:

[0343] The server compares your current intake with your daily nutrient needs to identify any nutrient deficiencies. It also compares your current intake with the recommended intake.

[0344] Step 10:

[0345] The server will suggest specific foods to supplement the missing nutrients, generating a notification such as "You're missing 20 grams of dietary fiber. Add brown rice or mushrooms."

[0346] Step 11:

[0347] The server sends the generated notification to the terminal, which displays it to the user.

[0348] Step 12:

[0349] The server uses an emotion recognition engine to analyze the user's emotions based on the food images and user input, identifying emotions such as joy, excitement, and satisfaction.

[0350] Step 13:

[0351] The server adjusts the suggested additional foods and menu items based on the identified emotional state: if the emotional state is "joy," it suggests more elaborate dishes; if the emotional state is "excited," it suggests new dishes and trending foods.

[0352] Step 14:

[0353] The user takes a photo of the inside of the refrigerator and uploads the image to the server using a dedicated app.

[0354] Step 15:

[0355] The server analyzes the received image of the refrigerator and applies image recognition algorithms to identify each food item inside.

[0356] Step 16:

[0357] The server will then suggest nutritionally balanced meals based on the identified food information, for example, "a tomato and cheese salad, lettuce and chicken sandwich."

[0358] Step 17:

[0359] The server sends the proposed menu information to the terminal, which displays it to the user.

[0360] Step 18:

[0361] The server records the user's nutritional intake and weight on a daily basis and generates data to be displayed in calendar format.

[0362] Step 19:

[0363] The server transmits the generated daily record data to the terminal, which displays it to the user in a calendar format.

[0364] Step 20:

[0365] The server creates a special nutritional management plan for users who are pregnant or dieting, and makes specific nutritional recommendations based on that plan.

[0366] Step 21:

[0367] The server sends recommendations based on a special nutritional management plan to the terminal, which displays them to the user.

[0368] Step 22:

[0369] The server tailors daily food and activity suggestions based on the user's emotional state: if the emotional state is "sad," it suggests specific foods and activities to lighten the mood.

[0370] Example 2

[0371] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0372] Today's consumers are increasingly interested in eating a nutritionally balanced diet and maintaining their health. However, it is difficult to understand which foods contain which nutrients and manage their nutrient intake in their daily lives. Furthermore, there is a lack of systems that provide dietary recommendations based not only on nutritional value but also on the user's emotions. Given this situation, it is necessary to efficiently and accurately manage nutrition and provide recommendations that respond to the user's emotions.

[0373] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0374] In this invention, the server

[0375] A means for acquiring an image of a dish;

[0376] means for pre-processing the acquired images;

[0377] means for analyzing the preprocessed image data and identifying each food type in the image;

[0378] means for retrieving the nutritional values ​​of the identified foods from a database;

[0379] a means for calculating total nutritional value;

[0380] A means for displaying the calculated nutritional values ​​in a graph;

[0381] A method to identify nutrients that are lacking compared to the daily required nutrients and suggest additional foods that should be taken in,

[0382] means for recognizing a user's emotion and tailoring suggestions based on the emotion;

[0383] A means to acquire, preprocess, and analyze images of the refrigerator, identify the foods in the refrigerator, and propose nutritionally balanced meals.

[0384] A means for recording the user's daily nutritional intake and weight and displaying them in a calendar format;

[0385] A means of providing specific nutritional management plans for pregnant or dieting users and making nutritional suggestions based on those plans

[0386] This allows for efficient and accurate nutritional management and meal suggestions that reflect the user's emotions.

[0387] "Food image" is visual data about food that is photographed or acquired by the user.

[0388] "Preprocessing" refers to initial processing of captured or acquired images, such as adjusting their size or converting their format, to make them easier to analyze.

[0389] "Image recognition algorithm" refers to a computational method used to identify specific objects or features within an image.

[0390] "Food type" identifies the various ingredients and food items contained in the photographed food image.

[0391] "Nutritional value" is data that indicates the amount of energy and various nutrients (e.g., protein, lipids, carbohydrates, vitamins, minerals, etc.) contained in food.

[0392] A "database" is a collection of information in which nutritional information about foods is systematically organized and stored.

[0393] "Daily Nutrient Requirements" refers to the standard of nutrients that a user should consume each day.

[0394] "Additional foods" refers to specific ingredients or dishes recommended to supplement missing nutrients.

[0395] An "emotion recognition engine" refers to technology for analyzing a user's emotional state, specifically recognizing a user's emotions such as joy, excitement, and satisfaction from images and input data.

[0396] "Refrigerator image" refers to visual data of the inside of a refrigerator photographed or acquired by a user.

[0397] A "nutritional balanced menu" refers to a meal plan that is nutritionally balanced based on specific food information.

[0398] "Calendar format" refers to a method of organizing data by date and displaying it in a visually easy-to-understand format.

[0399] "Nutrition Management Plan" refers to a plan designed to support optimal nutritional intake for a user in a specific situation, such as during pregnancy or while dieting.

[0400] System Overview

[0401] The present invention combines a system that acquires images of dishes, analyzes the images to identify the type of food, calculates and displays the nutritional value of each food, and an emotion engine that recognizes the user's emotions. This system provides meal suggestions based on the user's nutritional balance and emotional state when managing their daily diet.

[0402] Hardware and software used

[0403] The user uses a device such as a smartphone. A dedicated app is installed on the device, and it is equipped with a pre-processing function for captured images. The pre-processed image data is sent to a server via a network. The following software is running on the server:

[0404] Image recognition algorithms (e.g., convolutional neural networks)

[0405] Database Management System (DBMS)

[0406] Emotion Recognition Engine

[0407] Visualization tools (e.g., D3.js, Chart.js)

[0408] Program processing overview

[0409] The user takes a photo of a dish and launches a dedicated app. The device receives the image and performs preprocessing such as resizing and format conversion. The preprocessed image data is then sent over the network to a server. The server uses an image recognition algorithm to identify the type of food in the image and retrieves the nutritional value of each food item from a database. Based on the retrieved nutritional value information, the server calculates and displays the total calories and the sum of each nutrient.

[0410] The server then compares the user's daily nutrient needs with their current intake to identify any nutrient deficiencies. When suggesting specific foods, an emotion recognition engine analyzes the user's emotional state and adjusts the suggestions accordingly. For example, if the user expresses joy, the server will suggest more colorful dishes to enhance the enjoyment.

[0411] Users can also take photos of the inside of their refrigerator and upload them using a dedicated app. The server uses an image recognition algorithm to identify the food items in the refrigerator and suggests menu items based on that information. In this case, an emotion recognition engine also analyzes the user's emotions and offers the most appropriate menu.

[0412] The server also records the user's daily nutritional intake and weight data and displays it in a calendar format. For users who are pregnant or on a diet, the server provides special nutritional management plans and suggests nutrients based on those plans.

[0413] Specific examples

[0414] Food image processing

[0415] 1. The user takes a photo of their lunch (e.g., salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[0416] 2. The device preprocesses the image and sends the formatted image to the server.

[0417] 3. The server uses image recognition algorithms to identify each food item.

[0418] 4. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[0419] 5. The server graphs the calculation results and sends them to the terminal for display to the user.

[0420] 6. The server will identify any missing nutrients (e.g., dietary fiber) and suggest foods to add (e.g., brown rice).

[0421] 7. The server uses an emotion engine to analyze the user's emotions from the images and input data and adjust the suggestions.

[0422] Checking the ingredients in the refrigerator and suggesting menus

[0423] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[0424] 2. The device preprocesses the image and sends the formatted image to the server.

[0425] 3. The server uses image recognition algorithms to identify the food in the refrigerator.

[0426] 4. The server will suggest a nutritionally balanced meal based on the identified foods (e.g., tomato and cheese salad).

[0427] 5. The server analyzes the user's emotions through an emotion recognition engine and adjusts the menu accordingly.

[0428] Example prompts to be input to the generative AI model

[0429] Analyze the following food images, identify the type of food for each, calculate the nutritional value of each food, and generate a sentence suggesting to the user what nutrients they may be missing:

[0430] Food image: [Image link]

[0431] It should also analyze the user's emotional state and provide meal suggestions based on those emotions.

[0432] Through such a system, users can receive nutritional management and emotionally-based meal suggestions, enabling them to enjoy a healthier and more fulfilling diet.

[0433] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0434] Step 1:

[0435] The user takes a photo of a dish with their smartphone and launches the dedicated app. The user then uploads this image to the dedicated app. The input is the photo of the dish, and the output is image data for preprocessing.

[0436] Step 2:

[0437] The device receives the captured image and performs preprocessing such as image resizing and format conversion. The input is the image uploaded by the user, and the output is the preprocessed image data. This preprocessing enables efficient image analysis.

[0438] Step 3:

[0439] The terminal sends the preprocessed image data to the server via the network. The input is the preprocessed image data, and the output is the transmission of the image data to the server.

[0440] Step 4:

[0441] The server uses an image recognition algorithm (e.g., convolutional neural network) to analyze the received images. The input is the preprocessed image data, and the output is the analyzed data including the type of each food. Based on this analyzed data, the server retrieves the nutritional value of each food from the database.

[0442] Step 5:

[0443] The server calculates the total calories and total of each nutrient based on the acquired nutritional information. The input is the nutritional value data of the identified food, and the output is the total nutritional value calculation. This calculation includes calories, proteins, fats, carbohydrates, and other major nutrients.

[0444] Step 6:

[0445] The server visualizes the calculation results in graph format (e.g., bar graph, pie chart). The input is the calculation result of total nutritional value, and the output is the visualized graph data. The graph makes it easier for users to intuitively understand the nutritional information.

[0446] Step 7:

[0447] The server sends the visualized data to the terminal and displays it to the user. The input is the visualized graph data, and the output is the graph displayed on the user's terminal.

[0448] Step 8:

[0449] The server compares the user's daily required nutrients with their current intake and identifies any nutrient deficiencies. The input is the daily required nutrient data and current intake data, and the output is the results of identifying the nutrient deficiencies.

[0450] Step 9:

[0451] The server suggests specific foods to supplement the missing nutrients. For example, it generates a notification saying, "You need 20 more grams of dietary fiber. Try adding brown rice or mushrooms." The input is the result of identifying the missing nutrients, and the output is a notification with specific food suggestions.

[0452] Step 10:

[0453] The terminal displays this notification to the user.,The input is the food suggestion notification from the server, and,the output is the suggestion content displayed on the user's,terminal.

[0454] Step 11:

[0455] The server uses an emotion recognition engine to analyze the user's emotions. The input is food images and user input data, and the output is data indicating the user's emotional state. Specifically, it generates data indicating the user's emotions, such as joy, excitement, and satisfaction.

[0456] Step 12:

[0457] The server adjusts the suggested additional foods and meal options based on the identified emotional state. The input is the user's emotional state data, and the output is the adjusted food and meal options.

[0458] Step 13:

[0459] The user takes a photo of the inside of the refrigerator and uploads the image to the server using a dedicated app. The input is the photo of the inside of the refrigerator, and the output is the transmission of image data to the server.

[0460] Step 14:

[0461] The server uses image recognition technology to identify the food in the refrigerator. The input is image data of the inside of the refrigerator, and the output is data of the identified food in the refrigerator.

[0462] Step 15:

[0463] The server then proposes a nutritionally balanced menu based on the identified foods. The input is the food data in the refrigerator, and the output is a nutritionally balanced menu proposal.

[0464] Step 16:

[0465] The server records daily nutritional intake and weight data and displays them in a calendar format. The input is the user's daily nutritional intake and weight data, and the output is the display data in a calendar format.

[0466] Step 17:

[0467] The server provides a special nutritional management plan for pregnant or dieting users and makes nutritional recommendations based on the plan. The input is the special nutritional management plan data, and the output is the nutritional recommendations based on the plan.

[0468] (Application example 2)

[0469] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0470] In modern society, nutritional management and emotional care are important for busy people to maintain a healthy diet. However, achieving this requires a lot of time and effort. Furthermore, when using a delivery service, it is not easy to determine whether the ordered food is healthy. Furthermore, there are no systems that can suggest meals based on the user's emotional state. Therefore, there is a need for a system that supports users' nutritional management and suggests optimal meals based on the user's emotional state.

[0471] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring food images, means for analyzing the acquired images and identifying the type of each food in the images, means for acquiring the nutritional values ​​of the identified foods from a database and calculating the total nutritional value, means for displaying the calculated nutritional values ​​in a graph, means for identifying nutrients that are deficient compared to the daily nutrient requirement and suggesting additional foods to be consumed, means for analyzing emotions based on user input and adjusting the suggested foods and menu contents, and means for acquiring images of delivered dishes and implementing the above means to suggest the next order. This allows users to instantly receive healthy meal suggestions that are appropriate for their emotional state even when using a delivery service.

[0472] The "means for acquiring an image of a dish" refers to a method or device that allows a user to take a photo of a dish and input it into the system.

[0473] The "means for analyzing the image and identifying the type of each food in the image" refers to an algorithm or program for analyzing the acquired image of food and identifying the type of each food.

[0474] "Means for obtaining the nutritional value of identified foods from a database and calculating the total nutritional value" refers to a method or device that obtains nutritional information for each identified food from a database and adds up the individual nutrients to calculate the total nutritional value.

[0475] The "means for displaying the calculated nutritional value in a graph" refers to a method or device for displaying the calculated nutritional value in a graph format to make it easier to understand visually.

[0476] "Means for identifying nutrients that are lacking by comparing with the daily required nutrients and suggesting foods that should be taken additionally" refers to a method or device that compares the user's daily required nutrients with their current intake status, identifies nutrients that are lacking, and suggests foods that should be taken to make up for the lack.

[0477] "Means for analyzing emotions based on user input and adjusting suggested food and menu items" refers to a method or device that analyzes a user's emotions based on text and other data entered by the user and optimizes suggested food and menu items according to the results.

[0478] "Means for taking images of delivered food and implementing the above-mentioned means to make suggestions for the next order" refers to a method or device for taking and taking images of delivered food, analyzing them, and making suggestions that will be useful for the next order.

[0479] This invention combines a system that acquires images of food, analyzes them to identify the type of food, calculates and displays the nutritional value, and an emotion engine that recognizes the user's emotions. This system goes beyond nutritional management and can also make meal suggestions based on the user's emotional state.

[0480] System configuration

[0481] The system consists of the following main components:

[0482] 1. Method for acquiring food images: The user takes a photo of the food with their smartphone and launches a dedicated app.

[0483] 2. Image analysis and food identification: The server receives the captured images and uses image recognition algorithms to identify the type of each food item.

[0484] 3. Means for obtaining and calculating nutritional value: The server obtains the nutritional value of the identified food from the database and calculates it.

[0485] 4. Nutritional value display means: The calculated nutritional value is visualized in graph form and sent to the user's device.

[0486] 5. Identifying nutrient deficiencies and suggesting foods: Identifying nutrients that are lacking compared to the daily required nutrients and suggesting additional foods that should be taken in.

[0487] 6. Sentiment analysis and suggestion adjustment: Recognize the user's emotions and adjust the suggestion content based on those emotions.

[0488] 7. Next order suggestion method for delivered food: By taking an image of the delivered food and implementing the above method, next order suggestion is made.

[0489] Hardware and software used

[0490] 1. Hardware:

[0491] Smartphone (used to capture food images)

[0492] Server (used for image analysis and data processing)

[0493] 2. Software:

[0494] Dedicated app (takes pictures of food on a smartphone and communicates with the server)

[0495] Image recognition algorithm (using convolutional neural networks)

[0496] Nutritional value database (used to obtain nutritional information)

[0497] Emotion recognition engine (analyzes the user's emotional state)

[0498] More examples

[0499] Usage example 1:

[0500] The user takes a photo of their lunch (salad, chicken steak, and rice) and uploads it through a dedicated app. The image is analyzed on the server, and the salad, chicken steak, and rice are identified. The nutritional value of each food item is retrieved from the database, and the total nutritional value is calculated. The calculation results are displayed in graph form, and based on the user's nutrient deficiencies, further foods to supplement are suggested.

[0501] Usage example 2:

[0502] Users take a photo of the inside of their refrigerator with their smartphone and upload the image through a dedicated app. The server analyzes the image and identifies the foods in the refrigerator. Based on the results, nutritionally balanced meals are suggested to the user.

[0503] Example prompt sentence:

[0504] "I want to develop a system that analyzes food images, calculates nutritional value, and makes meal recommendations based on the user's feelings. This system allows users to input their thoughts along with an image of the delivery food, and then makes recommendations based on that."

[0505] This allows users to instantly receive healthy meal suggestions that suit their emotional state, even when using a delivery service.

[0506] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0507] Step 1:

[0508] The user takes a photo of the food with their smartphone and launches a dedicated app.

[0509] Input: Food photo

[0510] Output: Food photo data

[0511] How it works: A user takes a photo of a dish using their smartphone and uploads the image to a dedicated app. The app compresses and encodes the image data and converts it into a format that can be sent.

[0512] Step 2:

[0513] The terminal receives the captured image, preprocesses the image data, and then sends it to the server.

[0514] Input: Compressed and encoded food photo data

[0515] Output: Preprocessed image data

[0516] Specific operation: The terminal performs preprocessing such as resizing and format conversion of the image data, and then sends the image data to the server.

[0517] Step 3:

[0518] The server uses image recognition algorithms to analyze the received images.

[0519] Input: Preprocessed image data

[0520] Output: Data about food types

[0521] How it works: An image recognition algorithm (e.g., a convolutional neural network) on the server analyzes the preprocessed image and identifies the type of food in the image. The identified food type data is output.

[0522] Step 4:

[0523] The server retrieves the nutritional values ​​of the identified foods from a database and calculates the total nutritional value.

[0524] Input: Food type data, nutritional value database

[0525] Output: Calculated nutritional information

[0526] Specific operation: The server obtains the nutritional information of each food from the nutritional value database based on the type of food, and calculates the overall nutritional value by adding up the nutritional values ​​of each food.

[0527] Step 5:

[0528] The server visualizes the calculated nutritional values ​​in graph form and sends them to the terminal.

[0529] Input: Calculated nutritional information

[0530] Output: Nutritional information in graphical format

[0531] Specific operation: The server visualizes the calculation results in the form of a graph (e.g., bar graph, pie chart) and sends the generated graph to the terminal.

[0532] Step 6:

[0533] The terminal displays the received nutritional information in graph form to the user.

[0534] Input: Nutritional information in graph format

[0535] Output: The displayed graph

[0536] Specific operation: The device receives the nutritional information in graph form and displays it on the screen.

[0537] Step 7:

[0538] The server compares the daily nutrient requirements with the current intake, identifies any nutrient deficiencies, and suggests additional foods to consume.

[0539] Input: Calculated nutritional information, daily required nutrients information

[0540] Output: Suggestions for supplemental foods

[0541] Specific operation: The server compares the calculated nutritional value information with the daily nutrient requirement information to identify any nutrient deficiencies. It then searches the database for foods that can make up for those deficiencies and generates recommendations for the user.

[0542] Step 8:

[0543] The server analyzes emotions based on user input and adjusts the food and menu suggestions.

[0544] Input: User text input, emotion recognition engine

[0545] Output: Food recommendations based on emotional state

[0546] Specific operation: The user inputs their thoughts about the meal in text format, and the data is analyzed by an emotion recognition engine. Based on the user's emotional state (happiness, sadness, satisfaction, etc.), the suggested foods and menu contents are adjusted.

[0547] Step 9:

[0548] The server acquires an image of the delivered food and implements the above-mentioned means to suggest the next order.

[0549] Input: Image of the delivered food

[0550] Output: Next order proposal

[0551] How it works: The user takes a photo of the food they have delivered and uploads it through a dedicated app. The server analyzes the image and performs the steps described above to suggest foods suitable for the next delivery order.

[0552] This allows users to instantly receive healthy meal suggestions that suit their emotional state, even when using a delivery service.

[0553] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0554] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0555] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0556] [Second embodiment]

[0557] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0558] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0559] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0560] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0561] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0562] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0563] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0564] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0565] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0566] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0567] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0568] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0569] System Overview

[0570] The present invention is a system that acquires images of dishes, analyzes the images to identify the type of food, and calculates and displays the nutritional values ​​of each food. It also has the function of acquiring images of a refrigerator, identifying the foods in the refrigerator, and proposing menus based on the identified foods. A specific embodiment of this system is described below.

[0571] Main functions and program processing

[0572] 1. Acquiring and Preprocessing Images of Dishes

[0573] The user takes a photo of the food with their smartphone and launches the dedicated app. The device receives the photo, performs basic pre-processing, and then sends it to the server.

[0574] 2. Image Recognition

[0575] The server uses an image recognition model (e.g., convolutional neural network) to analyze the received images. First, it identifies the type of food in the image, and then uses that information to retrieve the nutritional value of each food from a database. Through this process, it can identify foods such as "salad," "chicken steak," and "rice" that appear in the photo.

[0576] 3. Calculating and displaying nutritional values

[0577] The server calculates the total calories and the total of each nutrient based on the acquired nutritional information, and visualizes the results in a graph format (e.g., bar graph, pie chart) and sends it to the device, allowing the user to easily check it.

[0578] 4. Nutrition Suggestions

[0579] The server compares the user's daily nutrient requirements with their current intake and identifies any nutrient deficiencies. It also suggests specific foods to supplement the deficiencies. For example, it generates a notification such as, "You need 20 grams more dietary fiber. Try adding brown rice or mushrooms." The device displays this notification to the user.

[0580] 5. Refrigerator image acquisition and food ingredient recognition

[0581] The user takes a photo of the inside of their refrigerator and uploads the image using a dedicated app. The device receives the image and sends it to a server. The server uses image recognition technology to identify the food in the refrigerator. Based on this, the system suggests nutritionally balanced meals.

[0582] 6. Daily Records and Management

[0583] The server records the user's daily nutritional intake and weight and displays the data in a calendar format, allowing users to easily understand their health status. It also provides special nutritional management plans for pregnant and dieting users, and suggests nutrients based on those plans.

[0584] Specific examples

[0585] Food image processing

[0586] 1. The user takes a photo of their lunch (salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[0587] 2. The server receives the image and uses image recognition algorithms to identify salad, chicken steak, and rice.

[0588] 3. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[0589] 4. The server graphs the calculation results and sends them to the terminal for display to the user.

[0590] 5. The server identifies the nutrients that are lacking (e.g., dietary fiber) and suggests specific foods (e.g., brown rice).

[0591] Checking the ingredients in the refrigerator and suggesting menus

[0592] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[0593] 2. The server receives the images and uses image recognition algorithms to identify the food in the refrigerator.

[0594] 3. The server will suggest a nutritionally balanced menu (e.g., tomato and cheese salad) based on the identified foods.

[0595] Daily records and health management

[0596] 1. The server records daily nutritional intake and weight and displays them in a calendar format.

[0597] 2. The server provides special nutritional management plans for users who are pregnant or on a diet and makes suggestions based on them.

[0598] This allows users to easily manage their nutritional intake and efficiently maintain their health.

[0599] The processing flow will be explained below.

[0600] Step 1:

[0601] The user takes a picture of the food with their smartphone and launches a dedicated app.

[0602] Step 2:

[0603] The device receives the captured image and performs preprocessing on the image data (e.g., image resizing, format conversion).

[0604] Step 3:

[0605] The terminal transmits the image data after the preprocessing to the server.

[0606] Step 4:

[0607] The server analyzes the received image data and applies an image recognition algorithm (e.g., convolutional neural network) to identify each food item in the image.

[0608] Step 5:

[0609] Based on the identified food information, the server retrieves nutritional data (e.g., calories, protein, fat, carbohydrates) for each food from the database.

[0610] Step 6:

[0611] The server calculates the overall nutritional value of each food item based on the nutritional data it has acquired, by adding up the calories and nutrients of each food item.

[0612] Step 7:

[0613] The server then graphs the results, generating visual bar and pie charts of calories, protein, fat, and carbohydrates, for example.

[0614] Step 8:

[0615] The server sends the graphed nutritional data to the terminal, which displays the data to the user.

[0616] Step 9:

[0617] The server compares your current intake with your daily nutrient needs and identifies any nutrient deficiencies. Specifically, it compares your current intake with the recommended intake.

[0618] Step 10:

[0619] The server will suggest specific foods to supplement the missing nutrients, generating a notification such as "You are missing 20 grams of dietary fiber. Add brown rice or mushrooms."

[0620] Step 11:

[0621] The server generates a notification and sends it to the terminal, which displays it to the user.

[0622] Step 12:

[0623] The user takes a picture of the refrigerator and uploads it to the server using a dedicated app.

[0624] Step 13:

[0625] The server receives the image of the refrigerator and applies image recognition algorithms to identify the food items inside the refrigerator.

[0626] Step 14:

[0627] The server will then suggest nutritionally balanced meals based on the identified food data, such as a tomato and cheese salad and a lettuce and chicken sandwich.

[0628] Step 15:

[0629] The server sends the proposed menu information to the terminal, which displays this information to the user.

[0630] Step 16:

[0631] The server records the user's nutritional intake and weight on a daily basis and generates data to be displayed in calendar format.

[0632] Step 17:

[0633] The server sends the generated daily record data to the terminal, which displays it to the user in a calendar format.

[0634] Step 18:

[0635] The server creates a special nutritional management plan for users who are pregnant or dieting, and makes specific nutritional recommendations based on that plan.

[0636] Step 19:

[0637] The server sends recommendations based on a special nutritional management plan to the terminal, which displays them to the user.

[0638] Example 1

[0639] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0640] Conventional nutrition management systems require users to manually input food data, which is time-consuming and often lacks accuracy. Furthermore, they are unable to efficiently manage ingredients and suggest menus for users, making it difficult to plan nutritionally balanced meals. Furthermore, recording daily nutritional intake and weight is cumbersome, making it difficult to provide effective nutritional advice to specific users, such as those who are pregnant or on a diet.

[0641] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0642] In this invention, the server includes a means for preprocessing food images, a means for identifying the type of food in the image using a convolutional neural network, and a means for retrieving nutritional values ​​from a database and calculating the total nutritional value. This allows a user to simply take a photo of a dish, and the system automatically calculates the nutritional value of each food and suggests missing nutrients and foods that should be consumed in addition, enabling efficient and accurate nutritional management. Furthermore, the system also includes functions for preprocessing refrigerator images, identifying foods in the refrigerator using image recognition technology, and suggesting menus based on the identified foods, as well as a function for recording the user's nutritional intake and weight daily and displaying them in calendar format, thereby enabling comprehensive dietary and health management.

[0643] "Means for acquiring images of food" refers to the function that allows a user to take a photo of food using an input device such as a smartphone or camera and import it into the system.

[0644] "Means for preprocessing acquired images" refers to processing such as noise removal, resolution adjustment, and color correction performed on received images to improve the accuracy of image analysis.

[0645] The "means for transmitting preprocessed images to a server" refers to a function for transferring image data for which preprocessing has been completed to a server via the Internet.

[0646] "Means for identifying each food type in an image using a convolutional neural network" refers to the ability to use a convolutional neural network, a type of deep learning technology, to identify each food object in a preprocessed image and classify its type.

[0647] "Means for retrieving the nutritional value of identified foods from a database and calculating the total nutritional value" refers to a function for retrieving nutritional value information for each identified food from a database and calculating the total calories and the total value of each nutrient.

[0648] "Means for displaying calculated nutritional values ​​in a graph" refers to a function that visualizes and displays the calculated nutritional values ​​in the form of a bar graph, pie chart, etc., so that the user can easily understand them intuitively.

[0649] "Means to identify nutrients that are lacking compared to the daily required nutrients and suggest additional foods that should be taken" refers to a function that compares the user's daily nutritional intake standard with their current intake status, identifies nutrients that are lacking, and suggests specific foods to make up for the deficiency.

[0650] "Means for acquiring images of the refrigerator" refers to a function that allows a user to take a photo of the inside of the refrigerator using an input device such as a smartphone or camera and import it into the system.

[0651] "Means for identifying food items in a refrigerator using image recognition technology" refers to the use of deep learning or other image recognition algorithms to identify each food object in a photograph of the refrigerator and determine its type and quantity.

[0652] "Means for suggesting nutritionally balanced menus based on identified foods" refers to a function that suggests nutritionally balanced meal menus to the user based on the foods present in the refrigerator.

[0653] "Means for recording the user's nutritional intake and weight on a daily basis and displaying it in calendar format" refers to a function that records the user's daily nutrition intake and weight fluctuations and visually displays the data in calendar format.

[0654] "A means of providing specific nutritional management plans for users who are pregnant or on a diet and making nutritional suggestions based on those plans" refers to the function of creating appropriate nutritional management plans for users with specific health conditions or goals, and making specific nutritional supplement suggestions based on those plans.

[0655] System Overview

[0656] This system captures images of food and the contents of a refrigerator, identifies the type of food through image analysis, and calculates and displays its nutritional value. The server uses image recognition technology to allow users to easily manage their diet and nutrition via their smartphone. It also supports health management by recording the user's daily nutritional intake and weight and visualizing them in a calendar format.

[0657] Main functions and program processing

[0658] Acquiring and preprocessing food images

[0659] A user takes a photo of a dish using a smartphone and launches a dedicated app. The device receives the captured image and performs preprocessing such as noise reduction, resolution adjustment, and color correction. This preprocessing is performed using image analysis software (e.g., OpenCV). The preprocessed image is then sent to the server using a secure protocol (e.g., HTTPS).

[0660] Image Recognition

[0661] The server receives the image and uses a convolutional neural network (e.g., using TensorFlow or PyTorch) to identify the type of food in the image. This identifies foods such as "salad," "chicken steak," and "rice." The server then retrieves the nutritional value information for each food from a database (e.g., an SQL database).

[0662] Nutritional Value Calculation and Labeling

[0663] The server calculates the total calories and the total of each nutrient based on the acquired nutritional information. These calculation results are converted into graphs (bar graphs, pie charts, etc.) using a visualization tool (e.g., Matplotlib), and then sent to the terminal to be displayed intuitively to the user.

[0664] Nutrition Suggestions

[0665] The server compares the user's daily nutrient needs with their current intake. It then identifies nutrient deficiencies and suggests specific foods to fill the gaps. For example, it generates a notification such as, "You need 20 more grams of dietary fiber. Try adding brown rice or mushrooms." The device then displays this notification to the user.

[0666] Refrigerator image acquisition and food ingredient recognition

[0667] The user takes a photo of the inside of the refrigerator with their smartphone and uploads the image using a dedicated app. The device performs preprocessing and sends the image to the server. The server uses an image recognition algorithm (e.g., YOLOv4 or EfficientDet) to identify the foods in the refrigerator and uses that information to suggest nutritionally balanced meals.

[0668] Daily records and management

[0669] The server records the user's daily nutritional intake and weight and stores this data in a database. A view displaying the recorded data in a calendar format is generated and sent to the device. This allows the user to grasp their own health information at a glance. It also provides special nutritional management plans (e.g., for pregnancy or dieting) and suggests nutrients that are suitable for the user.

[0670] Specific examples

[0671] A concrete example of food image processing

[0672] 1. The user takes a photo of their lunch (salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[0673] 2. The server receives the image and uses an image recognition algorithm (e.g., TensorFlow's convolutional neural network) to identify salad, chicken steak, and rice.

[0674] 3. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[0675] 4. The server graphs the calculation results and sends them to the terminal for display to the user.

[0676] 5. The server identifies the nutrients that are lacking (e.g., dietary fiber) and suggests specific foods (e.g., brown rice).

[0677] Checking the ingredients in the refrigerator and suggesting a menu

[0678] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[0679] 2. The server receives the images and uses an image recognition algorithm (e.g., YOLOv4) to identify the food in the refrigerator.

[0680] 3. The server will suggest a nutritionally balanced menu (e.g., tomato and cheese salad) based on the identified foods.

[0681] Examples of daily records and health management

[0682] 1. The server records daily nutritional intake and weight and displays them in a calendar format.

[0683] 2. The server provides special nutritional management plans for users who are pregnant or on a diet and makes suggestions based on them.

[0684] By combining all of the above functions, users can easily and efficiently manage their nutritional intake and health. This system will be an important tool that contributes to raising consumer health awareness.

[0685] Example prompts to be input to the generative AI model

[0686] "Analyze this food photo to identify the types of foods it contains and their nutritional value."

[0687] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0688] Step 1: The user takes a picture of the food and launches the dedicated app.

[0689] The user takes a photo of the food using the camera function of their smartphone, opens the dedicated app, selects the image, and presses a button to start processing. The input of this step is the image taken by the camera, and the output is the image imported into the dedicated app.

[0690] Step 2: The device preprocesses the image.

[0691] The device performs preprocessing on the captured image, such as noise removal, resolution adjustment, and color correction, to improve the accuracy of image analysis. Specifically, it uses image processing libraries such as OpenCV. The input is the image taken by the user, and the output is a clear, preprocessed image.

[0692] Step 3: The device sends the preprocessed image to the server.

[0693] The device sends the preprocessed image data to the server using the HTTPS protocol. Data is encrypted during transmission to ensure security. The input is the preprocessed image, and the output is the image sent to the server.

[0694] Step 4: The server receives the image and applies the convolutional neural network.

[0695] The server stores the received images and begins analyzing them using TensorFlow or PyTorch, identifying each food type using a convolutional neural network. The input for this step is the image sent to the server, and the output is a list of each identified food object.

[0696] Step 5: The server retrieves the nutritional value of the food from the database.

[0697] For each food item identified as a result of the analysis, the server retrieves nutritional information from a database (e.g., an SQL database). The input is a list of identified foods, and the output is the nutritional information for each food item.

[0698] Step 6: The server calculates the total nutritional value.

[0699] The server calculates the total calories and the total of each nutrient based on the nutritional value information of each food. The input is the nutritional value information of each food, and the output is the total nutritional value.

[0700] Step 7: The server graphs the calculation results and sends them to the terminal.

[0701] The server converts the calculation results into graphs (such as bar graphs or pie charts) using a visualization tool (e.g., Matplotlib) and sends them to the terminal. The terminal receives them and displays them to the user. The input is the total nutritional value, and the output is the graphed data.

[0702] Step 8: Your server will provide you with nutrients you are lacking and suggestions for adding more.

[0703] The server compares the user's daily nutrient needs with their current intake to identify any nutrient deficiencies, and then suggests specific foods to supplement the deficiencies. The input is the user's intake and nutritional criteria, and the output is a notification of the recommendations.

[0704] Step 9: The user takes a picture of the inside of the refrigerator and uploads the image.

[0705] The user takes a picture of the inside of the refrigerator with their smartphone and uploads the image to a dedicated app. The input is the image of the inside of the refrigerator, and the output is the image uploaded to the app.

[0706] Step 10: The device sends the image of the refrigerator to the server.

[0707] The device performs preprocessing and then sends the image to the server. The input is the preprocessed image and the output is the image sent to the server.

[0708] Step 11: The server applies image recognition algorithms to identify the food in the refrigerator.

[0709] The server uses image recognition techniques such as YOLOv4 and EfficientDet to identify the foods in the refrigerator. The input is an image of the refrigerator sent to the server, and the output is a list of identified foods.

[0710] Step 12: The server suggests a menu.

[0711] The server creates a nutritionally balanced menu based on the identified food information and proposes it to the user. The input is a list of foods in the refrigerator, and the output is a proposed menu.

[0712] Step 13: The server records the user's nutritional intake and weight and displays them in a calendar format.

[0713] The server records the user's daily nutritional intake and weight and displays the data in a calendar format. The input is the user's intake information and weight data, and the output is a calendar display.

[0714] Step 14: The server provides a special nutrition plan.

[0715] The server provides special nutritional management plans for pregnant and dieting users and makes specific nutrition recommendations based on those plans. The input is each user's specific conditions, and the output is a notification of recommendations based on the plan.

[0716] (Application example 1)

[0717] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0718] Currently, it is important to eat meals that are conscious of health management and nutritional balance, but it is not easy to actually understand the contents of one's daily diet and manage it appropriately. In addition, when using delivery services, there are few ways to understand the nutritional value of the food, making it difficult for customers to make healthy choices. Furthermore, it is time-consuming to plan a menu that effectively uses the ingredients in the refrigerator. To solve these issues, a system that provides more detailed nutritional information and makes healthy meal suggestions is needed.

[0719] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0720] In this invention, the server includes means for acquiring images of dishes, means for analyzing the acquired images and identifying the type of each food in the images, means for acquiring the nutritional value of the identified foods from a database and calculating the total nutritional value, means for displaying the calculated nutritional value in a graph, means for identifying nutrients that are lacking by comparing them with the daily nutrient requirement and suggesting additional foods that should be consumed, means for a user to upload an image of a dish when ordering delivery and display the nutritional value information in real time, means for automatically calculating nutritional information for menus provided by the delivery company and suggesting it to the customer, means for recognizing the contents of the customer's refrigerator and suggesting healthy menus based on that, and means for managing the user's daily nutritional intake and suggesting an optimal meal plan.

[0721] This allows users to use information about ingredients in their refrigerators at home when ordering delivery to plan nutritionally balanced meals, enabling them to make healthy meal choices. It also allows for efficient and effective daily nutrition management.

[0722] "Cuisine" refers to food that has been prepared by cooking and that has been modified to improve its nutritional value or palatability.

[0723] "Capturing an image" refers to the process of capturing visual information as digital data using a device such as a camera or scanner.

[0724] "Analyzing images" refers to the process of applying pattern recognition and machine learning algorithms to captured image data to extract and identify specific information.

[0725] "Identifying the type of food" refers to classifying objects recognized from acquired image data into specific food categories.

[0726] "Retrieving nutritional values ​​from a database" means retrieving the specific calculations and nutritional components of a food from a pre-stored information source.

[0727] "Calculating total nutritional value" refers to the act of adding up the individual nutritional components of a specified food and calculating their sum.

[0728] "Displaying in a graph" means presenting calculated numerical information or data in the form of a bar graph, pie chart, or the like to make it visually easier to understand.

[0729] "Nutrient identification" means diagnosing whether you have a deficiency or excess of certain key components in your daily diet or nutrition plan.

[0730] "Suggesting foods to consume" refers to the act of recommending specific foods or ingredients to supplement missing nutrients.

[0731] "Delivery order" refers to the act of a user making a request to have food delivered by a delivery company.

[0732] "Recognizing the contents of a refrigerator" means performing image analysis to identify the types and quantities of food and ingredients stored in the refrigerator.

[0733] "Suggesting a menu" means creating and presenting a meal plan that takes nutritional balance into consideration and combines multiple dishes and foods.

[0734] "Managing daily nutritional intake" refers to a user recording how much nutrients they consume each day and adjusting their health and eating habits based on that information.

[0735] "Proposing the optimal meal plan" means designing and recommending the best meal content based on an individual's nutritional status and lifestyle.

[0736] To implement the present invention, the system operates as follows.

[0737] First, the user takes a photo of the food with their smartphone and launches the dedicated app. The device receives the photo, performs basic preprocessing, and then sends it to the server. This preprocessing uses an image processing library such as OpenCV.

[0738] The server then uses a pre-trained image recognition model (e.g., a TensorFlow / Keras convolutional neural network) to analyze the received image. The image recognition model identifies the type of food and uses that information to retrieve the nutritional values ​​of each food item from a database. At this stage, the necessary nutritional value data is obtained based on the information extracted from the image, such as "salad" or "chicken steak."

[0739] The server then calculates the total calories and the total of each nutrient based on the acquired nutritional information. The calculation results are visualized in a graph format (e.g., bar graph, pie chart) and sent to the device. This allows the user to check the nutritional value of the food they have photographed at a glance.

[0740] The server also compares the user's daily nutrient needs with their current intake to identify any nutrient deficiencies. It also has the ability to suggest specific foods to fill the gaps. For example, it generates a notification that says, "You need 20 more grams of dietary fiber. Try adding brown rice or mushrooms." and displays it on the device.

[0741] Additionally, the system allows users to take a photo of the inside of their refrigerator and upload the image using a dedicated app. The device receives the image and sends it to a server, which uses image recognition technology to identify the foods in the refrigerator. Based on the results, the system suggests nutritionally balanced meals.

[0742] The server also has a function to record the user's daily nutritional intake and weight and display them in a calendar format, allowing users to easily understand their health status. It also provides special nutritional management plans for users who are pregnant or on a diet, and makes suggestions based on those plans.

[0743] When using a delivery service, users can upload a photo of the food they want to order and view its nutritional information in real time. The system also includes a function to automatically calculate the nutritional information of the menu items offered by the delivery service and provide suggestions to customers. For example, users can be provided with nutritional information for delivery foods such as pizza and burgers.

[0744] Specific examples

[0745] 1. A user uploads a photo of a pizza.

[0746] 2. The server analyzes the image, identifies the pizza's ingredients, and retrieves its nutritional information from a database.

[0747] 3. The server calculates the nutritional value and displays it in a graph.

[0748] 4. The server will identify any nutrients that are lacking and make suggestions such as salads.

[0749] Example prompt sentence:

[0750] Upload an image of the food you want to order and we'll display nutritional information in real time and suggest meals that take into account the ingredients in your refrigerator.

[0751] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0752] Step 1:

[0753] The user takes a photo of the food with their smartphone and launches a dedicated app.

[0754] Input: Food image taken with a smartphone

[0755] Specific operation: The user takes a photo of the food and presses the "Upload image" button on the dedicated app.

[0756] Step 2:

[0757] The device receives the captured photo, performs some basic pre-processing, and then sends it to the server.

[0758] Input: Photographed food image

[0759] Data processing: Using the OpenCV library, preprocessing such as image resizing and noise reduction is performed.

[0760] Specific operation: After processing the image, the device uploads it to the server.

[0761] Step 3:

[0762] The server uses a pre-trained image recognition model to analyze the images it receives.

[0763] Input: Preprocessed food images

[0764] Data computation: A convolutional neural network (CNN) using TensorFlow / Keras is used to identify the type of food in the image.

[0765] What it does: The server runs a CNN model to classify the food in the image.

[0766] Step 4:

[0767] The server retrieves the nutritional values ​​of the identified foods from the database and calculates the total nutritional value.

[0768] Input: List of identified foods

[0769] Data acquisition: Obtain the nutritional value information of each food from the nutritional value database.

[0770] Data calculation: The nutritional value of each food is added up to calculate the total calories and the total of each nutrient.

[0771] Specific operation: The server issues a database query and calculates nutritional values ​​based on the retrieved data.

[0772] Step 5:

[0773] The server generates data that displays the calculated nutritional values ​​in a graph and sends it to the terminal.

[0774] Input: Totaled nutritional data

[0775] Data manipulation: Formatting data into visually understandable formats such as bar graphs and pie charts.

[0776] Specific operation: The server uses a graph generation tool to create visual data and transfers it to the terminal.

[0777] Step 6:

[0778] The terminal displays the graph to the user.

[0779] Input: Graph data sent from the server

[0780] What it does: A dedicated app displays graphs and provides visual feedback to the user.

[0781] Step 7:

[0782] The server compares the user's daily nutrient requirements with their current intake and identifies any nutrients that are lacking.

[0783] Input: User's nutritional intake data and nutrient requirements

[0784] Data calculation: Calculates nutrient deficiencies based on current intake.

[0785] What it does: The server analyzes current nutritional intake data and identifies nutrients that are lacking.

[0786] Step 8:

[0787] The server suggests specific foods to supplement the missing nutrients and sends a notification to the device.

[0788] Input: Missing nutrient data

[0789] Data generation: Generate a list of foods to supplement missing nutrients.

[0790] What happens: The server selects food suggestions and generates a notification with that information.

[0791] Step 9:

[0792] Users take a photo of the inside of their refrigerator and upload the image using a dedicated app.

[0793] Input: Photo of the inside of a refrigerator

[0794] What happens: The user takes a photo of the refrigerator and presses the upload button in the app.

[0795] Step 10:

[0796] The device receives a photo of the inside of the refrigerator and sends it to the server.

[0797] Input: Photo of the inside of a refrigerator

[0798] Specific operation: The device receives the photo, preprocesses it, and uploads it to the server.

[0799] Step 11:

[0800] The server uses image recognition technology to identify food in the refrigerator and suggest nutritionally balanced meals.

[0801] Input: Photo of the inside of a refrigerator

[0802] Data Computing: Using image recognition algorithms to classify and identify food items in the refrigerator.

[0803] Specific operation: The server performs image analysis and generates a menu based on the food information in the refrigerator.

[0804] Step 12:

[0805] The server records the user's daily nutritional intake and weight and displays them in calendar format.

[0806] Input: User nutritional intake and weight data

[0807] Data processing: Visually formatting data in a calendar format.

[0808] Specific operation: The server organizes daily data and provides it to the device as a calendar display.

[0809] Step 13:

[0810] The server provides special nutritional management plans for pregnant or dieting users and suggests nutrients based on the plans.

[0811] Input: User's special health status data (e.g., pregnant, dieting)

[0812] Data calculation: Create a special nutritional management plan and generate nutritional recommendations based on it.

[0813] Specific operation: The server creates a special plan and sends a notification containing the proposal to the device.

[0814] Example prompt sentence:

[0815] Upload an image of the food you want to order and we'll display nutritional information in real time and suggest meals that take into account the ingredients in your refrigerator.

[0816] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0817] System Overview

[0818] The present invention combines a system that acquires images of dishes, analyzes the images to identify the type of food, calculates and displays the nutritional value, and an emotion engine that recognizes the user's emotions. This system not only provides nutritional management but also makes meal suggestions based on the user's emotional state. Specific embodiments of this system are described below.

[0819] Main functions and program processing

[0820] 1. Acquiring and Preprocessing Images of Dishes

[0821] The user takes a photo of the food with their smartphone and launches the dedicated app. The device receives the image, preprocesses it (e.g., resizes the image, converts the format), and then sends it to the server.

[0822] 2. Image Recognition

[0823] The server uses an image recognition algorithm (e.g., convolutional neural network) to analyze the received images. It identifies the type of food in the image and uses that information to retrieve the nutritional value of each food from a database. Through this process, it can identify foods such as "salad," "chicken steak," and "rice" that appear in the photo.

[0824] 3. Calculating and displaying nutritional values

[0825] The server calculates the total calories and the total of each nutrient based on the acquired nutritional information, and visualizes the results in a graph format (e.g., bar graph, pie chart) and sends it to the device, allowing the user to easily check it.

[0826] 4. Nutrition Suggestions

[0827] The server compares the user's daily nutrient requirements with their current intake and identifies any nutrient deficiencies. It also suggests specific foods to supplement the deficiencies. For example, it generates a notification saying, "You need 20 grams more dietary fiber. Try adding brown rice or mushrooms." The device displays this notification to the user.

[0828] 5. Emotion Recognition by Emotion Engine

[0829] The server uses an emotion recognition engine to analyze the user's emotions, specifically identifying emotions such as joy, excitement, and satisfaction based on the food image and user input.

[0830] 6. Adjusting offers based on emotions

[0831] The server may tailor suggested additional foods or meal options based on the identified emotional state, for example, if the user is expressing a feeling of joy, more colorful dishes may be suggested to enhance enjoyment.

[0832] 7. Refrigerator image acquisition and food ingredient recognition

[0833] The user takes a photo of the inside of their refrigerator and uploads the image to a server using a dedicated app. The server then uses image recognition technology to identify the food in the refrigerator. Based on this information, the system suggests nutritionally balanced meals.

[0834] 8. Daily Records and Management

[0835] The server records the user's daily nutritional intake and weight and displays the data in a calendar format, allowing users to easily understand their health status. It also provides special nutritional management plans for pregnant and dieting users, and suggests nutrients based on those plans.

[0836] Specific examples

[0837] Food image processing

[0838] 1. The user takes a photo of their lunch (salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[0839] 2. The server receives the image and uses image recognition algorithms to identify salad, chicken steak, and rice.

[0840] 3. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[0841] 4. The server graphs the calculation results and sends them to the terminal for display to the user.

[0842] 5. The server identifies the nutrients that are lacking (e.g., dietary fiber) and suggests specific foods (e.g., brown rice).

[0843] 6. The server uses an emotion engine to analyze the user's emotions from the photos and input data and adjust the suggestions.

[0844] Checking the ingredients in the refrigerator and suggesting menus

[0845] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[0846] 2. The server receives the images and uses image recognition algorithms to identify the food in the refrigerator.

[0847] 3. The server will suggest a nutritionally balanced menu (e.g., tomato and cheese salad) based on the identified foods.

[0848] 4. The server analyzes the user's emotions through an emotion recognition engine and adjusts the menu accordingly.

[0849] Daily records and health management

[0850] 1. The server records daily nutritional intake and weight and displays them in a calendar format.

[0851] 2. The server provides special nutritional management plans for users who are pregnant or on a diet and makes suggestions based on them.

[0852] 3. The server tailors daily meal and activity suggestions based on the user's emotional state.

[0853] This allows users to not only manage their nutrition, but also receive meal suggestions based on their emotional state, allowing them to enjoy a richer diet.

[0854] The processing flow will be explained below.

[0855] Step 1:

[0856] The user takes a picture of the food with their smartphone and launches a dedicated app.

[0857] Step 2:

[0858] The device receives the captured image and performs preprocessing on the image data (e.g., image resizing, format conversion).

[0859] Step 3:

[0860] The terminal transmits the preprocessed image data to the server.

[0861] Step 4:

[0862] The server analyzes the received image data and uses an image recognition algorithm (e.g., convolutional neural network) to identify each food item in the image.

[0863] Step 5:

[0864] Based on the identified food information, the server retrieves nutritional data (e.g., calories, protein, fat, carbohydrates) for each food from the database.

[0865] Step 6:

[0866] The server calculates the overall nutritional value of each food based on the nutritional data of each food item, adding up the calories and nutrients of each food item.

[0867] Step 7:

[0868] The server then graphs the results, generating visual bar and pie charts of calories, protein, fat, and carbohydrates.

[0869] Step 8:

[0870] The server sends the graphed nutritional data to the terminal, which displays it to the user.

[0871] Step 9:

[0872] The server compares your current intake with your daily nutrient needs to identify any nutrient deficiencies. It also compares your current intake with the recommended intake.

[0873] Step 10:

[0874] The server will suggest specific foods to supplement the missing nutrients, generating a notification such as "You're missing 20 grams of dietary fiber. Add brown rice or mushrooms."

[0875] Step 11:

[0876] The server sends the generated notification to the terminal, which displays it to the user.

[0877] Step 12:

[0878] The server uses an emotion recognition engine to analyze the user's emotions based on the food images and user input, identifying emotions such as joy, excitement, and satisfaction.

[0879] Step 13:

[0880] The server adjusts the suggested additional foods and menu items based on the identified emotional state: if the emotional state is "joy," it suggests more elaborate dishes; if the emotional state is "excited," it suggests new dishes and trending foods.

[0881] Step 14:

[0882] The user takes a photo of the inside of the refrigerator and uploads the image to the server using a dedicated app.

[0883] Step 15:

[0884] The server analyzes the received image of the refrigerator and applies image recognition algorithms to identify each food item inside.

[0885] Step 16:

[0886] The server will then suggest nutritionally balanced meals based on the identified food information, for example, "a tomato and cheese salad, lettuce and chicken sandwich."

[0887] Step 17:

[0888] The server sends the proposed menu information to the terminal, which displays it to the user.

[0889] Step 18:

[0890] The server records the user's nutritional intake and weight on a daily basis and generates data to be displayed in calendar format.

[0891] Step 19:

[0892] The server transmits the generated daily record data to the terminal, which displays it to the user in a calendar format.

[0893] Step 20:

[0894] The server creates a special nutritional management plan for users who are pregnant or dieting, and makes specific nutritional recommendations based on that plan.

[0895] Step 21:

[0896] The server sends recommendations based on a special nutritional management plan to the terminal, which displays them to the user.

[0897] Step 22:

[0898] The server tailors daily food and activity suggestions based on the user's emotional state: if the emotional state is "sad," it suggests specific foods and activities to lighten the mood.

[0899] Example 2

[0900] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0901] Today's consumers are increasingly interested in eating a nutritionally balanced diet and maintaining their health. However, it is difficult to understand which foods contain which nutrients and manage their nutrient intake in their daily lives. Furthermore, there is a lack of systems that provide dietary recommendations based not only on nutritional value but also on the user's emotions. Given this situation, it is necessary to efficiently and accurately manage nutrition and provide recommendations that respond to the user's emotions.

[0902] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0903] In this invention, the server

[0904] A means for acquiring an image of a dish;

[0905] means for pre-processing the acquired images;

[0906] means for analyzing the preprocessed image data and identifying each food type in the image;

[0907] means for retrieving the nutritional values ​​of the identified foods from a database;

[0908] a means for calculating total nutritional value;

[0909] A means for displaying the calculated nutritional values ​​in a graph;

[0910] A method to identify nutrients that are lacking compared to the daily required nutrients and suggest additional foods that should be taken in,

[0911] means for recognizing a user's emotion and tailoring suggestions based on the emotion;

[0912] A means to acquire, preprocess, and analyze images of the refrigerator, identify the foods in the refrigerator, and propose nutritionally balanced meals.

[0913] A means for recording the user's daily nutritional intake and weight and displaying them in a calendar format;

[0914] A means of providing specific nutritional management plans for pregnant or dieting users and making nutritional suggestions based on those plans

[0915] This allows for efficient and accurate nutritional management and meal suggestions that reflect the user's emotions.

[0916] "Food image" is visual data about food that is photographed or acquired by the user.

[0917] "Preprocessing" refers to initial processing of captured or acquired images, such as adjusting their size or converting their format, to make them easier to analyze.

[0918] "Image recognition algorithm" refers to a computational method used to identify specific objects or features within an image.

[0919] "Food type" identifies the various ingredients and food items contained in the photographed food image.

[0920] "Nutritional value" is data that indicates the amount of energy and various nutrients (e.g., protein, lipids, carbohydrates, vitamins, minerals, etc.) contained in food.

[0921] A "database" is a collection of information in which nutritional information about foods is systematically organized and stored.

[0922] "Daily Nutrient Requirements" refers to the standard of nutrients that a user should consume each day.

[0923] "Additional foods" refers to specific ingredients or dishes recommended to supplement missing nutrients.

[0924] An "emotion recognition engine" refers to technology for analyzing a user's emotional state, specifically recognizing a user's emotions such as joy, excitement, and satisfaction from images and input data.

[0925] "Refrigerator image" refers to visual data of the inside of a refrigerator photographed or acquired by a user.

[0926] A "nutritional balanced menu" refers to a meal plan that is nutritionally balanced based on specific food information.

[0927] "Calendar format" refers to a method of organizing data by date and displaying it in a visually easy-to-understand format.

[0928] "Nutrition Management Plan" refers to a plan designed to support optimal nutritional intake for a user in a specific situation, such as during pregnancy or while dieting.

[0929] System Overview

[0930] The present invention combines a system that acquires images of dishes, analyzes the images to identify the type of food, calculates and displays the nutritional value of each food, and an emotion engine that recognizes the user's emotions. This system provides meal suggestions based on the user's nutritional balance and emotional state when managing their daily diet.

[0931] Hardware and software used

[0932] The user uses a device such as a smartphone. A dedicated app is installed on the device, and it is equipped with a pre-processing function for captured images. The pre-processed image data is sent to a server via a network. The following software is running on the server:

[0933] Image recognition algorithms (e.g., convolutional neural networks)

[0934] Database Management System (DBMS)

[0935] Emotion Recognition Engine

[0936] Visualization tools (e.g., D3.js, Chart.js)

[0937] Program processing overview

[0938] The user takes a photo of a dish and launches a dedicated app. The device receives the image and performs preprocessing such as resizing and format conversion. The preprocessed image data is then sent over the network to a server. The server uses an image recognition algorithm to identify the type of food in the image and retrieves the nutritional value of each food item from a database. Based on the retrieved nutritional value information, the server calculates and displays the total calories and the sum of each nutrient.

[0939] The server then compares the user's daily nutrient needs with their current intake to identify any nutrient deficiencies. When suggesting specific foods, an emotion recognition engine analyzes the user's emotional state and adjusts the suggestions accordingly. For example, if the user expresses joy, the server will suggest more colorful dishes to enhance the enjoyment.

[0940] Users can also take photos of the inside of their refrigerator and upload them using a dedicated app. The server uses an image recognition algorithm to identify the food items in the refrigerator and suggests menu items based on that information. In this case, an emotion recognition engine also analyzes the user's emotions and offers the most appropriate menu.

[0941] The server also records the user's daily nutritional intake and weight data and displays it in a calendar format. For users who are pregnant or on a diet, the server provides special nutritional management plans and suggests nutrients based on those plans.

[0942] Specific examples

[0943] Food image processing

[0944] 1. The user takes a photo of their lunch (e.g., salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[0945] 2. The device preprocesses the image and sends the formatted image to the server.

[0946] 3. The server uses image recognition algorithms to identify each food item.

[0947] 4. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[0948] 5. The server graphs the calculation results and sends them to the terminal for display to the user.

[0949] 6. The server will identify any missing nutrients (e.g., dietary fiber) and suggest foods to add (e.g., brown rice).

[0950] 7. The server uses an emotion engine to analyze the user's emotions from the images and input data and adjust the suggestions.

[0951] Checking the ingredients in the refrigerator and suggesting menus

[0952] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[0953] 2. The device preprocesses the image and sends the formatted image to the server.

[0954] 3. The server uses image recognition algorithms to identify the food in the refrigerator.

[0955] 4. The server will suggest a nutritionally balanced meal based on the identified foods (e.g., tomato and cheese salad).

[0956] 5. The server analyzes the user's emotions through an emotion recognition engine and adjusts the menu accordingly.

[0957] Example prompts to be input to the generative AI model

[0958] Analyze the following food images, identify the type of food for each, calculate the nutritional value of each food, and generate a sentence suggesting to the user what nutrients they may be missing:

[0959] Food image: [Image link]

[0960] It should also analyze the user's emotional state and provide meal suggestions based on those emotions.

[0961] Through such a system, users can receive nutritional management and emotionally-based meal suggestions, enabling them to enjoy a healthier and more fulfilling diet.

[0962] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0963] Step 1:

[0964] The user takes a photo of a dish with their smartphone and launches the dedicated app. The user then uploads this image to the dedicated app. The input is the photo of the dish, and the output is image data for preprocessing.

[0965] Step 2:

[0966] The device receives the captured image and performs preprocessing such as image resizing and format conversion. The input is the image uploaded by the user, and the output is the preprocessed image data. This preprocessing enables efficient image analysis.

[0967] Step 3:

[0968] The terminal sends the preprocessed image data to the server via the network. The input is the preprocessed image data, and the output is the transmission of the image data to the server.

[0969] Step 4:

[0970] The server uses an image recognition algorithm (e.g., convolutional neural network) to analyze the received images. The input is the preprocessed image data, and the output is the analyzed data including the type of each food. Based on this analyzed data, the server retrieves the nutritional value of each food from the database.

[0971] Step 5:

[0972] The server calculates the total calories and total of each nutrient based on the acquired nutritional information. The input is the nutritional value data of the identified food, and the output is the total nutritional value calculation. This calculation includes calories, proteins, fats, carbohydrates, and other major nutrients.

[0973] Step 6:

[0974] The server visualizes the calculation results in graph format (e.g., bar graph, pie chart). The input is the calculation result of total nutritional value, and the output is the visualized graph data. The graph makes it easier for users to intuitively understand the nutritional information.

[0975] Step 7:

[0976] The server sends the visualized data to the terminal and displays it to the user. The input is the visualized graph data, and the output is the graph displayed on the user's terminal.

[0977] Step 8:

[0978] The server compares the user's daily required nutrients with their current intake and identifies any nutrient deficiencies. The input is the daily required nutrient data and current intake data, and the output is the results of identifying the nutrient deficiencies.

[0979] Step 9:

[0980] The server suggests specific foods to supplement the missing nutrients. For example, it generates a notification saying, "You need 20 more grams of dietary fiber. Try adding brown rice or mushrooms." The input is the result of identifying the missing nutrients, and the output is a notification with specific food suggestions.

[0981] Step 10:

[0982] The terminal displays this notification to the user.,The input is the food suggestion notification from the server, and,the output is the suggestion content displayed on the user's,terminal.

[0983] Step 11:

[0984] The server uses an emotion recognition engine to analyze the user's emotions. The input is food images and user input data, and the output is data indicating the user's emotional state. Specifically, it generates data indicating the user's emotions, such as joy, excitement, and satisfaction.

[0985] Step 12:

[0986] The server adjusts the suggested additional foods and meal options based on the identified emotional state. The input is the user's emotional state data, and the output is the adjusted food and meal options.

[0987] Step 13:

[0988] The user takes a photo of the inside of the refrigerator and uploads the image to the server using a dedicated app. The input is the photo of the inside of the refrigerator, and the output is the transmission of image data to the server.

[0989] Step 14:

[0990] The server uses image recognition technology to identify the food in the refrigerator. The input is image data of the inside of the refrigerator, and the output is data of the identified food in the refrigerator.

[0991] Step 15:

[0992] The server then proposes a nutritionally balanced menu based on the identified foods. The input is the food data in the refrigerator, and the output is a nutritionally balanced menu proposal.

[0993] Step 16:

[0994] The server records daily nutritional intake and weight data and displays them in a calendar format. The input is the user's daily nutritional intake and weight data, and the output is the display data in a calendar format.

[0995] Step 17:

[0996] The server provides a special nutritional management plan for pregnant or dieting users and makes nutritional recommendations based on the plan. The input is the special nutritional management plan data, and the output is the nutritional recommendations based on the plan.

[0997] (Application example 2)

[0998] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0999] In modern society, nutritional management and emotional care are important for busy people to maintain a healthy diet. However, achieving this requires a lot of time and effort. Furthermore, when using a delivery service, it is not easy to determine whether the ordered food is healthy. Furthermore, there are no systems that can suggest meals based on the user's emotional state. Therefore, there is a need for a system that supports users' nutritional management and suggests optimal meals based on the user's emotional state.

[1000] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring food images, means for analyzing the acquired images and identifying the type of each food in the images, means for acquiring the nutritional values ​​of the identified foods from a database and calculating the total nutritional value, means for displaying the calculated nutritional values ​​in a graph, means for identifying nutrients that are deficient compared to the daily nutrient requirement and suggesting additional foods to be consumed, means for analyzing emotions based on user input and adjusting the suggested foods and menu contents, and means for acquiring images of delivered dishes and implementing the above means to suggest the next order. This allows users to instantly receive healthy meal suggestions that are appropriate for their emotional state even when using a delivery service.

[1001] The "means for acquiring an image of a dish" refers to a method or device that allows a user to take a photo of a dish and input it into the system.

[1002] The "means for analyzing the image and identifying the type of each food in the image" refers to an algorithm or program for analyzing the acquired image of food and identifying the type of each food.

[1003] "Means for obtaining the nutritional value of identified foods from a database and calculating the total nutritional value" refers to a method or device that obtains nutritional information for each identified food from a database and adds up the individual nutrients to calculate the total nutritional value.

[1004] The "means for displaying the calculated nutritional value in a graph" refers to a method or device for displaying the calculated nutritional value in a graph format to make it easier to understand visually.

[1005] "Means for identifying nutrients that are lacking by comparing with the daily required nutrients and suggesting foods that should be taken additionally" refers to a method or device that compares the user's daily required nutrients with their current intake status, identifies nutrients that are lacking, and suggests foods that should be taken to make up for the lack.

[1006] "Means for analyzing emotions based on user input and adjusting suggested food and menu items" refers to a method or device that analyzes a user's emotions based on text and other data entered by the user and optimizes suggested food and menu items according to the results.

[1007] "Means for taking images of delivered food and implementing the above-mentioned means to make suggestions for the next order" refers to a method or device for taking and taking images of delivered food, analyzing them, and making suggestions that will be useful for the next order.

[1008] This invention combines a system that acquires images of food, analyzes them to identify the type of food, calculates and displays the nutritional value, and an emotion engine that recognizes the user's emotions. This system goes beyond nutritional management and can also make meal suggestions based on the user's emotional state.

[1009] System configuration

[1010] The system consists of the following main components:

[1011] 1. Method for acquiring food images: The user takes a photo of the food with their smartphone and launches a dedicated app.

[1012] 2. Image analysis and food identification: The server receives the captured images and uses image recognition algorithms to identify the type of each food item.

[1013] 3. Means for obtaining and calculating nutritional value: The server obtains the nutritional value of the identified food from the database and calculates it.

[1014] 4. Nutritional value display means: The calculated nutritional value is visualized in graph form and sent to the user's device.

[1015] 5. Identifying nutrient deficiencies and suggesting foods: Identifying nutrients that are lacking compared to the daily required nutrients and suggesting additional foods that should be taken in.

[1016] 6. Sentiment analysis and suggestion adjustment: Recognize the user's emotions and adjust the suggestion content based on those emotions.

[1017] 7. Next order suggestion method for delivered food: By taking an image of the delivered food and implementing the above method, next order suggestion is made.

[1018] Hardware and software used

[1019] 1. Hardware:

[1020] Smartphone (used to capture food images)

[1021] Server (used for image analysis and data processing)

[1022] 2. Software:

[1023] Dedicated app (takes pictures of food on a smartphone and communicates with the server)

[1024] Image recognition algorithm (using convolutional neural networks)

[1025] Nutritional value database (used to obtain nutritional information)

[1026] Emotion recognition engine (analyzes the user's emotional state)

[1027] More examples

[1028] Usage example 1:

[1029] The user takes a photo of their lunch (salad, chicken steak, and rice) and uploads it through a dedicated app. The image is analyzed on the server, and the salad, chicken steak, and rice are identified. The nutritional value of each food item is retrieved from the database, and the total nutritional value is calculated. The calculation results are displayed in graph form, and based on the user's nutrient deficiencies, further foods to supplement are suggested.

[1030] Usage example 2:

[1031] Users take a photo of the inside of their refrigerator with their smartphone and upload the image through a dedicated app. The server analyzes the image and identifies the foods in the refrigerator. Based on the results, nutritionally balanced meals are suggested to the user.

[1032] Example prompt sentence:

[1033] "I want to develop a system that analyzes images of food, calculates nutritional value, and makes suggestions for the next meal based on the user's emotions. This system allows users to input their thoughts along with an image of the delivery food, and then makes suggestions for the next order based on that."

[1034] This allows users to instantly receive healthy meal suggestions that suit their emotional state, even when using a delivery service.

[1035] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1036] Step 1:

[1037] The user takes a photo of the food with their smartphone and launches a dedicated app.

[1038] Input: Food photo

[1039] Output: Food photo data

[1040] How it works: A user takes a photo of a dish using their smartphone and uploads the image to a dedicated app. The app compresses and encodes the image data and converts it into a format that can be sent.

[1041] Step 2:

[1042] The terminal receives the captured image, preprocesses the image data, and then sends it to the server.

[1043] Input: Compressed and encoded food photo data

[1044] Output: Preprocessed image data

[1045] Specific operation: The terminal performs preprocessing such as resizing and format conversion of the image data, and then sends the image data to the server.

[1046] Step 3:

[1047] The server uses image recognition algorithms to analyze the received images.

[1048] Input: Preprocessed image data

[1049] Output: Data about food types

[1050] How it works: An image recognition algorithm (e.g., a convolutional neural network) on the server analyzes the preprocessed image and identifies the type of food in the image. The identified food type data is output.

[1051] Step 4:

[1052] The server retrieves the nutritional values ​​of the identified foods from a database and calculates the total nutritional value.

[1053] Input: Food type data, nutritional value database

[1054] Output: Calculated nutritional information

[1055] Specific operation: The server obtains the nutritional information of each food from the nutritional value database based on the type of food, and calculates the overall nutritional value by adding up the nutritional values ​​of each food.

[1056] Step 5:

[1057] The server visualizes the calculated nutritional values ​​in graph form and sends them to the terminal.

[1058] Input: Calculated nutritional information

[1059] Output: Nutritional information in graphical format

[1060] Specific operation: The server visualizes the calculation results in the form of a graph (e.g., bar graph, pie chart) and sends the generated graph to the terminal.

[1061] Step 6:

[1062] The terminal displays the received nutritional information in graph form to the user.

[1063] Input: Nutritional information in graph format

[1064] Output: The displayed graph

[1065] Specific operation: The device receives the nutritional information in graph form and displays it on the screen.

[1066] Step 7:

[1067] The server compares the daily nutrient requirements with the current intake, identifies any nutrient deficiencies, and suggests additional foods to consume.

[1068] Input: Calculated nutritional information, daily required nutrients information

[1069] Output: Suggestions for supplemental foods

[1070] Specific operation: The server compares the calculated nutritional value information with the daily nutrient requirement information to identify any nutrient deficiencies. It then searches the database for foods that can make up for those deficiencies and generates recommendations for the user.

[1071] Step 8:

[1072] The server analyzes emotions based on user input and adjusts the food and menu suggestions.

[1073] Input: User text input, emotion recognition engine

[1074] Output: Food recommendations based on emotional state

[1075] Specific operation: The user inputs their thoughts about the meal in text format, and the data is analyzed by an emotion recognition engine. Based on the user's emotional state (happiness, sadness, satisfaction, etc.), the suggested foods and menu contents are adjusted.

[1076] Step 9:

[1077] The server acquires an image of the delivered food and implements the above-mentioned means to suggest the next order.

[1078] Input: Image of the delivered food

[1079] Output: Next order proposal

[1080] How it works: The user takes a photo of the food they have delivered and uploads it through a dedicated app. The server analyzes the image and performs the steps described above to suggest foods suitable for the next delivery order.

[1081] This allows users to instantly receive healthy meal suggestions that suit their emotional state, even when using a delivery service.

[1082] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1083] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1084] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1085] [Third embodiment]

[1086] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1087] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1088] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1089] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1090] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1091] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1092] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1093] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1094] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1095] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1096] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1097] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1098] System Overview

[1099] The present invention is a system that acquires images of dishes, analyzes the images to identify the type of food, and calculates and displays the nutritional values ​​of each food. It also has the function of acquiring images of a refrigerator, identifying the foods in the refrigerator, and proposing menus based on the identified foods. A specific embodiment of this system is described below.

[1100] Main functions and program processing

[1101] 1. Acquiring and Preprocessing Images of Dishes

[1102] The user takes a photo of the food with their smartphone and launches the dedicated app. The device receives the photo, performs basic pre-processing, and then sends it to the server.

[1103] 2. Image Recognition

[1104] The server uses an image recognition model (e.g., convolutional neural network) to analyze the received images. First, it identifies the type of food in the image, and then uses that information to retrieve the nutritional value of each food from a database. Through this process, it can identify foods such as "salad," "chicken steak," and "rice" that appear in the photo.

[1105] 3. Calculating and displaying nutritional values

[1106] The server calculates the total calories and the total of each nutrient based on the acquired nutritional information, and visualizes the results in a graph format (e.g., bar graph, pie chart) and sends it to the device, allowing the user to easily check it.

[1107] 4. Nutrition Suggestions

[1108] The server compares the user's daily nutrient requirements with their current intake and identifies any nutrient deficiencies. It also suggests specific foods to supplement the deficiencies. For example, it generates a notification such as, "You need 20 grams more dietary fiber. Try adding brown rice or mushrooms." The device displays this notification to the user.

[1109] 5. Refrigerator image acquisition and food ingredient recognition

[1110] The user takes a photo of the inside of their refrigerator and uploads the image using a dedicated app. The device receives the image and sends it to a server. The server uses image recognition technology to identify the food in the refrigerator. Based on this, the system suggests nutritionally balanced meals.

[1111] 6. Daily Records and Management

[1112] The server records the user's daily nutritional intake and weight and displays the data in a calendar format, allowing users to easily understand their health status. It also provides special nutritional management plans for pregnant and dieting users, and suggests nutrients based on those plans.

[1113] Specific examples

[1114] Food image processing

[1115] 1. The user takes a photo of their lunch (salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[1116] 2. The server receives the image and uses image recognition algorithms to identify salad, chicken steak, and rice.

[1117] 3. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[1118] 4. The server graphs the calculation results and sends them to the terminal for display to the user.

[1119] 5. The server identifies the nutrients that are lacking (e.g., dietary fiber) and suggests specific foods (e.g., brown rice).

[1120] Checking the ingredients in the refrigerator and suggesting menus

[1121] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[1122] 2. The server receives the images and uses image recognition algorithms to identify the food in the refrigerator.

[1123] 3. The server will suggest a nutritionally balanced menu (e.g., tomato and cheese salad) based on the identified foods.

[1124] Daily records and health management

[1125] 1. The server records daily nutritional intake and weight and displays them in a calendar format.

[1126] 2. The server provides special nutritional management plans for users who are pregnant or on a diet and makes suggestions based on them.

[1127] This allows users to easily manage their nutritional intake and efficiently maintain their health.

[1128] The processing flow will be explained below.

[1129] Step 1:

[1130] The user takes a picture of the food with their smartphone and launches a dedicated app.

[1131] Step 2:

[1132] The device receives the captured image and performs preprocessing on the image data (e.g., image resizing, format conversion).

[1133] Step 3:

[1134] The terminal transmits the image data after the preprocessing to the server.

[1135] Step 4:

[1136] The server analyzes the received image data and applies an image recognition algorithm (e.g., convolutional neural network) to identify each food item in the image.

[1137] Step 5:

[1138] Based on the identified food information, the server retrieves nutritional data (e.g., calories, protein, fat, carbohydrates) for each food from the database.

[1139] Step 6:

[1140] The server calculates the overall nutritional value of each food item based on the nutritional data it has acquired, by adding up the calories and nutrients of each food item.

[1141] Step 7:

[1142] The server then graphs the results, generating visual bar and pie charts of calories, protein, fat, and carbohydrates, for example.

[1143] Step 8:

[1144] The server sends the graphed nutritional data to the terminal, which displays the data to the user.

[1145] Step 9:

[1146] The server compares your current intake with your daily nutrient needs and identifies any nutrient deficiencies. Specifically, it compares your current intake with the recommended intake.

[1147] Step 10:

[1148] The server will suggest specific foods to supplement the missing nutrients, generating a notification such as "You are missing 20 grams of dietary fiber. Add brown rice or mushrooms."

[1149] Step 11:

[1150] The server generates a notification and sends it to the terminal, which displays it to the user.

[1151] Step 12:

[1152] The user takes a picture of the refrigerator and uploads it to the server using a dedicated app.

[1153] Step 13:

[1154] The server receives the image of the refrigerator and applies image recognition algorithms to identify the food items inside the refrigerator.

[1155] Step 14:

[1156] The server will then suggest nutritionally balanced meals based on the identified food data, such as a tomato and cheese salad and a lettuce and chicken sandwich.

[1157] Step 15:

[1158] The server sends the proposed menu information to the terminal, which displays this information to the user.

[1159] Step 16:

[1160] The server records the user's nutritional intake and weight on a daily basis and generates data to be displayed in calendar format.

[1161] Step 17:

[1162] The server sends the generated daily record data to the terminal, which displays it to the user in a calendar format.

[1163] Step 18:

[1164] The server creates a special nutritional management plan for users who are pregnant or dieting, and makes specific nutritional recommendations based on that plan.

[1165] Step 19:

[1166] The server sends recommendations based on a special nutritional management plan to the terminal, which displays them to the user.

[1167] Example 1

[1168] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1169] Conventional nutrition management systems require users to manually input food data, which is time-consuming and often lacks accuracy. Furthermore, they are unable to efficiently manage ingredients and suggest menus for users, making it difficult to plan nutritionally balanced meals. Furthermore, recording daily nutritional intake and weight is cumbersome, making it difficult to provide effective nutritional advice to specific users, such as those who are pregnant or on a diet.

[1170] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1171] In this invention, the server includes a means for preprocessing food images, a means for identifying the type of food in the image using a convolutional neural network, and a means for retrieving nutritional values ​​from a database and calculating the total nutritional value. This allows a user to simply take a photo of a dish, and the system automatically calculates the nutritional value of each food and suggests missing nutrients and foods that should be consumed in addition, enabling efficient and accurate nutritional management. Furthermore, the system also includes functions for preprocessing refrigerator images, identifying foods in the refrigerator using image recognition technology, and suggesting menus based on the identified foods, as well as a function for recording the user's nutritional intake and weight daily and displaying them in calendar format, thereby enabling comprehensive dietary and health management.

[1172] "Means for acquiring images of food" refers to the function that allows a user to take a photo of food using an input device such as a smartphone or camera and import it into the system.

[1173] "Means for preprocessing acquired images" refers to processing such as noise removal, resolution adjustment, and color correction performed on received images to improve the accuracy of image analysis.

[1174] The "means for transmitting preprocessed images to a server" refers to a function for transferring image data for which preprocessing has been completed to a server via the Internet.

[1175] "Means for identifying each food type in an image using a convolutional neural network" refers to the ability to use a convolutional neural network, a type of deep learning technology, to identify each food object in a preprocessed image and classify its type.

[1176] "Means for retrieving the nutritional value of identified foods from a database and calculating the total nutritional value" refers to a function for retrieving nutritional value information for each identified food from a database and calculating the total calories and the total value of each nutrient.

[1177] "Means for displaying calculated nutritional values ​​in a graph" refers to a function that visualizes and displays the calculated nutritional values ​​in the form of a bar graph, pie chart, etc., so that the user can easily understand them intuitively.

[1178] "Means to identify nutrients that are lacking compared to the daily required nutrients and suggest additional foods that should be taken" refers to a function that compares the user's daily nutritional intake standard with their current intake status, identifies nutrients that are lacking, and suggests specific foods to make up for the deficiency.

[1179] "Means for acquiring images of the refrigerator" refers to a function that allows a user to take a photo of the inside of the refrigerator using an input device such as a smartphone or camera and import it into the system.

[1180] "Means for identifying food items in a refrigerator using image recognition technology" refers to the use of deep learning or other image recognition algorithms to identify each food object in a photograph of the refrigerator and determine its type and quantity.

[1181] "Means for suggesting nutritionally balanced menus based on identified foods" refers to a function that suggests nutritionally balanced meal menus to the user based on the foods present in the refrigerator.

[1182] "Means for recording the user's nutritional intake and weight on a daily basis and displaying it in calendar format" refers to a function that records the user's daily nutrition intake and weight fluctuations and visually displays the data in calendar format.

[1183] "A means of providing specific nutritional management plans for users who are pregnant or on a diet and making nutritional suggestions based on those plans" refers to the function of creating appropriate nutritional management plans for users with specific health conditions or goals, and making specific nutritional supplement suggestions based on those plans.

[1184] System Overview

[1185] This system captures images of food and the contents of a refrigerator, identifies the type of food through image analysis, and calculates and displays its nutritional value. The server uses image recognition technology to allow users to easily manage their diet and nutrition via their smartphone. It also supports health management by recording the user's daily nutritional intake and weight and visualizing them in a calendar format.

[1186] Main functions and program processing

[1187] Acquiring and preprocessing food images

[1188] A user takes a photo of a dish using a smartphone and launches a dedicated app. The device receives the captured image and performs preprocessing such as noise reduction, resolution adjustment, and color correction. This preprocessing is performed using image analysis software (e.g., OpenCV). The preprocessed image is then sent to the server using a secure protocol (e.g., HTTPS).

[1189] Image Recognition

[1190] The server receives the image and uses a convolutional neural network (e.g., using TensorFlow or PyTorch) to identify the type of food in the image. This identifies foods such as "salad," "chicken steak," and "rice." The server then retrieves the nutritional value information for each food from a database (e.g., an SQL database).

[1191] Nutritional Value Calculation and Labeling

[1192] The server calculates the total calories and the total of each nutrient based on the acquired nutritional information. These calculation results are converted into graphs (bar graphs, pie charts, etc.) using a visualization tool (e.g., Matplotlib), and then sent to the terminal to be displayed intuitively to the user.

[1193] Nutrition Suggestions

[1194] The server compares the user's daily nutrient needs with their current intake. It then identifies nutrient deficiencies and suggests specific foods to fill the gaps. For example, it generates a notification such as, "You need 20 more grams of dietary fiber. Try adding brown rice or mushrooms." The device then displays this notification to the user.

[1195] Refrigerator image acquisition and food ingredient recognition

[1196] The user takes a photo of the inside of the refrigerator with their smartphone and uploads the image using a dedicated app. The device performs preprocessing and sends the image to the server. The server uses an image recognition algorithm (e.g., YOLOv4 or EfficientDet) to identify the foods in the refrigerator and uses that information to suggest nutritionally balanced meals.

[1197] Daily records and management

[1198] The server records the user's daily nutritional intake and weight and stores this data in a database. A view displaying the recorded data in a calendar format is generated and sent to the device. This allows the user to grasp their own health information at a glance. It also provides special nutritional management plans (e.g., for pregnancy or dieting) and suggests nutrients that are suitable for the user.

[1199] Specific examples

[1200] A concrete example of food image processing

[1201] 1. The user takes a photo of their lunch (salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[1202] 2. The server receives the image and uses an image recognition algorithm (e.g., TensorFlow's convolutional neural network) to identify salad, chicken steak, and rice.

[1203] 3. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[1204] 4. The server graphs the calculation results and sends them to the terminal for display to the user.

[1205] 5. The server identifies the nutrients that are lacking (e.g., dietary fiber) and suggests specific foods (e.g., brown rice).

[1206] Checking the ingredients in the refrigerator and suggesting a menu

[1207] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[1208] 2. The server receives the images and uses an image recognition algorithm (e.g., YOLOv4) to identify the food in the refrigerator.

[1209] 3. The server will suggest a nutritionally balanced menu (e.g., tomato and cheese salad) based on the identified foods.

[1210] Examples of daily records and health management

[1211] 1. The server records daily nutritional intake and weight and displays them in a calendar format.

[1212] 2. The server provides special nutritional management plans for users who are pregnant or on a diet and makes suggestions based on them.

[1213] By combining all of the above functions, users can easily and efficiently manage their nutritional intake and health. This system will be an important tool that contributes to raising consumer health awareness.

[1214] Example prompts to be input to the generative AI model

[1215] "Analyze this food photo to identify the types of foods it contains and their nutritional value."

[1216] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1217] Step 1: The user takes a picture of the food and launches the dedicated app.

[1218] The user takes a photo of the food using the camera function of their smartphone, opens the dedicated app, selects the image, and presses a button to start processing. The input of this step is the image taken by the camera, and the output is the image imported into the dedicated app.

[1219] Step 2: The device preprocesses the image.

[1220] The device performs preprocessing on the captured image, such as noise removal, resolution adjustment, and color correction, to improve the accuracy of image analysis. Specifically, it uses image processing libraries such as OpenCV. The input is the image taken by the user, and the output is a clear, preprocessed image.

[1221] Step 3: The device sends the preprocessed image to the server.

[1222] The device sends the preprocessed image data to the server using the HTTPS protocol. Data is encrypted during transmission to ensure security. The input is the preprocessed image, and the output is the image sent to the server.

[1223] Step 4: The server receives the image and applies the convolutional neural network.

[1224] The server stores the received images and begins analyzing them using TensorFlow or PyTorch, identifying each food type using a convolutional neural network. The input for this step is the image sent to the server, and the output is a list of each identified food object.

[1225] Step 5: The server retrieves the nutritional value of the food from the database.

[1226] For each food item identified as a result of the analysis, the server retrieves nutritional information from a database (e.g., an SQL database). The input is a list of identified foods, and the output is the nutritional information for each food item.

[1227] Step 6: The server calculates the total nutritional value.

[1228] The server calculates the total calories and the total of each nutrient based on the nutritional value information of each food. The input is the nutritional value information of each food, and the output is the total nutritional value.

[1229] Step 7: The server graphs the calculation results and sends them to the terminal.

[1230] The server converts the calculation results into graphs (such as bar graphs or pie charts) using a visualization tool (e.g., Matplotlib) and sends them to the terminal. The terminal receives them and displays them to the user. The input is the total nutritional value, and the output is the graphed data.

[1231] Step 8: Your server will provide you with nutrients you are lacking and suggestions for adding more.

[1232] The server compares the user's daily nutrient needs with their current intake to identify any nutrient deficiencies, and then suggests specific foods to supplement the deficiencies. The input is the user's intake and nutritional criteria, and the output is a notification of the recommendations.

[1233] Step 9: The user takes a picture of the inside of the refrigerator and uploads the image.

[1234] The user takes a picture of the inside of the refrigerator with their smartphone and uploads the image to a dedicated app. The input is the image of the inside of the refrigerator, and the output is the image uploaded to the app.

[1235] Step 10: The device sends the image of the refrigerator to the server.

[1236] The device performs preprocessing and then sends the image to the server. The input is the preprocessed image and the output is the image sent to the server.

[1237] Step 11: The server applies image recognition algorithms to identify the food in the refrigerator.

[1238] The server uses image recognition techniques such as YOLOv4 and EfficientDet to identify the foods in the refrigerator. The input is an image of the refrigerator sent to the server, and the output is a list of identified foods.

[1239] Step 12: The server suggests a menu.

[1240] The server creates a nutritionally balanced menu based on the identified food information and proposes it to the user. The input is a list of foods in the refrigerator, and the output is a proposed menu.

[1241] Step 13: The server records the user's nutritional intake and weight and displays them in a calendar format.

[1242] The server records the user's daily nutritional intake and weight and displays the data in a calendar format. The input is the user's intake information and weight data, and the output is a calendar display.

[1243] Step 14: The server provides a special nutrition plan.

[1244] The server provides special nutritional management plans for pregnant and dieting users and makes specific nutrition recommendations based on those plans. The input is each user's specific conditions, and the output is a notification of recommendations based on the plan.

[1245] (Application example 1)

[1246] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1247] Currently, it is important to eat meals that are conscious of health management and nutritional balance, but it is not easy to actually understand the contents of one's daily diet and manage it appropriately. In addition, when using delivery services, there are few ways to understand the nutritional value of the food, making it difficult for customers to make healthy choices. Furthermore, it is time-consuming to plan a menu that effectively uses the ingredients in the refrigerator. To solve these issues, a system that provides more detailed nutritional information and makes healthy meal suggestions is needed.

[1248] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1249] In this invention, the server includes means for acquiring images of dishes, means for analyzing the acquired images and identifying the type of each food in the images, means for acquiring the nutritional value of the identified foods from a database and calculating the total nutritional value, means for displaying the calculated nutritional value in a graph, means for identifying nutrients that are lacking by comparing them with the daily nutrient requirement and suggesting additional foods that should be consumed, means for a user to upload an image of a dish when ordering delivery and display the nutritional value information in real time, means for automatically calculating nutritional information for menus provided by the delivery company and suggesting it to the customer, means for recognizing the contents of the customer's refrigerator and suggesting healthy menus based on that, and means for managing the user's daily nutritional intake and suggesting an optimal meal plan.

[1250] This allows users to use information about ingredients in their refrigerators at home when ordering delivery to plan nutritionally balanced meals, enabling them to make healthy meal choices. It also allows for efficient and effective daily nutrition management.

[1251] "Cuisine" refers to food that has been prepared by cooking and that has been modified to improve its nutritional value or palatability.

[1252] "Capturing an image" refers to the process of capturing visual information as digital data using a device such as a camera or scanner.

[1253] "Analyzing images" refers to the process of applying pattern recognition and machine learning algorithms to captured image data to extract and identify specific information.

[1254] "Identifying the type of food" refers to classifying objects recognized from acquired image data into specific food categories.

[1255] "Retrieving nutritional values ​​from a database" means retrieving the specific calculations and nutritional components of a food from a pre-stored information source.

[1256] "Calculating total nutritional value" refers to the act of adding up the individual nutritional components of a specified food and calculating their sum.

[1257] "Displaying in a graph" means presenting calculated numerical information or data in the form of a bar graph, pie chart, or the like to make it visually easier to understand.

[1258] "Nutrient identification" means diagnosing whether you have a deficiency or excess of certain key components in your daily diet or nutrition plan.

[1259] "Suggesting foods to consume" refers to the act of recommending specific foods or ingredients to supplement missing nutrients.

[1260] "Delivery order" refers to the act of a user making a request to have food delivered by a delivery company.

[1261] "Recognizing the contents of a refrigerator" means performing image analysis to identify the types and quantities of food and ingredients stored in the refrigerator.

[1262] "Suggesting a menu" means creating and presenting a meal plan that takes nutritional balance into consideration and combines multiple dishes and foods.

[1263] "Managing daily nutritional intake" refers to a user recording how much nutrients they consume each day and adjusting their health and eating habits based on that information.

[1264] "Proposing the optimal meal plan" means designing and recommending the best meal content based on an individual's nutritional status and lifestyle.

[1265] To implement the present invention, the system operates as follows.

[1266] First, the user takes a photo of the food with their smartphone and launches the dedicated app. The device receives the photo, performs basic preprocessing, and then sends it to the server. This preprocessing uses an image processing library such as OpenCV.

[1267] The server then uses a pre-trained image recognition model (e.g., a TensorFlow / Keras convolutional neural network) to analyze the received image. The image recognition model identifies the type of food and uses that information to retrieve the nutritional values ​​of each food item from a database. At this stage, the necessary nutritional value data is obtained based on the information extracted from the image, such as "salad" or "chicken steak."

[1268] The server then calculates the total calories and the total of each nutrient based on the acquired nutritional information. The calculation results are visualized in a graph format (e.g., bar graph, pie chart) and sent to the device. This allows the user to check the nutritional value of the food they have photographed at a glance.

[1269] The server also compares the user's daily nutrient needs with their current intake to identify any nutrient deficiencies. It also has the ability to suggest specific foods to fill the gaps. For example, it generates a notification that says, "You need 20 more grams of dietary fiber. Try adding brown rice or mushrooms." and displays it on the device.

[1270] Additionally, the system allows users to take a photo of the inside of their refrigerator and upload the image using a dedicated app. The device receives the image and sends it to a server, which uses image recognition technology to identify the foods in the refrigerator. Based on the results, the system suggests nutritionally balanced meals.

[1271] The server also has a function to record the user's daily nutritional intake and weight and display them in a calendar format, allowing users to easily understand their health status. It also provides special nutritional management plans for users who are pregnant or on a diet, and makes suggestions based on those plans.

[1272] When using a delivery service, users can upload a photo of the food they want to order and view its nutritional information in real time. The system also includes a function to automatically calculate the nutritional information of the menu items offered by the delivery service and provide suggestions to customers. For example, users can be provided with nutritional information for delivery foods such as pizza and burgers.

[1273] Specific examples

[1274] 1. A user uploads a photo of a pizza.

[1275] 2. The server analyzes the image, identifies the pizza's ingredients, and retrieves its nutritional information from a database.

[1276] 3. The server calculates the nutritional value and displays it in a graph.

[1277] 4. The server will identify any nutrients that are lacking and make suggestions such as salads.

[1278] Example prompt sentence:

[1279] Upload an image of the food you want to order and we'll display nutritional information in real time and suggest meals that take into account the ingredients in your refrigerator.

[1280] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1281] Step 1:

[1282] The user takes a photo of the food with their smartphone and launches a dedicated app.

[1283] Input: Food image taken with a smartphone

[1284] Specific operation: The user takes a photo of the food and presses the "Upload image" button on the dedicated app.

[1285] Step 2:

[1286] The device receives the captured photo, performs some basic pre-processing, and then sends it to the server.

[1287] Input: Photographed food image

[1288] Data processing: Using the OpenCV library, preprocessing such as image resizing and noise reduction is performed.

[1289] Specific operation: After processing the image, the device uploads it to the server.

[1290] Step 3:

[1291] The server uses a pre-trained image recognition model to analyze the images it receives.

[1292] Input: Preprocessed food images

[1293] Data computation: A convolutional neural network (CNN) using TensorFlow / Keras is used to identify the type of food in the image.

[1294] What it does: The server runs a CNN model to classify the food in the image.

[1295] Step 4:

[1296] The server retrieves the nutritional values ​​of the identified foods from the database and calculates the total nutritional value.

[1297] Input: List of identified foods

[1298] Data acquisition: Obtain the nutritional value information of each food from the nutritional value database.

[1299] Data calculation: The nutritional value of each food is added up to calculate the total calories and the total of each nutrient.

[1300] Specific operation: The server issues a database query and calculates nutritional values ​​based on the retrieved data.

[1301] Step 5:

[1302] The server generates data that displays the calculated nutritional values ​​in a graph and sends it to the terminal.

[1303] Input: Totaled nutritional data

[1304] Data manipulation: Formatting data into visually understandable formats such as bar graphs and pie charts.

[1305] Specific operation: The server uses a graph generation tool to create visual data and transfers it to the terminal.

[1306] Step 6:

[1307] The terminal displays the graph to the user.

[1308] Input: Graph data sent from the server

[1309] What it does: A dedicated app displays graphs and provides visual feedback to the user.

[1310] Step 7:

[1311] The server compares the user's daily nutrient requirements with their current intake and identifies any nutrients that are lacking.

[1312] Input: User's nutritional intake data and nutrient requirements

[1313] Data calculation: Calculates nutrient deficiencies based on current intake.

[1314] What it does: The server analyzes current nutritional intake data and identifies nutrients that are lacking.

[1315] Step 8:

[1316] The server suggests specific foods to supplement the missing nutrients and sends a notification to the device.

[1317] Input: Missing nutrient data

[1318] Data generation: Generate a list of foods to supplement missing nutrients.

[1319] What happens: The server selects food suggestions and generates a notification with that information.

[1320] Step 9:

[1321] Users take a photo of the inside of their refrigerator and upload the image using a dedicated app.

[1322] Input: Photo of the inside of a refrigerator

[1323] What happens: The user takes a photo of the refrigerator and presses the upload button in the app.

[1324] Step 10:

[1325] The device receives a photo of the inside of the refrigerator and sends it to the server.

[1326] Input: Photo of the inside of a refrigerator

[1327] Specific operation: The device receives the photo, preprocesses it, and uploads it to the server.

[1328] Step 11:

[1329] The server uses image recognition technology to identify food in the refrigerator and suggest nutritionally balanced meals.

[1330] Input: Photo of the inside of a refrigerator

[1331] Data Computing: Using image recognition algorithms to classify and identify food items in the refrigerator.

[1332] Specific operation: The server performs image analysis and generates a menu based on the food information in the refrigerator.

[1333] Step 12:

[1334] The server records the user's daily nutritional intake and weight and displays them in calendar format.

[1335] Input: User nutritional intake and weight data

[1336] Data processing: Visually formatting data in a calendar format.

[1337] Specific operation: The server organizes daily data and provides it to the device as a calendar display.

[1338] Step 13:

[1339] The server provides special nutritional management plans for pregnant or dieting users and suggests nutrients based on the plans.

[1340] Input: User's special health status data (e.g., pregnant, dieting)

[1341] Data calculation: Create a special nutritional management plan and generate nutritional recommendations based on it.

[1342] Specific operation: The server creates a special plan and sends a notification containing the proposal to the device.

[1343] Example prompt sentence:

[1344] Upload an image of the food you want to order and we'll display nutritional information in real time and suggest meals that take into account the ingredients in your refrigerator.

[1345] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1346] System Overview

[1347] The present invention combines a system that acquires images of dishes, analyzes the images to identify the type of food, calculates and displays the nutritional value, and an emotion engine that recognizes the user's emotions. This system not only provides nutritional management but also makes meal suggestions based on the user's emotional state. Specific embodiments of this system are described below.

[1348] Main functions and program processing

[1349] 1. Acquiring and Preprocessing Images of Dishes

[1350] The user takes a photo of the food with their smartphone and launches the dedicated app. The device receives the image, preprocesses it (e.g., resizes the image, converts the format), and then sends it to the server.

[1351] 2. Image Recognition

[1352] The server uses an image recognition algorithm (e.g., convolutional neural network) to analyze the received images. It identifies the type of food in the image and uses that information to retrieve the nutritional value of each food from a database. Through this process, it can identify foods such as "salad," "chicken steak," and "rice" that appear in the photo.

[1353] 3. Calculating and displaying nutritional values

[1354] The server calculates the total calories and the total of each nutrient based on the acquired nutritional information, and visualizes the results in a graph format (e.g., bar graph, pie chart) and sends it to the device, allowing the user to easily check it.

[1355] 4. Nutrition Suggestions

[1356] The server compares the user's daily nutrient requirements with their current intake and identifies any nutrient deficiencies. It also suggests specific foods to supplement the deficiencies. For example, it generates a notification saying, "You need 20 grams more dietary fiber. Try adding brown rice or mushrooms." The device displays this notification to the user.

[1357] 5. Emotion Recognition by Emotion Engine

[1358] The server uses an emotion recognition engine to analyze the user's emotions, specifically identifying emotions such as joy, excitement, and satisfaction based on the food image and user input.

[1359] 6. Adjusting offers based on emotions

[1360] The server may tailor suggested additional foods or meal options based on the identified emotional state, for example, if the user is expressing a feeling of joy, more colorful dishes may be suggested to enhance enjoyment.

[1361] 7. Refrigerator image acquisition and food ingredient recognition

[1362] The user takes a photo of the inside of their refrigerator and uploads the image to a server using a dedicated app. The server then uses image recognition technology to identify the food in the refrigerator. Based on this information, the system suggests nutritionally balanced meals.

[1363] 8. Daily Records and Management

[1364] The server records the user's daily nutritional intake and weight and displays the data in a calendar format, allowing users to easily understand their health status. It also provides special nutritional management plans for pregnant and dieting users, and suggests nutrients based on those plans.

[1365] Specific examples

[1366] Food image processing

[1367] 1. The user takes a photo of their lunch (salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[1368] 2. The server receives the image and uses image recognition algorithms to identify salad, chicken steak, and rice.

[1369] 3. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[1370] 4. The server graphs the calculation results and sends them to the terminal for display to the user.

[1371] 5. The server identifies the nutrients that are lacking (e.g., dietary fiber) and suggests specific foods (e.g., brown rice).

[1372] 6. The server uses an emotion engine to analyze the user's emotions from the photos and input data and adjust the suggestions.

[1373] Checking the ingredients in the refrigerator and suggesting menus

[1374] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[1375] 2. The server receives the images and uses image recognition algorithms to identify the food in the refrigerator.

[1376] 3. The server will suggest a nutritionally balanced menu (e.g., tomato and cheese salad) based on the identified foods.

[1377] 4. The server analyzes the user's emotions through an emotion recognition engine and adjusts the menu accordingly.

[1378] Daily records and health management

[1379] 1. The server records daily nutritional intake and weight and displays them in a calendar format.

[1380] 2. The server provides special nutritional management plans for users who are pregnant or on a diet and makes suggestions based on them.

[1381] 3. The server tailors daily meal and activity suggestions based on the user's emotional state.

[1382] This allows users to not only manage their nutrition, but also receive meal suggestions based on their emotional state, allowing them to enjoy a richer diet.

[1383] The processing flow will be explained below.

[1384] Step 1:

[1385] The user takes a picture of the food with their smartphone and launches a dedicated app.

[1386] Step 2:

[1387] The device receives the captured image and performs preprocessing on the image data (e.g., image resizing, format conversion).

[1388] Step 3:

[1389] The terminal transmits the preprocessed image data to the server.

[1390] Step 4:

[1391] The server analyzes the received image data and uses an image recognition algorithm (e.g., convolutional neural network) to identify each food item in the image.

[1392] Step 5:

[1393] Based on the identified food information, the server retrieves nutritional data (e.g., calories, protein, fat, carbohydrates) for each food from the database.

[1394] Step 6:

[1395] The server calculates the overall nutritional value of each food based on the nutritional data of each food item, adding up the calories and nutrients of each food item.

[1396] Step 7:

[1397] The server then graphs the results, generating visual bar and pie charts of calories, protein, fat, and carbohydrates.

[1398] Step 8:

[1399] The server sends the graphed nutritional data to the terminal, which displays it to the user.

[1400] Step 9:

[1401] The server compares your current intake with your daily nutrient needs to identify any nutrient deficiencies. It also compares your current intake with the recommended intake.

[1402] Step 10:

[1403] The server will suggest specific foods to supplement the missing nutrients, generating a notification such as "You're missing 20 grams of dietary fiber. Add brown rice or mushrooms."

[1404] Step 11:

[1405] The server sends the generated notification to the terminal, which displays it to the user.

[1406] Step 12:

[1407] The server uses an emotion recognition engine to analyze the user's emotions based on the food images and user input, identifying emotions such as joy, excitement, and satisfaction.

[1408] Step 13:

[1409] The server adjusts the suggested additional foods and menu items based on the identified emotional state: if the emotional state is "joy," it suggests more elaborate dishes; if the emotional state is "excited," it suggests new dishes and trending foods.

[1410] Step 14:

[1411] The user takes a photo of the inside of the refrigerator and uploads the image to the server using a dedicated app.

[1412] Step 15:

[1413] The server analyzes the received image of the refrigerator and applies image recognition algorithms to identify each food item inside.

[1414] Step 16:

[1415] The server will then suggest nutritionally balanced meals based on the identified food information, for example, "a tomato and cheese salad, lettuce and chicken sandwich."

[1416] Step 17:

[1417] The server sends the proposed menu information to the terminal, which displays it to the user.

[1418] Step 18:

[1419] The server records the user's nutritional intake and weight on a daily basis and generates data to be displayed in calendar format.

[1420] Step 19:

[1421] The server transmits the generated daily record data to the terminal, which displays it to the user in a calendar format.

[1422] Step 20:

[1423] The server creates a special nutritional management plan for users who are pregnant or dieting, and makes specific nutritional recommendations based on that plan.

[1424] Step 21:

[1425] The server sends recommendations based on a special nutritional management plan to the terminal, which displays them to the user.

[1426] Step 22:

[1427] The server tailors daily food and activity suggestions based on the user's emotional state: if the emotional state is "sad," it suggests specific foods and activities to lighten the mood.

[1428] Example 2

[1429] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1430] Today's consumers are increasingly interested in eating a nutritionally balanced diet and maintaining their health. However, it is difficult to understand which foods contain which nutrients and manage their nutrient intake in their daily lives. Furthermore, there is a lack of systems that provide dietary recommendations based not only on nutritional value but also on the user's emotions. Given this situation, it is necessary to efficiently and accurately manage nutrition and provide recommendations that respond to the user's emotions.

[1431] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1432] In this invention, the server

[1433] A means for acquiring an image of a dish;

[1434] means for pre-processing the acquired images;

[1435] means for analyzing the preprocessed image data and identifying each food type in the image;

[1436] means for retrieving the nutritional values ​​of the identified foods from a database;

[1437] a means for calculating total nutritional value;

[1438] A means for displaying the calculated nutritional values ​​in a graph;

[1439] A method to identify nutrients that are lacking compared to the daily required nutrients and suggest additional foods that should be taken in,

[1440] means for recognizing a user's emotion and tailoring suggestions based on the emotion;

[1441] A means to acquire, preprocess, and analyze images of the refrigerator, identify the foods in the refrigerator, and propose nutritionally balanced meals.

[1442] A means for recording the user's daily nutritional intake and weight and displaying them in a calendar format;

[1443] A means of providing specific nutritional management plans for pregnant or dieting users and making nutritional suggestions based on those plans

[1444] This allows for efficient and accurate nutritional management and meal suggestions that reflect the user's emotions.

[1445] "Food image" is visual data about food that is photographed or acquired by the user.

[1446] "Preprocessing" refers to initial processing of captured or acquired images, such as adjusting their size or converting their format, to make them easier to analyze.

[1447] "Image recognition algorithm" refers to a computational method used to identify specific objects or features within an image.

[1448] "Food type" identifies the various ingredients and food items contained in the photographed food image.

[1449] "Nutritional value" is data that indicates the amount of energy and various nutrients (e.g., protein, lipids, carbohydrates, vitamins, minerals, etc.) contained in food.

[1450] A "database" is a collection of information in which nutritional information about foods is systematically organized and stored.

[1451] "Daily Nutrient Requirements" refers to the standard of nutrients that a user should consume each day.

[1452] "Additional foods" refers to specific ingredients or dishes recommended to supplement missing nutrients.

[1453] An "emotion recognition engine" refers to technology for analyzing a user's emotional state, specifically recognizing a user's emotions such as joy, excitement, and satisfaction from images and input data.

[1454] "Refrigerator image" refers to visual data of the inside of a refrigerator photographed or acquired by a user.

[1455] A "nutritional balanced menu" refers to a meal plan that is nutritionally balanced based on specific food information.

[1456] "Calendar format" refers to a method of organizing data by date and displaying it in a visually easy-to-understand format.

[1457] "Nutrition Management Plan" refers to a plan designed to support optimal nutritional intake for a user in a specific situation, such as during pregnancy or while dieting.

[1458] System Overview

[1459] The present invention combines a system that acquires images of dishes, analyzes the images to identify the type of food, calculates and displays the nutritional value of each food, and an emotion engine that recognizes the user's emotions. This system provides meal suggestions based on the user's nutritional balance and emotional state when managing their daily diet.

[1460] Hardware and software used

[1461] The user uses a device such as a smartphone. A dedicated app is installed on the device, and it is equipped with a pre-processing function for captured images. The pre-processed image data is sent to a server via a network. The following software is running on the server:

[1462] Image recognition algorithms (e.g., convolutional neural networks)

[1463] Database Management System (DBMS)

[1464] Emotion Recognition Engine

[1465] Visualization tools (e.g., D3.js, Chart.js)

[1466] Program processing overview

[1467] The user takes a photo of a dish and launches a dedicated app. The device receives the image and performs preprocessing such as resizing and format conversion. The preprocessed image data is then sent over the network to a server. The server uses an image recognition algorithm to identify the type of food in the image and retrieves the nutritional value of each food item from a database. Based on the retrieved nutritional value information, the server calculates and displays the total calories and the sum of each nutrient.

[1468] The server then compares the user's daily nutrient needs with their current intake to identify any nutrient deficiencies. When suggesting specific foods, an emotion recognition engine analyzes the user's emotional state and adjusts the suggestions accordingly. For example, if the user expresses joy, the server will suggest more colorful dishes to enhance the enjoyment.

[1469] Users can also take photos of the inside of their refrigerator and upload them using a dedicated app. The server uses an image recognition algorithm to identify the food items in the refrigerator and suggests menu items based on that information. In this case, an emotion recognition engine also analyzes the user's emotions and offers the most appropriate menu.

[1470] The server also records the user's daily nutritional intake and weight data and displays it in a calendar format. For users who are pregnant or on a diet, the server provides special nutritional management plans and suggests nutrients based on those plans.

[1471] Specific examples

[1472] Food image processing

[1473] 1. The user takes a photo of their lunch (e.g., salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[1474] 2. The device preprocesses the image and sends the formatted image to the server.

[1475] 3. The server uses image recognition algorithms to identify each food item.

[1476] 4. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[1477] 5. The server graphs the calculation results and sends them to the terminal for display to the user.

[1478] 6. The server will identify any missing nutrients (e.g., dietary fiber) and suggest foods to add (e.g., brown rice).

[1479] 7. The server uses an emotion engine to analyze the user's emotions from the images and input data and adjust the suggestions.

[1480] Checking the ingredients in the refrigerator and suggesting menus

[1481] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[1482] 2. The device preprocesses the image and sends the formatted image to the server.

[1483] 3. The server uses image recognition algorithms to identify the food in the refrigerator.

[1484] 4. The server will suggest a nutritionally balanced meal based on the identified foods (e.g., tomato and cheese salad).

[1485] 5. The server analyzes the user's emotions through an emotion recognition engine and adjusts the menu accordingly.

[1486] Example prompts to be input to the generative AI model

[1487] Analyze the following food images, identify the type of food for each, calculate the nutritional value of each food, and generate a sentence suggesting to the user what nutrients they may be missing:

[1488] Food image: [Image link]

[1489] It should also analyze the user's emotional state and provide meal suggestions based on those emotions.

[1490] Through such a system, users can receive nutritional management and emotionally-based meal suggestions, enabling them to enjoy a healthier and more fulfilling diet.

[1491] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1492] Step 1:

[1493] The user takes a photo of a dish with their smartphone and launches the dedicated app. The user then uploads this image to the dedicated app. The input is the photo of the dish, and the output is image data for preprocessing.

[1494] Step 2:

[1495] The device receives the captured image and performs preprocessing such as image resizing and format conversion. The input is the image uploaded by the user, and the output is the preprocessed image data. This preprocessing enables efficient image analysis.

[1496] Step 3:

[1497] The terminal sends the preprocessed image data to the server via the network. The input is the preprocessed image data, and the output is the transmission of the image data to the server.

[1498] Step 4:

[1499] The server uses an image recognition algorithm (e.g., convolutional neural network) to analyze the received images. The input is the preprocessed image data, and the output is the analyzed data including the type of each food. Based on this analyzed data, the server retrieves the nutritional value of each food from the database.

[1500] Step 5:

[1501] The server calculates the total calories and total of each nutrient based on the acquired nutritional information. The input is the nutritional value data of the identified food, and the output is the total nutritional value calculation. This calculation includes calories, proteins, fats, carbohydrates, and other major nutrients.

[1502] Step 6:

[1503] The server visualizes the calculation results in graph format (e.g., bar graph, pie chart). The input is the calculation result of total nutritional value, and the output is the visualized graph data. The graph makes it easier for users to intuitively understand the nutritional information.

[1504] Step 7:

[1505] The server sends the visualized data to the terminal and displays it to the user. The input is the visualized graph data, and the output is the graph displayed on the user's terminal.

[1506] Step 8:

[1507] The server compares the user's daily required nutrients with their current intake and identifies any nutrient deficiencies. The input is the daily required nutrient data and current intake data, and the output is the results of identifying the nutrient deficiencies.

[1508] Step 9:

[1509] The server suggests specific foods to supplement the missing nutrients. For example, it generates a notification saying, "You need 20 more grams of dietary fiber. Try adding brown rice or mushrooms." The input is the result of identifying the missing nutrients, and the output is a notification with specific food suggestions.

[1510] Step 10:

[1511] The terminal displays this notification to the user.,The input is the food suggestion notification from the server, and,the output is the suggestion content displayed on the user's,terminal.

[1512] Step 11:

[1513] The server uses an emotion recognition engine to analyze the user's emotions. The input is food images and user input data, and the output is data indicating the user's emotional state. Specifically, it generates data indicating the user's emotions, such as joy, excitement, and satisfaction.

[1514] Step 12:

[1515] The server adjusts the suggested additional foods and meal options based on the identified emotional state. The input is the user's emotional state data, and the output is the adjusted food and meal options.

[1516] Step 13:

[1517] The user takes a photo of the inside of the refrigerator and uploads the image to the server using a dedicated app. The input is the photo of the inside of the refrigerator, and the output is the transmission of image data to the server.

[1518] Step 14:

[1519] The server uses image recognition technology to identify the food in the refrigerator. The input is image data of the inside of the refrigerator, and the output is data of the identified food in the refrigerator.

[1520] Step 15:

[1521] The server then proposes a nutritionally balanced menu based on the identified foods. The input is the food data in the refrigerator, and the output is a nutritionally balanced menu proposal.

[1522] Step 16:

[1523] The server records daily nutritional intake and weight data and displays them in a calendar format. The input is the user's daily nutritional intake and weight data, and the output is the display data in a calendar format.

[1524] Step 17:

[1525] The server provides a special nutritional management plan for pregnant or dieting users and makes nutritional recommendations based on the plan. The input is the special nutritional management plan data, and the output is the nutritional recommendations based on the plan.

[1526] (Application example 2)

[1527] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1528] In modern society, nutritional management and emotional care are important for busy people to maintain a healthy diet. However, achieving this requires a lot of time and effort. Furthermore, when using a delivery service, it is not easy to determine whether the ordered food is healthy. Furthermore, there are no systems that can suggest meals based on the user's emotional state. Therefore, there is a need for a system that supports users' nutritional management and suggests optimal meals based on the user's emotional state.

[1529] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring food images, means for analyzing the acquired images and identifying the type of each food in the images, means for acquiring the nutritional values ​​of the identified foods from a database and calculating the total nutritional value, means for displaying the calculated nutritional values ​​in a graph, means for identifying nutrients that are deficient compared to the daily nutrient requirement and suggesting additional foods to be consumed, means for analyzing emotions based on user input and adjusting the suggested foods and menu contents, and means for acquiring images of delivered dishes and implementing the above means to suggest the next order. This allows users to instantly receive healthy meal suggestions that are appropriate for their emotional state even when using a delivery service.

[1530] The "means for acquiring an image of a dish" refers to a method or device that allows a user to take a photo of a dish and input it into the system.

[1531] The "means for analyzing the image and identifying the type of each food in the image" refers to an algorithm or program for analyzing the acquired image of food and identifying the type of each food.

[1532] "Means for obtaining the nutritional value of identified foods from a database and calculating the total nutritional value" refers to a method or device that obtains nutritional information for each identified food from a database and adds up the individual nutrients to calculate the total nutritional value.

[1533] The "means for displaying the calculated nutritional value in a graph" refers to a method or device for displaying the calculated nutritional value in a graph format to make it easier to understand visually.

[1534] "Means for identifying nutrients that are lacking by comparing with the daily required nutrients and suggesting foods that should be taken additionally" refers to a method or device that compares the user's daily required nutrients with their current intake status, identifies nutrients that are lacking, and suggests foods that should be taken to make up for the lack.

[1535] "Means for analyzing emotions based on user input and adjusting suggested food and menu items" refers to a method or device that analyzes a user's emotions based on text and other data entered by the user and optimizes suggested food and menu items according to the results.

[1536] "Means for taking images of delivered food and implementing the above-mentioned means to make suggestions for the next order" refers to a method or device for taking and taking images of delivered food, analyzing them, and making suggestions that will be useful for the next order.

[1537] This invention combines a system that acquires images of food, analyzes them to identify the type of food, calculates and displays the nutritional value, and an emotion engine that recognizes the user's emotions. This system goes beyond nutritional management and can also make meal suggestions based on the user's emotional state.

[1538] System configuration

[1539] The system consists of the following main components:

[1540] 1. Method for acquiring food images: The user takes a photo of the food with their smartphone and launches a dedicated app.

[1541] 2. Image analysis and food identification: The server receives the captured images and uses image recognition algorithms to identify the type of each food item.

[1542] 3. Means for obtaining and calculating nutritional value: The server obtains the nutritional value of the identified food from the database and calculates it.

[1543] 4. Nutritional value display means: The calculated nutritional value is visualized in graph form and sent to the user's device.

[1544] 5. Identifying nutrient deficiencies and suggesting foods: Identifying nutrients that are lacking compared to the daily required nutrients and suggesting additional foods that should be taken in.

[1545] 6. Sentiment analysis and suggestion adjustment: Recognize the user's emotions and adjust the suggestion content based on those emotions.

[1546] 7. Next order suggestion method for delivered food: By taking an image of the delivered food and implementing the above method, next order suggestion is made.

[1547] Hardware and software used

[1548] 1. Hardware:

[1549] Smartphone (used to capture food images)

[1550] Server (used for image analysis and data processing)

[1551] 2. Software:

[1552] Dedicated app (takes pictures of food on a smartphone and communicates with the server)

[1553] Image recognition algorithm (using convolutional neural networks)

[1554] Nutritional value database (used to obtain nutritional information)

[1555] Emotion recognition engine (analyzes the user's emotional state)

[1556] More examples

[1557] Usage example 1:

[1558] The user takes a photo of their lunch (salad, chicken steak, and rice) and uploads it through a dedicated app. The image is analyzed on the server, and the salad, chicken steak, and rice are identified. The nutritional value of each food item is retrieved from the database, and the total nutritional value is calculated. The calculation results are displayed in graph form, and based on the user's nutrient deficiencies, further foods to supplement are suggested.

[1559] Usage example 2:

[1560] Users take a photo of the inside of their refrigerator with their smartphone and upload the image through a dedicated app. The server analyzes the image and identifies the foods in the refrigerator. Based on the results, nutritionally balanced meals are suggested to the user.

[1561] Example prompt sentence:

[1562] "I want to develop a system that analyzes food images, calculates nutritional value, and makes meal recommendations based on the user's feelings. This system allows users to input their thoughts along with an image of the delivery food, and then makes recommendations based on that."

[1563] This allows users to instantly receive healthy meal suggestions that suit their emotional state, even when using a delivery service.

[1564] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1565] Step 1:

[1566] The user takes a photo of the food with their smartphone and launches a dedicated app.

[1567] Input: Food photo

[1568] Output: Food photo data

[1569] How it works: A user takes a photo of a dish using their smartphone and uploads the image to a dedicated app. The app compresses and encodes the image data and converts it into a format that can be sent.

[1570] Step 2:

[1571] The terminal receives the captured image, preprocesses the image data, and then sends it to the server.

[1572] Input: Compressed and encoded food photo data

[1573] Output: Preprocessed image data

[1574] Specific operation: The terminal performs preprocessing such as resizing and format conversion of the image data, and then sends the image data to the server.

[1575] Step 3:

[1576] The server uses image recognition algorithms to analyze the received images.

[1577] Input: Preprocessed image data

[1578] Output: Data about food types

[1579] How it works: An image recognition algorithm (e.g., a convolutional neural network) on the server analyzes the preprocessed image and identifies the type of food in the image. The identified food type data is output.

[1580] Step 4:

[1581] The server retrieves the nutritional values ​​of the identified foods from a database and calculates the total nutritional value.

[1582] Input: Food type data, nutritional value database

[1583] Output: Calculated nutritional information

[1584] Specific operation: The server obtains the nutritional information of each food from the nutritional value database based on the type of food, and calculates the overall nutritional value by adding up the nutritional values ​​of each food.

[1585] Step 5:

[1586] The server visualizes the calculated nutritional values ​​in graph form and sends them to the terminal.

[1587] Input: Calculated nutritional information

[1588] Output: Nutritional information in graphical format

[1589] Specific operation: The server visualizes the calculation results in the form of a graph (e.g., bar graph, pie chart) and sends the generated graph to the terminal.

[1590] Step 6:

[1591] The terminal displays the received nutritional information in graph form to the user.

[1592] Input: Nutritional information in graph format

[1593] Output: The displayed graph

[1594] Specific operation: The device receives the nutritional information in graph form and displays it on the screen.

[1595] Step 7:

[1596] The server compares the daily nutrient requirements with the current intake, identifies any nutrient deficiencies, and suggests additional foods to consume.

[1597] Input: Calculated nutritional information, daily required nutrients information

[1598] Output: Suggestions for supplemental foods

[1599] Specific operation: The server compares the calculated nutritional value information with the daily nutrient requirement information to identify any nutrient deficiencies. It then searches the database for foods that can make up for those deficiencies and generates recommendations for the user.

[1600] Step 8:

[1601] The server analyzes emotions based on user input and adjusts the food and menu suggestions.

[1602] Input: User text input, emotion recognition engine

[1603] Output: Food recommendations based on emotional state

[1604] Specific operation: The user inputs their thoughts about the meal in text format, and the data is analyzed by an emotion recognition engine. Based on the user's emotional state (happiness, sadness, satisfaction, etc.), the suggested foods and menu contents are adjusted.

[1605] Step 9:

[1606] The server acquires an image of the delivered food and implements the above-mentioned means to suggest the next order.

[1607] Input: Image of the delivered food

[1608] Output: Next order proposal

[1609] How it works: The user takes a photo of the food they have delivered and uploads it through a dedicated app. The server analyzes the image and performs the steps described above to suggest foods suitable for the next delivery order.

[1610] This allows users to instantly receive healthy meal suggestions that suit their emotional state, even when using a delivery service.

[1611] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1612] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1613] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1614] [Fourth embodiment]

[1615] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1616] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1617] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1618] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1619] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1620] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1621] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1622] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1623] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1624] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1625] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1626] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1627] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1628] System Overview

[1629] The present invention is a system that acquires images of dishes, analyzes the images to identify the type of food, and calculates and displays the nutritional values ​​of each food. It also has the function of acquiring images of a refrigerator, identifying the foods in the refrigerator, and proposing menus based on the identified foods. A specific embodiment of this system is described below.

[1630] Main functions and program processing

[1631] 1. Acquiring and Preprocessing Images of Dishes

[1632] The user takes a photo of the food with their smartphone and launches the dedicated app. The device receives the photo, performs basic pre-processing, and then sends it to the server.

[1633] 2. Image Recognition

[1634] The server uses an image recognition model (e.g., convolutional neural network) to analyze the received images. First, it identifies the type of food in the image, and then uses that information to retrieve the nutritional value of each food from a database. Through this process, it can identify foods such as "salad," "chicken steak," and "rice" that appear in the photo.

[1635] 3. Calculating and displaying nutritional values

[1636] The server calculates the total calories and the total of each nutrient based on the acquired nutritional information, and visualizes the results in a graph format (e.g., bar graph, pie chart) and sends it to the device, allowing the user to easily check it.

[1637] 4. Nutrition Suggestions

[1638] The server compares the user's daily nutrient requirements with their current intake and identifies any nutrient deficiencies. It also suggests specific foods to supplement the deficiencies. For example, it generates a notification such as, "You need 20 grams more dietary fiber. Try adding brown rice or mushrooms." The device displays this notification to the user.

[1639] 5. Refrigerator image acquisition and food ingredient recognition

[1640] The user takes a photo of the inside of their refrigerator and uploads the image using a dedicated app. The device receives the image and sends it to a server. The server uses image recognition technology to identify the food in the refrigerator. Based on this, the system suggests nutritionally balanced meals.

[1641] 6. Daily Records and Management

[1642] The server records the user's daily nutritional intake and weight and displays the data in a calendar format, allowing users to easily understand their health status. It also provides special nutritional management plans for pregnant and dieting users, and suggests nutrients based on those plans.

[1643] Specific examples

[1644] Food image processing

[1645] 1. The user takes a photo of their lunch (salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[1646] 2. The server receives the image and uses image recognition algorithms to identify salad, chicken steak, and rice.

[1647] 3. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[1648] 4. The server graphs the calculation results and sends them to the terminal for display to the user.

[1649] 5. The server identifies the nutrients that are lacking (e.g., dietary fiber) and suggests specific foods (e.g., brown rice).

[1650] Checking the ingredients in the refrigerator and suggesting menus

[1651] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[1652] 2. The server receives the images and uses image recognition algorithms to identify the food in the refrigerator.

[1653] 3. The server will suggest a nutritionally balanced menu (e.g., tomato and cheese salad) based on the identified foods.

[1654] Daily records and health management

[1655] 1. The server records daily nutritional intake and weight and displays them in a calendar format.

[1656] 2. The server provides special nutritional management plans for users who are pregnant or on a diet and makes suggestions based on them.

[1657] This allows users to easily manage their nutritional intake and efficiently maintain their health.

[1658] The processing flow will be explained below.

[1659] Step 1:

[1660] The user takes a picture of the food with their smartphone and launches a dedicated app.

[1661] Step 2:

[1662] The device receives the captured image and performs preprocessing on the image data (e.g., image resizing, format conversion).

[1663] Step 3:

[1664] The terminal transmits the image data after the preprocessing to the server.

[1665] Step 4:

[1666] The server analyzes the received image data and applies an image recognition algorithm (e.g., convolutional neural network) to identify each food item in the image.

[1667] Step 5:

[1668] Based on the identified food information, the server retrieves nutritional data (e.g., calories, protein, fat, carbohydrates) for each food from the database.

[1669] Step 6:

[1670] The server calculates the overall nutritional value of each food item based on the nutritional data it has acquired, by adding up the calories and nutrients of each food item.

[1671] Step 7:

[1672] The server then graphs the results, generating visual bar and pie charts of calories, protein, fat, and carbohydrates, for example.

[1673] Step 8:

[1674] The server sends the graphed nutritional data to the terminal, which displays the data to the user.

[1675] Step 9:

[1676] The server compares your current intake with your daily nutrient needs and identifies any nutrient deficiencies. Specifically, it compares your current intake with the recommended intake.

[1677] Step 10:

[1678] The server will suggest specific foods to supplement the missing nutrients, generating a notification such as "You are missing 20 grams of dietary fiber. Add brown rice or mushrooms."

[1679] Step 11:

[1680] The server generates a notification and sends it to the terminal, which displays it to the user.

[1681] Step 12:

[1682] The user takes a picture of the refrigerator and uploads it to the server using a dedicated app.

[1683] Step 13:

[1684] The server receives the image of the refrigerator and applies image recognition algorithms to identify the food items inside the refrigerator.

[1685] Step 14:

[1686] The server will then suggest nutritionally balanced meals based on the identified food data, such as a tomato and cheese salad and a lettuce and chicken sandwich.

[1687] Step 15:

[1688] The server sends the proposed menu information to the terminal, which displays this information to the user.

[1689] Step 16:

[1690] The server records the user's nutritional intake and weight on a daily basis and generates data to be displayed in calendar format.

[1691] Step 17:

[1692] The server sends the generated daily record data to the terminal, which displays it to the user in a calendar format.

[1693] Step 18:

[1694] The server creates a special nutritional management plan for users who are pregnant or dieting, and makes specific nutritional recommendations based on that plan.

[1695] Step 19:

[1696] The server sends recommendations based on a special nutritional management plan to the terminal, which displays them to the user.

[1697] Example 1

[1698] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1699] Conventional nutrition management systems require users to manually input food data, which is time-consuming and often lacks accuracy. Furthermore, they are unable to efficiently manage ingredients and suggest menus for users, making it difficult to plan nutritionally balanced meals. Furthermore, recording daily nutritional intake and weight is cumbersome, making it difficult to provide effective nutritional advice to specific users, such as those who are pregnant or on a diet.

[1700] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1701] In this invention, the server includes a means for preprocessing food images, a means for identifying the type of food in the image using a convolutional neural network, and a means for retrieving nutritional values ​​from a database and calculating the total nutritional value. This allows a user to simply take a photo of a dish, and the system automatically calculates the nutritional value of each food and suggests missing nutrients and foods that should be consumed in addition, enabling efficient and accurate nutritional management. Furthermore, the system also includes functions for preprocessing refrigerator images, identifying foods in the refrigerator using image recognition technology, and suggesting menus based on the identified foods, as well as a function for recording the user's nutritional intake and weight daily and displaying them in calendar format, thereby enabling comprehensive dietary and health management.

[1702] "Means for acquiring images of food" refers to the function that allows a user to take a photo of food using an input device such as a smartphone or camera and import it into the system.

[1703] "Means for preprocessing acquired images" refers to processing such as noise removal, resolution adjustment, and color correction performed on received images to improve the accuracy of image analysis.

[1704] The "means for transmitting preprocessed images to a server" refers to a function for transferring image data for which preprocessing has been completed to a server via the Internet.

[1705] "Means for identifying each food type in an image using a convolutional neural network" refers to the ability to use a convolutional neural network, a type of deep learning technology, to identify each food object in a preprocessed image and classify its type.

[1706] "Means for retrieving the nutritional value of identified foods from a database and calculating total nutritional value" refers to a function for retrieving nutritional value information for each identified food from a database and calculating the total calories and total value of each nutrient.

[1707] "Means for displaying calculated nutritional values ​​in a graph" refers to a function that visualizes and displays the calculated nutritional values ​​in the form of a bar graph, pie chart, etc., so that the user can easily understand them intuitively.

[1708] "Means to identify nutrients that are lacking compared to the daily required nutrients and suggest additional foods that should be taken" refers to a function that compares the user's daily nutritional intake standard with their current intake status, identifies nutrients that are lacking, and suggests specific foods to make up for the deficiency.

[1709] "Means for acquiring images of the refrigerator" refers to a function that allows a user to take a photo of the inside of the refrigerator using an input device such as a smartphone or camera and import the photo into the system.

[1710] "Means for identifying food items in a refrigerator using image recognition technology" refers to the use of deep learning or other image recognition algorithms to identify each food object in a photograph of the refrigerator and determine its type and quantity.

[1711] "Means for suggesting nutritionally balanced menus based on identified foods" refers to a function that suggests nutritionally balanced meal menus to the user based on the foods present in the refrigerator.

[1712] "Means for recording the user's nutritional intake and weight on a daily basis and displaying it in calendar format" refers to a function that records the user's daily nutrition intake and weight fluctuations and visually displays the data in calendar format.

[1713] "A means of providing specific nutritional management plans for users who are pregnant or on a diet and making nutritional suggestions based on those plans" refers to the function of creating appropriate nutritional management plans for users with specific health conditions or goals, and making specific nutritional supplement suggestions based on those plans.

[1714] System Overview

[1715] This system captures images of food and the contents of a refrigerator, identifies the type of food through image analysis, and calculates and displays its nutritional value. The server uses image recognition technology to allow users to easily manage their diet and nutrition via their smartphone. It also supports health management by recording the user's daily nutritional intake and weight and visualizing them in a calendar format.

[1716] Main functions and program processing

[1717] Acquiring and preprocessing food images

[1718] A user takes a photo of a dish using a smartphone and launches a dedicated app. The device receives the captured image and performs preprocessing such as noise reduction, resolution adjustment, and color correction. This preprocessing is performed using image analysis software (e.g., OpenCV). The preprocessed image is then sent to the server using a secure protocol (e.g., HTTPS).

[1719] Image Recognition

[1720] The server receives the image and uses a convolutional neural network (e.g., using TensorFlow or PyTorch) to identify the type of food in the image. This identifies foods such as "salad," "chicken steak," and "rice." The server then retrieves the nutritional value information for each food from a database (e.g., an SQL database).

[1721] Nutritional Value Calculation and Labeling

[1722] The server calculates the total calories and the total of each nutrient based on the acquired nutritional information. These calculation results are converted into graphs (bar graphs, pie charts, etc.) using a visualization tool (e.g., Matplotlib), and then sent to the terminal to be displayed intuitively to the user.

[1723] Nutrition Suggestions

[1724] The server compares the user's daily nutrient needs with their current intake. It then identifies nutrient deficiencies and suggests specific foods to fill the gaps. For example, it generates a notification such as, "You need 20 more grams of dietary fiber. Try adding brown rice or mushrooms." The device then displays this notification to the user.

[1725] Refrigerator image acquisition and food ingredient recognition

[1726] The user takes a photo of the inside of the refrigerator with their smartphone and uploads the image using a dedicated app. The device performs preprocessing and sends the image to the server. The server uses an image recognition algorithm (e.g., YOLOv4 or EfficientDet) to identify the foods in the refrigerator and uses that information to suggest nutritionally balanced meals.

[1727] Daily records and management

[1728] The server records the user's daily nutritional intake and weight and stores this data in a database. A view displaying the recorded data in a calendar format is generated and sent to the device. This allows the user to grasp their own health information at a glance. It also provides special nutritional management plans (e.g., for pregnancy or dieting) and suggests nutrients that are suitable for the user.

[1729] Specific examples

[1730] A concrete example of food image processing

[1731] 1. The user takes a photo of their lunch (salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[1732] 2. The server receives the image and uses an image recognition algorithm (e.g., TensorFlow's convolutional neural network) to identify salad, chicken steak, and rice.

[1733] 3. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[1734] 4. The server graphs the calculation results and sends them to the terminal for display to the user.

[1735] 5. The server identifies the nutrients that are lacking (e.g., dietary fiber) and suggests specific foods (e.g., brown rice).

[1736] Checking the ingredients in the refrigerator and suggesting a menu

[1737] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[1738] 2. The server receives the images and uses an image recognition algorithm (e.g., YOLOv4) to identify the food in the refrigerator.

[1739] 3. The server will suggest a nutritionally balanced menu (e.g., tomato and cheese salad) based on the identified foods.

[1740] Examples of daily records and health management

[1741] 1. The server records daily nutritional intake and weight and displays them in a calendar format.

[1742] 2. The server provides special nutritional management plans for users who are pregnant or on a diet and makes suggestions based on them.

[1743] By combining all of the above functions, users can easily and efficiently manage their nutritional intake and health. This system will be an important tool that contributes to raising consumer health awareness.

[1744] Example prompts to be input to the generative AI model

[1745] "Analyze this food photo to identify the types of foods it contains and their nutritional value."

[1746] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1747] Step 1: The user takes a picture of the food and launches the dedicated app.

[1748] The user takes a photo of the food using the camera function of their smartphone, opens the dedicated app, selects the image, and presses a button to start processing. The input of this step is the image taken by the camera, and the output is the image imported into the dedicated app.

[1749] Step 2: The device preprocesses the image.

[1750] The device performs preprocessing on the captured image, such as noise removal, resolution adjustment, and color correction, to improve the accuracy of image analysis. Specifically, it uses image processing libraries such as OpenCV. The input is the image taken by the user, and the output is a clear, preprocessed image.

[1751] Step 3: The device sends the preprocessed image to the server.

[1752] The device sends the preprocessed image data to the server using the HTTPS protocol. Data is encrypted during transmission to ensure security. The input is the preprocessed image, and the output is the image sent to the server.

[1753] Step 4: The server receives the image and applies the convolutional neural network.

[1754] The server stores the received images and begins analyzing them using TensorFlow or PyTorch, identifying each food type using a convolutional neural network. The input for this step is the image sent to the server, and the output is a list of each identified food object.

[1755] Step 5: The server retrieves the nutritional value of the food from the database.

[1756] For each food item identified as a result of the analysis, the server retrieves nutritional information from a database (e.g., an SQL database). The input is a list of identified foods, and the output is the nutritional information for each food item.

[1757] Step 6: The server calculates the total nutritional value.

[1758] The server calculates the total calories and the total of each nutrient based on the nutritional value information of each food. The input is the nutritional value information of each food, and the output is the total nutritional value.

[1759] Step 7: The server graphs the calculation results and sends them to the terminal.

[1760] The server converts the calculation results into graphs (such as bar graphs or pie charts) using a visualization tool (e.g., Matplotlib) and sends them to the terminal. The terminal receives them and displays them to the user. The input is the total nutritional value, and the output is the graphed data.

[1761] Step 8: Your server will provide you with nutrients you are lacking and suggestions for adding more.

[1762] The server compares the user's daily nutrient needs with their current intake to identify any nutrient deficiencies, and then suggests specific foods to supplement the deficiencies. The input is the user's intake and nutritional criteria, and the output is a notification of the recommendations.

[1763] Step 9: The user takes a picture of the inside of the refrigerator and uploads the image.

[1764] The user takes a picture of the inside of the refrigerator with their smartphone and uploads the image to a dedicated app. The input is the image of the inside of the refrigerator, and the output is the image uploaded to the app.

[1765] Step 10: The device sends the image of the refrigerator to the server.

[1766] The device performs preprocessing and then sends the image to the server. The input is the preprocessed image and the output is the image sent to the server.

[1767] Step 11: The server applies image recognition algorithms to identify the food in the refrigerator.

[1768] The server uses image recognition techniques such as YOLOv4 and EfficientDet to identify the foods in the refrigerator. The input is an image of the refrigerator sent to the server, and the output is a list of identified foods.

[1769] Step 12: The server suggests a menu.

[1770] The server creates a nutritionally balanced menu based on the identified food information and proposes it to the user. The input is a list of foods in the refrigerator, and the output is a proposed menu.

[1771] Step 13: The server records the user's nutritional intake and weight and displays them in a calendar format.

[1772] The server records the user's daily nutritional intake and weight and displays the data in a calendar format. The input is the user's intake information and weight data, and the output is a calendar display.

[1773] Step 14: The server provides a special nutrition plan.

[1774] The server provides special nutritional management plans for pregnant and dieting users and makes specific nutrition recommendations based on those plans. The input is each user's specific conditions, and the output is a notification of recommendations based on the plan.

[1775] (Application example 1)

[1776] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1777] Currently, it is important to eat meals that are conscious of health management and nutritional balance, but it is not easy to actually understand the contents of one's daily diet and manage it appropriately. In addition, when using delivery services, there are few ways to understand the nutritional value of the food, making it difficult for customers to make healthy choices. Furthermore, it is time-consuming to plan a menu that effectively uses the ingredients in the refrigerator. To solve these issues, a system that provides more detailed nutritional information and makes healthy meal suggestions is needed.

[1778] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1779] In this invention, the server includes means for acquiring images of dishes, means for analyzing the acquired images and identifying the type of each food in the images, means for acquiring the nutritional value of the identified foods from a database and calculating the total nutritional value, means for displaying the calculated nutritional value in a graph, means for identifying nutrients that are lacking by comparing them with the daily nutrient requirement and suggesting additional foods that should be consumed, means for a user to upload an image of a dish when ordering delivery and display the nutritional value information in real time, means for automatically calculating nutritional information for menus provided by the delivery company and suggesting it to the customer, means for recognizing the contents of the customer's refrigerator and suggesting healthy menus based on that, and means for managing the user's daily nutritional intake and suggesting an optimal meal plan.

[1780] This allows users to use information about ingredients in their refrigerators at home when ordering delivery to plan nutritionally balanced meals, enabling them to make healthy meal choices. It also allows for efficient and effective daily nutrition management.

[1781] "Cuisine" refers to food that has been prepared by cooking and that has been modified to improve its nutritional value or palatability.

[1782] "Capturing an image" refers to the process of capturing visual information as digital data using a device such as a camera or scanner.

[1783] "Analyzing an image" refers to the process of applying pattern recognition and machine learning algorithms to captured image data to extract and identify specific information.

[1784] "Identifying the type of food" refers to classifying objects recognized from acquired image data into specific food categories.

[1785] "Retrieving nutritional values ​​from a database" means retrieving the specific calculations and nutritional components of a food from a pre-stored information source.

[1786] "Calculating total nutritional value" refers to the act of adding up the individual nutritional components of a specified food and calculating their sum.

[1787] "Displaying in a graph" means presenting calculated numerical information or data in the form of a bar graph, pie chart, or the like to make it visually easier to understand.

[1788] "Nutrient identification" means diagnosing whether you have a deficiency or excess of certain key components in your daily diet or nutrition plan.

[1789] "Suggesting foods to consume" refers to the act of recommending specific foods or ingredients to supplement missing nutrients.

[1790] "Delivery order" refers to the act of a user making a request to have food delivered by a delivery company.

[1791] "Recognizing the contents of a refrigerator" means performing image analysis to identify the types and quantities of food and ingredients stored in the refrigerator.

[1792] "Suggesting a menu" means creating and presenting a meal plan that takes nutritional balance into consideration and combines multiple dishes and foods.

[1793] "Managing daily nutritional intake" refers to a user recording how much nutrients they consume each day and adjusting their health and eating habits based on that information.

[1794] "Proposing the optimal meal plan" means designing and recommending the best meal content based on an individual's nutritional status and lifestyle.

[1795] To implement the present invention, the system operates as follows.

[1796] First, the user takes a photo of the food with their smartphone and launches the dedicated app. The device receives the photo, performs basic preprocessing, and then sends it to the server. This preprocessing uses an image processing library such as OpenCV.

[1797] The server then uses a pre-trained image recognition model (e.g., a TensorFlow / Keras convolutional neural network) to analyze the received image. The image recognition model identifies the type of food and uses that information to retrieve the nutritional values ​​of each food item from a database. At this stage, the necessary nutritional value data is obtained based on the information extracted from the image, such as "salad" or "chicken steak."

[1798] The server then calculates the total calories and the total of each nutrient based on the acquired nutritional information. The calculation results are visualized in a graph format (e.g., bar graph, pie chart) and sent to the device. This allows the user to check the nutritional value of the food they have photographed at a glance.

[1799] The server also compares the user's daily nutrient needs with their current intake to identify any nutrient deficiencies. It also has the ability to suggest specific foods to fill the gaps. For example, it generates a notification that says, "You need 20 more grams of dietary fiber. Try adding brown rice or mushrooms." and displays it on the device.

[1800] Additionally, the system allows users to take a photo of the inside of their refrigerator and upload the image using a dedicated app. The device receives the image and sends it to a server, which uses image recognition technology to identify the foods in the refrigerator. Based on the results, the system suggests nutritionally balanced meals.

[1801] The server also has a function to record the user's daily nutritional intake and weight and display them in a calendar format, allowing users to easily understand their health status. It also provides special nutritional management plans for users who are pregnant or on a diet, and makes suggestions based on those plans.

[1802] When using a delivery service, users can upload a photo of the food they want to order and view its nutritional information in real time. The system also includes a function to automatically calculate the nutritional information of the menu items offered by the delivery service and provide suggestions to customers. For example, users can be provided with nutritional information for delivery foods such as pizza and burgers.

[1803] Specific examples

[1804] 1. A user uploads a photo of a pizza.

[1805] 2. The server analyzes the image, identifies the pizza's ingredients, and retrieves its nutritional information from a database.

[1806] 3. The server calculates the nutritional value and displays it in a graph.

[1807] 4. The server will identify any nutrients that are lacking and make suggestions such as salads.

[1808] Example prompt sentence:

[1809] Upload an image of the food you want to order and we'll display nutritional information in real time and suggest meals that take into account the ingredients in your refrigerator.

[1810] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1811] Step 1:

[1812] The user takes a photo of the food with their smartphone and launches a dedicated app.

[1813] Input: Food image taken with a smartphone

[1814] Specific operation: The user takes a photo of the food and presses the "Upload image" button on the dedicated app.

[1815] Step 2:

[1816] The device receives the captured photo, performs some basic pre-processing, and then sends it to the server.

[1817] Input: Photographed food image

[1818] Data processing: Using the OpenCV library, preprocessing such as image resizing and noise reduction is performed.

[1819] Specific operation: After processing the image, the device uploads it to the server.

[1820] Step 3:

[1821] The server uses a pre-trained image recognition model to analyze the images it receives.

[1822] Input: Preprocessed food images

[1823] Data computation: A convolutional neural network (CNN) using TensorFlow / Keras is used to identify the type of food in the image.

[1824] What it does: The server runs a CNN model to classify the food in the image.

[1825] Step 4:

[1826] The server retrieves the nutritional values ​​of the identified foods from the database and calculates the total nutritional value.

[1827] Input: List of identified foods

[1828] Data acquisition: Obtain the nutritional value information of each food from the nutritional value database.

[1829] Data calculation: The nutritional value of each food is added up to calculate the total calories and the total of each nutrient.

[1830] Specific operation: The server issues a database query and calculates nutritional values ​​based on the retrieved data.

[1831] Step 5:

[1832] The server generates data that displays the calculated nutritional values ​​in a graph and sends it to the terminal.

[1833] Input: Totaled nutritional data

[1834] Data manipulation: Formatting data into visually understandable formats such as bar graphs and pie charts.

[1835] Specific operation: The server uses a graph generation tool to create visual data and transfers it to the terminal.

[1836] Step 6:

[1837] The terminal displays the graph to the user.

[1838] Input: Graph data sent from the server

[1839] What it does: A dedicated app displays graphs and provides visual feedback to the user.

[1840] Step 7:

[1841] The server compares the user's daily nutrient requirements with their current intake and identifies any nutrients that are lacking.

[1842] Input: User's nutritional intake data and nutrient requirements

[1843] Data calculation: Calculates nutrient deficiencies based on current intake.

[1844] What it does: The server analyzes current nutritional intake data and identifies nutrients that are lacking.

[1845] Step 8:

[1846] The server suggests specific foods to supplement the missing nutrients and sends a notification to the device.

[1847] Input: Missing nutrient data

[1848] Data generation: Generate a list of foods to supplement missing nutrients.

[1849] What happens: The server selects food suggestions and generates a notification with that information.

[1850] Step 9:

[1851] Users take a photo of the inside of their refrigerator and upload the image using a dedicated app.

[1852] Input: Photo of the inside of a refrigerator

[1853] What happens: The user takes a photo of the refrigerator and presses the upload button in the app.

[1854] Step 10:

[1855] The device receives a photo of the inside of the refrigerator and sends it to the server.

[1856] Input: Photo of the inside of a refrigerator

[1857] Specific operation: The device receives the photo, preprocesses it, and uploads it to the server.

[1858] Step 11:

[1859] The server uses image recognition technology to identify food in the refrigerator and suggest nutritionally balanced meals.

[1860] Input: Photo of the inside of a refrigerator

[1861] Data Computing: Using image recognition algorithms to classify and identify food items in the refrigerator.

[1862] Specific operation: The server performs image analysis and generates a menu based on the food information in the refrigerator.

[1863] Step 12:

[1864] The server records the user's daily nutritional intake and weight and displays them in calendar format.

[1865] Input: User nutritional intake and weight data

[1866] Data processing: Visually formatting data in a calendar format.

[1867] Specific operation: The server organizes daily data and provides it to the device as a calendar display.

[1868] Step 13:

[1869] The server provides special nutritional management plans for pregnant or dieting users and suggests nutrients based on the plans.

[1870] Input: User's special health status data (e.g., pregnant, dieting)

[1871] Data calculation: Create a special nutritional management plan and generate nutritional recommendations based on it.

[1872] Specific operation: The server creates a special plan and sends a notification containing the proposal to the device.

[1873] Example prompt sentence:

[1874] Upload an image of the food you want to order and we'll display nutritional information in real time and suggest meals that take into account the ingredients in your refrigerator.

[1875] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1876] System Overview

[1877] The present invention combines a system that acquires images of dishes, analyzes the images to identify the type of food, calculates and displays the nutritional value, and an emotion engine that recognizes the user's emotions. This system not only provides nutritional management but also makes meal suggestions based on the user's emotional state. Specific embodiments of this system are described below.

[1878] Main functions and program processing

[1879] 1. Acquiring and Preprocessing Images of Dishes

[1880] The user takes a photo of the food with their smartphone and launches the dedicated app. The device receives the image, preprocesses it (e.g., resizes the image, converts the format), and then sends it to the server.

[1881] 2. Image Recognition

[1882] The server uses an image recognition algorithm (e.g., convolutional neural network) to analyze the received images. It identifies the type of food in the image and uses that information to retrieve the nutritional value of each food from a database. Through this process, it can identify foods such as "salad," "chicken steak," and "rice" that appear in the photo.

[1883] 3. Calculating and displaying nutritional values

[1884] The server calculates the total calories and the total of each nutrient based on the acquired nutritional information, and visualizes the results in a graph format (e.g., bar graph, pie chart) and sends it to the device, allowing the user to easily check it.

[1885] 4. Nutrition Suggestions

[1886] The server compares the user's daily nutrient requirements with their current intake and identifies any nutrient deficiencies. It also suggests specific foods to supplement the deficiencies. For example, it generates a notification saying, "You need 20 grams more dietary fiber. Try adding brown rice or mushrooms." The device displays this notification to the user.

[1887] 5. Emotion Recognition by Emotion Engine

[1888] The server uses an emotion recognition engine to analyze the user's emotions, specifically identifying emotions such as joy, excitement, and satisfaction based on the food image and user input.

[1889] 6. Adjusting offers based on emotions

[1890] The server may tailor suggested additional foods or meal options based on the identified emotional state, for example, if the user is expressing a feeling of joy, more colorful dishes may be suggested to enhance enjoyment.

[1891] 7. Refrigerator image acquisition and food ingredient recognition

[1892] The user takes a photo of the inside of their refrigerator and uploads the image to a server using a dedicated app. The server then uses image recognition technology to identify the food in the refrigerator. Based on this information, the system suggests nutritionally balanced meals.

[1893] 8. Daily Records and Management

[1894] The server records the user's daily nutritional intake and weight and displays the data in a calendar format, allowing users to easily understand their health status. It also provides special nutritional management plans for pregnant and dieting users, and suggests nutrients based on those plans.

[1895] Specific examples

[1896] Food image processing

[1897] 1. The user takes a photo of their lunch (salad, chicken steak, and rice) with their smartphone and uploads it using a dedicated app.

[1898] 2. The server receives the image and uses image recognition algorithms to identify salad, chicken steak, and rice.

[1899] 3. The server retrieves the nutritional value of each food item from the database and calculates the total nutritional value.

[1900] 4. The server graphs the calculation results and sends them to the terminal for display to the user.

[1901] 5. The server identifies the nutrients that are lacking (e.g., dietary fiber) and suggests specific foods (e.g., brown rice).

[1902] 6. The server uses an emotion engine to analyze the user's emotions from the photos and input data and adjust the suggestions.

[1903] Checking the ingredients in the refrigerator and suggesting menus

[1904] 1. The user takes a photo of the inside of the refrigerator and uploads it using a dedicated app.

[1905] 2. The server receives the images and uses image recognition algorithms to identify the food in the refrigerator.

[1906] 3. The server will suggest a nutritionally balanced menu (e.g., tomato and cheese salad) based on the identified foods.

[1907] 4. The server analyzes the user's emotions through an emotion recognition engine and adjusts the menu accordingly.

[1908] Daily records and health management

[1909] 1. The server records daily nutritional intake and weight and displays them in a calendar format.

[1910] 2. The server provides special nutritional management plans for users who are pregnant or on a diet and makes suggestions based on them.

[1911] 3. The server tailors daily meal and activity suggestions based on the user's emotional state.

[1912] This allows users to not only manage their nutrition, but also receive meal suggestions based on their emotional state, allowing them to enjoy a richer diet.

[1913] The processing flow will be explained below.

[1914] Step 1:

[1915] The user takes a picture of the food with their smartphone and launches a dedicated app.

[1916] Step 2:

[1917] The device receives the captured image and performs preprocessing on the image data (e.g., image resizing, format conversion).

[1918] Step 3:

[1919] The terminal transmits the preprocessed image data to the server.

[1920] Step 4:

[1921] The server analyzes the received image data and uses an image recognition algorithm (e.g., convolutional neural network) to identify each food item in the image.

[1922] Step 5:

[1923] Based on the identified food information, the server retrieves nutritional data (e.g., calories, protein, fat, carbohydrates) for each food from the database.

[1924] Step 6:

[1925] The server calculates the overall nutritional value of each food based on the nutritional data of each food item, adding up the calories and nutrients of each food item.

[1926] Step 7:

[1927] The server then graphs the results, generating visual bar and pie charts of calories, protein, fat, and carbohydrates.

[1928] Step 8:

[1929] The server sends the graphed nutritional data to the terminal, which displays it to the user.

[1930] Step 9:

[1931] The server compares your current intake with your daily nutrient needs to identify any nutrient deficiencies. It also compares your current intake with the recommended intake.

[1932] Step 10:

[1933] The server will suggest specific foods to supplement the missing nutrients, generating a notification such as "You're missing 20 grams of dietary fiber. Add brown rice or mushrooms."

[1934] Step 11:

[1935] The server sends the generated notification to the terminal, which displays it to the user.

[1936] Step 12:

[1937] The server uses an emotion recognition engine to analyze the user's emotions based on the food images and user input, identifying emotions such as joy, excitement, and satisfaction.

[1938] Step 13:

[1939] The server adjusts the suggested additional foods and menu items based on the identified emotional state: if the emotional state is "joy," it suggests more elaborate dishes; if the emotional state is "excited," it suggests new dishes and trending foods.

[1940] Step 14:

[1941] The user takes a photo of the inside of the refrigerator and uploads the image to the server using a dedicated app.

[1942] Step 15:

[1943] The server analyzes the received image of the refrigerator and applies image recognition algorithms to identify each food item inside.

[1944] Step 16:

[1945] The server will then suggest nutritionally balanced meals based on the identified food information, for example, "a tomato and cheese salad, lettuce and chicken sandwich."

[1946] Step 17:

[1947] The server sends the proposed menu information to the terminal, which displays it to the user.

[1948] Step 18:

[1949] The server records the user's nutritional intake and weight on a daily basis and generates data to be displayed in calendar format.

[1950] Step 19:

[1951] The server transmits the generated daily record data to the terminal, which displays it to the user in a calendar format.

[1952] Step 20:

[1953] The server creates a special nutritional management plan for users who are pregnant or dieting, and makes specific nutritional recommendations based on that plan.

[1954] Step 21:

[1955] The server sends recommendations based on a special nutritional management plan to the terminal, which displays them to the user.

[1956] Step 22:

[1957] The server tailors daily food and activity suggestions based on the user's emotional state: if the emotional state is "sad," it suggests specific foods and activities to lighten the mood.

[1958] Example 2

[1959] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1960] Today's consumers are increasingly interested in eating a nutritionally balanced diet and maintaining their health. However, it is difficult to understand which foods contain which nutrients and manage their nutrient intake in their daily lives. Furthermore, there is a lack of systems that provide dietary recommendations based not only on nutritional value but also on the user's emotions. Given this situation, it is necessary to efficiently and accurately manage nutrition and provide recommendations that respond to the user's emotions.

[1961] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1962] In this invention, the server

[1963] A means for acquiring an image of a dish;

[1964] means for pre-processing the acquired images;

[1965] means for analyzing the preprocessed image data and identifying each food type in the image;

[1966] means for retrieving the nutritional values ​​of the identified foods from a database;

[1967] a means for calculating total nutritional value;

[1968] A means for displaying the calculated nutritional values ​​in a graph;

[1969] A method to identify nutrients that are lacking compared to the daily required nutrients and suggest additional foods that should be taken in,

[1970] means for recognizing a user's emotion and tailoring suggestions based on the emotion;

[1971] A means to acquire, preprocess, and analyze images of the refrigerator, identify the foods in the refrigerator, and propose nutritionally balanced meals.

[1972] A means for recording the user's daily nutritional intake and weight and displaying them in a calendar format;

[1973] A means of providing specific nutritional management plans for pregnant or dieting users and making nutritional suggestions based on those plans

[1974] This allows for efficient and accurate nutritional management and meal suggestions that reflect the user's emotions.

[1975] "Food image" is visual data about food that is photographed or acquired by the user.

[1976] "Preprocessing" refers to initial processing of captured or acquired images, such as adjusting their size or converting their format, to make them easier to analyze.

[1977] "Image recognition algorithm" refers to a computational method used to identify specific objects or features within an image.

[1978] "Food type" identifies the various ingredients and food items contained in the photographed food image.

[1979] "Nutritional value" is data that indicates the amount of energy and various nutrients (e.g., protein, lipids, carbohydrates, vitamins, minerals, etc.) contained in food.

[1980] A "database" is a collection of information in which nutritional information about foods is systematically organized and stored.

[1981] "Daily Nutrient Requirements" refers to the standard of nutrients that a user should consume each day.

[1982] "Additional foods" refers to specific ingredients or dishes recommended to supplement missing nutrients.

[1983] An "emotion recognition engine" refers to technology for analyzing a user's emotional state, specifically recognizing a user's emotions such as joy, excitement, and satisfaction from images and input data.

[1984] "Refrigerator image" refers to visual data of the inside of a refrigerator photographed or acquired by a user.

[1985] A "nutritional balanced menu" refers to a meal plan that is nutritionally balanced based on specific food information.

[1986] "Calendar format" refers to a method of organizing data by date and displaying it in a visually easy-to-understand format.

[1987] "Nutrition Management Plan" refers to a plan designed to support optimal nutritional intake for a user in a specific situation, such as during pregnancy or while dieting.

[1988] System Overview

[1989] The present invention combines a system that acquires images of dishes, analyzes the images to identify the type of food, calculates and displays the nutritional value of each food, and an emotion engine that recognizes the user's emotions. This system provides meal suggestions based on the user's nutritional balance and emotional state when managing their daily diet.

[1990] Hardware and software used

[1991] The user uses a device such as a smartphone. A dedicated app is installed on the device, and it is equipped with a function for pre-processing captured images. The pre-processed image data is sent to a server via a network. The following software is running on the server:

[1992] ...

Claims

1. A means for acquiring an image of a dish; means for analyzing the acquired images and identifying the type of each food item in the images; means for retrieving the nutritional values ​​of the identified foods from the database and calculating the total nutritional value; A means for displaying the calculated nutritional values ​​in a graph; A system that identifies nutrients that are lacking compared to the daily required nutrients and includes a means to suggest additional foods that should be consumed.

2. a means for acquiring an image of the refrigerator; A means for analyzing the acquired image and identifying food items in the refrigerator; The system according to claim 1 , further comprising means for suggesting a nutritionally balanced menu based on the identified foods.

3. A means for recording the user's daily nutritional intake and weight and displaying them in a calendar format; The system of claim 1 , further comprising means for providing a specific nutritional management plan for a user who is pregnant or dieting, and making nutritional suggestions based thereon.

Images