System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing an image of food; means for transmitting the captured image to a server; means for analyzing the image and identifying the food by the server; means for generating a menu based on the identified food; means for analyzing nutritional values of the generated menu and converting the nutritional values into a graph and numerical values; means for presenting insufficient nutrients based on the analysis result; and means for displaying information on the menu and the nutritional values on a user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's households, busy daily lives and nutritional management have become important issues. Managing food inventory and preparing nutritionally balanced meals are not easy, and even greater efforts are required for dieting and nutritional management during pregnancy. This invention was devised to solve these issues and support effective nutritional management and the efficient use of ingredients. [Means for solving the problem]

[0005] The present invention provides a system that effectively solves these problems, including a means for taking images of ingredients, a means for sending the taken images to a server, a means for the server to analyze the images and identify ingredients, a means for generating a menu based on the identified ingredients, a means for analyzing, graphing, and quantifying the nutritional value of the generated menu, a means for presenting nutrients that are lacking based on the analysis results, and a means for displaying the menu and nutritional value information on a user terminal. Furthermore, the system also includes a means for the server to provide information corresponding to questions from a user via voice input, and a means for managing daily menus and nutritional intake status in a calendar format and providing past and future information, enabling comprehensive nutritional management and efficient use of ingredients.

[0006] "Ingredients" refers to raw materials or foods used in cooking or eating.

[0007] "Image" means visual data containing visual information, including photographs and depictions.

[0008] "Server" refers to a computer system that stores, manages, and processes data on a network.

[0009] A "menu" refers to a combination of dishes or a menu planned for a particular meal.

[0010] "Nutritional value" refers to an indicator of the quantity and quality of nutrients (protein, lipids, carbohydrates, vitamins, minerals, etc.) contained in food.

[0011] "Graphing" refers to the visual representation of data or information, using formats such as bar graphs or pie charts.

[0012] "Quantification" refers to the representation of quantitative data in numerical form.

[0013] "Nutrients" refer to substances necessary for maintaining life activities and supporting growth and health, and mainly include proteins, lipids, carbohydrates, vitamins, minerals, etc.

[0014] "User terminal" refers to a device (e.g., smartphone, tablet, PC, etc.) that a user operates directly to access the system.

[0015] "Voice input" refers to the method by which a user communicates instructions or questions to a system using voice.

[0016] The "calendar format" refers to a format in which daily information is visually displayed on a calendar, and refers to a management method that includes past and future information. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention relates to a system for nutritional management that takes images of ingredients in a refrigerator, analyzes the images, and generates optimal menus. This system provides users with a means for efficiently managing their daily nutrition and making effective use of ingredients.

[0039] System Overview

[0040] This system is operated through devices such as smartphones and tablets. The devices have the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on that information. The nutritional value of the generated menu is analyzed and visually displayed to the user.

[0041] Specific Embodiments of the System

[0042] 1. Take a photo of the ingredients and send it

[0043] A user takes a photo of the food in the refrigerator using a device such as a smartphone. The image is then sent to a server via an application. At this time, the device compresses the image data and processes it to enable quick transmission.

[0044] 2. Image Analysis

[0045] The server analyzes the received image. An image recognition algorithm is used for the analysis, and ingredients are automatically identified. For example, ingredients such as chicken, tomatoes, and spices are identified. The algorithm for analyzing the image uses technologies such as YOLO (You Only Look Once) and TensorFlow.

[0046] 3. Menu generation

[0047] The server then searches the database for the most suitable recipe based on the identified ingredients. It generates the optimal menu, taking into account the user's nutritional needs and the expiration dates of the ingredients. For example, it might suggest a menu item such as "chicken steak with tomato salsa."

[0048] 4. Analysis and presentation of nutritional value

[0049] The server analyzes the nutritional value of the generated menu. The analysis uses the nutritional information of each ingredient registered in the database. The amounts of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified. This nutritional information is then graphed and displayed visually to the user.

[0050] 5. Suggestions for nutrient deficiencies

[0051] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For those nutrients, it suggests alternative foods or additional ingredients. For example, if you are deficient in vitamin D, it suggests eating mushrooms or fish.

[0052] 6. Menu and nutrition information display

[0053] The device receives the menu and nutritional information from the server and displays it to the user, who can then practice a balanced diet based on the visually displayed graphs and list-style information.

[0054] 7. Voice question function

[0055] Users can use voice input to ask questions or get more detailed information. For example, if they ask, "How do I cook this dish?", the server will analyze the voice data and provide the appropriate cooking method. The answer will be played back as text or audio.

[0056] 8. Calendar Management

[0057] The device has a function to manage daily menus and nutritional intake status in a calendar format. Users can view past data and future plans and manage their daily eating habits. This allows for long-term nutritional management.

[0058] Specific Examples

[0059] For example, suppose a user takes a photo of their refrigerator and sees that it contains "chicken, tomatoes, and spices." The server analyzes the photo and identifies these ingredients. The server then suggests a meal such as "chicken steak with tomato salsa," and analyzes and visualizes its nutritional value (calories, protein, vitamin C, etc.). Furthermore, if the server determines that the user is lacking in vitamin D, it suggests adding mushrooms or fish. Users can check this information on their device and use it to plan their daily meals.

[0060] The above is an embodiment of the present invention. This system allows users to effectively use ingredients and efficiently manage nutrition.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] When a user launches the smartphone application and takes a photo of the ingredients in the refrigerator, the device activates its camera function and displays a screen on which the user can take a photo of the ingredients.

[0064] Step 2:

[0065] The user checks the photo they have taken and presses the "Send" button in the application. The device sends the photo data to the server.

[0066] Step 3:

[0067] The server receives the transmitted photo and runs an image recognition algorithm to analyze the received image data.

[0068] Step 4:

[0069] The server uses image recognition algorithms to identify ingredients in the received image, and lists the ingredients identified, for example, "chicken, tomatoes, and spices."

[0070] Step 5:

[0071] The server searches the database for the best recipe based on the identified ingredients, taking into account the user's nutritional needs (e.g., calorie restriction, vitamin intake, etc.) and expiration dates of the ingredients.

[0072] Step 6:

[0073] The server generates an optimal menu. For example, it might suggest "chicken steak with tomato salsa." Details of the suggested menu are stored in a database.

[0074] Step 7:

[0075] The server analyzes the nutritional value of the generated menu. The nutritional information includes calories, protein, fat, vitamins, minerals, etc. This information is obtained from a database.

[0076] Step 8:

[0077] The server quantifies the amount of each nutrient and calculates the total amount, then graphs the calculated nutritional information to generate data that can be displayed visually.

[0078] Step 9:

[0079] The server compares the analysis results with the daily required nutrients and identifies any deficiencies. For example, if there is a vitamin D deficiency, it will recommend additional ingredients or foods to supplement it.

[0080] Step 10:

[0081] The server transmits information including incomplete nutritional suggestions to the terminal, including menu details and nutritional value information.

[0082] Step 11:

[0083] The terminal displays the received information to the user in graph and list format so that the user can easily understand it.

[0084] Step 12:

[0085] Users can ask questions or get more information by speaking, for example, "How do I prepare this dish?"

[0086] Step 13:

[0087] The device converts the voice input into text and sends the text data to the server.

[0088] Step 14:

[0089] The server analyzes the question and generates an appropriate answer, such as "How to cook chicken steak..." and provides specific instructions.

[0090] Step 15:

[0091] The server generates a response and sends it to the terminal, which displays or plays it aloud to the user.

[0092] Step 16:

[0093] The device manages daily menus and nutritional intake in a calendar format, allowing users to check the calendar and view past data and upcoming plans.

[0094] In this way, by performing specific actions at each step, the system efficiently supports food inventory management and nutritional management.

[0095] Example 1

[0096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] Managing ingredients and planning nutritionally balanced meals is time-consuming and laborious, making it difficult for many people. It's particularly challenging to effectively utilize ingredients in the refrigerator and manage daily nutritional intake. Furthermore, it's difficult to instantly obtain specific cooking instructions and nutritional information for a menu. A system that can solve these problems is needed.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0099] In this invention, the server includes means for analyzing and identifying images of ingredients, means for generating menus based on the identified ingredients, and means for analyzing and presenting the nutritional values ​​of the generated menus, thereby enabling users to efficiently use ingredients in their refrigerators and easily create balanced meal plans.

[0100] "Means for taking pictures of ingredients" is a function for taking pictures of ingredients in the refrigerator using a camera on a smartphone, tablet, or other device.

[0101] The "means for transmitting the captured image to the information processing device" is a function for appropriately compressing the captured image and transmitting it to an information processing device on a server or cloud.

[0102] "Means for the information processing device to analyze the image and identify ingredients" refers to a function that enables an information processing device located on a server or cloud to detect and identify ingredients in an image using image recognition algorithms such as YOLO or TensorFlow.

[0103] The "means for generating a menu based on the identified ingredients" is a function for automatically suggesting a menu by selecting the optimal recipe based on the identified ingredient information, taking into consideration the user's nutritional needs and the expiration date of the ingredients.

[0104] The "means for analyzing, graphing, and quantifying the nutritional value of the generated menu" is a function that uses the nutritional information of each ingredient registered in the database to analyze the total calories, protein, lipids, vitamins, minerals, etc. of the generated menu and visually display this information.

[0105] The "means for suggesting missing nutrients based on the analysis results" is a function that compares the results of the nutritional analysis of the menu with the amount of nutrients needed per day, identifies missing nutrients, and suggests ingredients or alternative foods to replenish them.

[0106] The "means for displaying the menu and nutritional value information on a user device" is a function for visually displaying the generated menu and its nutritional information in graph or list format on a device such as a smartphone or tablet used by the user.

[0107] "Means for managing daily menus and nutritional intake status in calendar format and providing users with past and future information" is a function that displays the user's daily food records and nutritional intake status in calendar format, allowing them to refer to past data and future plans.

[0108] "Means for an information processing device to provide information corresponding to a question from a user via voice input" is a function that enables an information processing device on a server or cloud to analyze the content of a question when the user asks it via voice input and provide an appropriate answer in voice or text.

[0109] The "means for compressing images of ingredients and quickly transmitting them to an information processing device" is a function for appropriately compressing large amounts of ingredient images, improving communication speed and data transmission efficiency, and transmitting them to a server or cloud.

[0110] The present invention is a system for nutritional management that takes images of ingredients in a refrigerator, analyzes the images, and generates optimal menus. This system provides users with a means for efficiently managing their daily nutrition and making effective use of ingredients. Specific embodiments of the system are described below.

[0111] System hardware and software configuration

[0112] This system is operated using a device such as a smartphone or tablet. The device has the function of taking photos of ingredients and sending them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on that information. The nutritional value of the generated menu is analyzed and visually displayed to the user.

[0113] The device must be equipped with a high-resolution camera and internet connection. The server must have a powerful computer and a large database. Image recognition algorithms such as YOLO (You Only Look Once) and TensorFlow are used for image recognition.

[0114] Software used

[0115] Image recognition algorithms: YOLO, TensorFlow

[0116] Database management systems: MySQL, PostgreSQL

[0117] Cloud platforms: AWS (Amazon Web Services), Google Cloud Platform

[0118] Specific examples

[0119] For example, a user can take a photo of "chicken, tomatoes, and spices" in the refrigerator. The device compresses the image and quickly sends it to the server. The server analyzes the received image and identifies these ingredients. The server then uses its database to search for the most suitable recipe and suggests a menu item such as "chicken steak with tomato salsa." The server also analyzes and visualizes the nutritional value of this menu item (total calories, protein, vitamin C, etc.). Furthermore, if it is determined that the user is lacking in vitamin D, the server will suggest adding mushrooms or fish. Users can check this information on their device and use it to plan their daily meals.

[0120] Prompt Sentence Examples

[0121] You can use prompts like the following for your generative AI model:

[0122] A user takes a picture of the ingredients in their refrigerator and sends it to the server. This image shows chicken, tomatoes, and spices. Suggest a meal plan using these ingredients, analyze their nutritional value, and suggest additional ingredients if certain nutrients are lacking.

[0123] As described above, this system allows users to efficiently utilize ingredients and manage their nutrition. Users can also easily ask questions via voice input and instantly obtain specific cooking methods and additional nutritional information. This makes it easier to plan and follow balanced meals.

[0124] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0125] Step 1:

[0126] The user takes a photo of the ingredients. The user uses the camera on their smartphone or tablet to take a photo of the ingredients in the refrigerator. For example, they take a photo of chicken, tomatoes, spices, etc. stored in the refrigerator. In this case, the input data is the image of the ingredients, and the output data is the image file stored in the device.

[0127] Step 2:

[0128] The device preprocesses the captured image. Specifically, it compresses the image data to reduce the file size and make it easier to send to the server. In this process, the input is a high-resolution food image, and the output is compressed image data. For example, the image is compressed using the JPEG format.

[0129] Step 3:

[0130] The terminal sends the preprocessed image data to the server. The compressed image data is uploaded to the server via the Internet. The input in this step is the compressed image data, and the output is the image data stored on the server.

[0131] Step 4:

[0132] The server performs image analysis. It uses image recognition algorithms such as YOLO and TensorFlow to analyze the uploaded image. The input in this step is the image data stored on the server, and the output is a list of identified ingredients. For example, chicken, tomatoes, spices, etc. in the image are automatically identified.

[0133] Step 5:

[0134] The server generates a menu based on the identified ingredient information. It searches recipe information stored in a database and proposes an optimal menu, taking into account the user's nutritional needs and the expiration dates of ingredients. The input in this step is the identified ingredient list, and the output is the generated menu information. For example, a specific menu such as "chicken steak with tomato salsa" is proposed.

[0135] Step 6:

[0136] The server analyzes the nutritional value of the generated menu. Using the nutritional information of each ingredient registered in the database, it quantifies and visualizes nutrients such as total calories, protein, fat, vitamins, and minerals. The input in this step is the generated menu information, and the output is numerical information on nutritional value and graphed data.

[0137] Step 7:

[0138] The server identifies any nutrient deficiencies based on the analysis results and suggests alternative foods or additional ingredients. For example, if it determines that a person is deficient in vitamin D, it will suggest adding mushrooms or fish. The input in this step is numerical information on nutritional value, and the output is suggested information on deficient nutrients.

[0139] Step 8:

[0140] The device displays the generated menu and nutritional information to the user. Specifically, the information is displayed in a visually easy-to-understand graph or list format. The input in this step is the menu information and nutritional information, and the output is the visualized data displayed on the user device.

[0141] Step 9:

[0142] The user uses the voice question function. They can ask questions about cooking methods and additional nutritional information by voice input. For example, if they ask, "How do I cook this dish?", the server analyzes it and responds by voice or text. The input in this step is the voice question data, and the output is the response data from the server.

[0143] Step 10:

[0144] The device manages daily menus and nutritional intake status in a calendar format. Daily food records and nutritional intake status are compiled in a calendar format, and past data and future plans can be viewed. The input in this step is daily food data, and the output is information presented in calendar format.

[0145] (Application example 1)

[0146] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0147] Conventional refrigerator food management systems have had issues with the effective use of ingredients and insufficient nutritional management. In addition, because they are unable to propose optimal delivery menus that take into account the ingredients in the refrigerator, users are likely to waste ingredients or eat meals that are nutritionally unbalanced.

[0148] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0149] In this invention, the server includes means for taking images of ingredients, means for transmitting the taken images to the server, means for the server to analyze the images and identify ingredients, means for generating a menu based on the identified ingredients, means for analyzing the nutritional value of the generated menu and graphing and quantifying it, means for presenting nutrients that are lacking based on the analysis results, means for displaying information on the menu and nutritional values ​​on a user terminal, and means for suggesting an optimal delivery menu via the user terminal. This allows users to make the most of the ingredients in their refrigerators and use delivery services while maintaining a nutritionally balanced diet.

[0150] "Means for taking images of ingredients" refers to a device or function that uses a camera to take pictures of ingredients in the refrigerator.

[0151] "Means for transmitting the captured images to a server" refers to a function or process for transmitting the captured images to a remote server via the Internet or other communication means.

[0152] "Means for the server to analyze the image and identify ingredients" refers to the function of the server analyzing the image received by the server using machine learning algorithms or image recognition technology to accurately identify ingredients in the image.

[0153] The "means for generating a menu based on the identified ingredients" refers to a function that automatically suggests appropriate recipes and menus based on the identified ingredient information.

[0154] "Means for analyzing, graphing, and quantifying the nutritional value of the generated menu" refers to a function that analyzes the nutritional information of the proposed menu, graphs the results for visual display, and provides them as specific numerical data.

[0155] "Means for suggesting nutrients that are lacking based on the analysis results" refers to a function that suggests nutrients that the user is lacking and additional ingredients that are needed based on the results of the nutritional value analysis of the menu.

[0156] "Means for displaying the menu and nutritional value information on the user terminal" refers to a function for displaying the proposed menu and its nutritional value information on the user's terminal such as a smartphone or tablet.

[0157] "Means for proposing the most suitable delivery menu via the user terminal" refers to a function that provides the most suitable delivery menu taking into consideration the ingredients in the user's refrigerator and their nutritional needs.

[0158] The present invention relates to a system that takes images of ingredients in a refrigerator, analyzes the images to generate optimal menus, and also suggests food delivery services. This system provides a means for users to effectively use ingredients in the refrigerator while achieving nutritionally balanced meals.

[0159] System Overview

[0160] The system is operated through a user device such as a smartphone or tablet. The user device has the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on them. The generated menu is then analyzed for nutritional value and visually displayed to the user. The system also suggests optimal food delivery menus based on the ingredients in the user's refrigerator and their nutritional needs.

[0161] Specific Embodiments of the System

[0162] Photograph and send ingredients

[0163] A user takes a photo of the food in the refrigerator using a device such as a smartphone. The image is then sent to a server via an application. At this time, the device compresses the image data and processes it to enable quick transmission.

[0164] Image analysis

[0165] The server analyzes the received image. It uses an image recognition algorithm to automatically identify ingredients. For example, ingredients such as chicken, tomatoes, and spices can be identified. Technologies such as YOLO (You Only Look Once) and TensorFlow are used for the analysis.

[0166] Menu generation

[0167] The server then searches the database for the most suitable recipe based on the identified ingredients. It generates the optimal menu, taking into account the user's nutritional needs and the expiration dates of the ingredients. For example, it might suggest a menu item such as "chicken steak with tomato salsa."

[0168] Nutritional analysis and presentation

[0169] The server analyzes the nutritional value of the generated menu. The analysis uses the nutritional information of each ingredient registered in the database. The amounts of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified. This nutritional information is then graphed and displayed visually to the user.

[0170] Suggestions for nutrient deficiencies

[0171] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For those nutrients, it suggests alternative foods or additional ingredients. For example, if you are deficient in vitamin D, it suggests eating mushrooms or fish.

[0172] Food delivery menu suggestions

[0173] The system proposes optimal food delivery menus via the user's device, taking into account the ingredients in the refrigerator and the user's nutritional needs, allowing users to easily select nutritionally balanced meals through the delivery service.

[0174] Nutritional Information

[0175] The user's device displays the menu and nutritional information received from the server. The user can practice a balanced diet based on the visually displayed graphs and list format information.

[0176] Specific examples

[0177] Let's say a user takes a photo of their refrigerator and it shows "chicken, tomatoes, and spices." The server analyzes it and identifies these ingredients. The server then suggests a meal such as "chicken steak with tomato salsa," and analyzes and visualizes its nutritional value (calories, protein, vitamin C, etc.). Furthermore, if the server determines that the user is lacking in vitamin D, it suggests adding mushrooms or fish. The user can check this information on their device and use it to plan their daily meals.

[0178] Prompt Sentence Examples

[0179] "You analyzed images of the inside of the refrigerator and identified chicken, tomatoes, and onions. Please use these to suggest the best delivery menu. Also, please show the nutritional value of the suggested menu."

[0180] The above is an embodiment of the present invention. This system allows users to effectively use ingredients and efficiently manage nutrition.

[0181] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0182] Step 1:

[0183] A user takes a picture of the food in the refrigerator with their smartphone. At this time, the device uses its camera function to capture the image and saves the image data in the application. The input is the image of the food in the refrigerator, and the output is the saved image data.

[0184] Step 2:

[0185] The image data captured by the device is compressed and sent to the server. This process reduces the size of the image data so that it can be sent quickly. The input is the saved image data, and the output is the compressed image data.

[0186] Step 3:

[0187] The server decodes the received compressed image data and applies image analysis algorithms to identify ingredients. The server uses YOLO or TensorFlow to identify each ingredient in the image. The input is the compressed image data, and the output is a list of identified ingredients.

[0188] Step 4:

[0189] The server searches the database for the best recipe based on the ingredient list and generates a menu. The generated menu takes into account the user's nutritional needs and the expiration dates of the ingredients. The input is the identified ingredient list, and the output is the generated menu.

[0190] Step 5:

[0191] The server analyzes the nutritional value of the generated menu, quantifies the amount of each nutrient, and visually graphs it. The input is the generated menu, and the output is the analyzed nutritional value data and its graphical representation.

[0192] Step 6:

[0193] The server compares daily nutritional intake with the analysis results, identifies nutrient deficiencies, and suggests additional ingredients. The input is the analyzed nutritional value data, and the output is the nutrient deficiencies and suggestions for supplementing them.

[0194] Step 7:

[0195] The server sends the generated menu, nutritional value data, and suggested information on nutrient deficiencies to the user terminal, which then displays them. The input is the nutritional value data, suggested nutrient deficiencies, and the generated menu, and the output is the information displayed on the user terminal.

[0196] Step 8:

[0197] The user terminal proposes an optimal food delivery menu that takes into account the ingredients in the refrigerator and the user's nutritional needs. The input is the list of ingredients in the refrigerator and the user's nutritional needs, and the output is the optimal food delivery menu.

[0198] The above are the specific processing steps of the system program that realizes this application example. This allows users to make effective use of ingredients, efficiently manage their nutrition, and use the optimal food delivery menu.

[0199] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0200] The present invention relates to a system that takes pictures of ingredients in a refrigerator, analyzes the pictures, and generates optimal menus for nutritional management. A feature of the present invention is that by combining it with an emotion engine that recognizes the user's emotions, it is possible to suggest more personalized meals.

[0201] System Overview

[0202] This system is primarily operated through devices such as smartphones and tablets. The devices have the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on them. It also has a function to analyze the user's emotional state using an emotion engine and adjust the menu and nutritional suggestions as described below.

[0203] Specific Embodiments of the System

[0204] 1. Take a photo of the ingredients and send it

[0205] A user takes a photo of the food in the refrigerator using a device such as a smartphone. The image is then sent to a server via an application. The device compresses the image data and processes it to ensure fast transmission.

[0206] 2. Image Analysis

[0207] The server analyzes the received image. An image recognition algorithm is used for the analysis, and ingredients are automatically identified. For example, ingredients such as chicken, tomatoes, and spices are identified. The algorithm for analyzing the image uses technologies such as YOLO (You Only Look Once) and TensorFlow.

[0208] 3. Menu generation

[0209] The server then searches the database for the most suitable recipe based on the identified ingredients. It generates the optimal menu, taking into account the user's nutritional needs and the expiration dates of the ingredients. For example, it might suggest a menu item such as "chicken steak with tomato salsa."

[0210] 4. Analysis and presentation of nutritional value

[0211] The server analyzes the nutritional value of the generated menu. The analysis uses the nutritional information of each ingredient registered in the database. The amounts of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified. This nutritional information is then graphed and displayed visually to the user.

[0212] 5. Suggestions for nutrient deficiencies

[0213] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For those nutrients, it suggests alternative foods or additional ingredients. For example, if you are deficient in vitamin D, it suggests eating mushrooms or fish.

[0214] 6. Menu and nutrition information display

[0215] The device receives the menu and nutritional information from the server and displays it to the user, who can then practice a balanced diet based on the visually displayed graphs and list-style information.

[0216] 7. Voice question function

[0217] Users can use voice input to ask questions or get more detailed information. For example, if they ask, "How do I cook this dish?", the server will analyze the voice data and provide the appropriate cooking method. The answer will be played back as text or audio.

[0218] 8. Calendar Management

[0219] The device has a function to manage daily menus and nutritional intake status in a calendar format. Users can view past data and future plans and manage their daily eating habits. This allows for long-term nutritional management.

[0220] 9. Incorporating an Emotional Engine

[0221] The server incorporates an emotion engine to recognize emotions from the user's facial expressions and voice. This emotional data is reflected in meal suggestions. For example, if the user is feeling stressed, the server can suggest ingredients and menus that have a relaxing effect.

[0222] 10. Emotional Adjustment

[0223] The server analyzes the accumulated emotional data and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake. For example, if a particular ingredient or menu improves the user's mood, personalized suggestions will be made based on that information.

[0224] Specific Examples

[0225] For example, suppose a user takes a photo of their refrigerator and sees that it contains "chicken, tomatoes, and spices." The server analyzes the photo and identifies these ingredients. The server then suggests a menu item such as "chicken steak with tomato salsa," and also considers the user's emotional state to suggest adding ingredients that have a relaxing effect. The server also analyzes and visualizes nutritional values ​​(calories, protein, vitamin C, etc.), suggesting adding mushrooms or fish to compensate for vitamin D deficiency. Users can check this information on their device and use it to plan their daily meals.

[0226] The above is an embodiment of the present invention. This system allows users to effectively use ingredients and efficiently manage nutrition, and also allows users to receive personalized suggestions based on their emotions.

[0227] The processing flow will be explained below.

[0228] Step 1:

[0229] When a user launches the smartphone application and takes a photo of the ingredients in the refrigerator, the device activates its camera function and displays a screen on which the user can take a photo of the ingredients.

[0230] Step 2:

[0231] The user checks the photo they have taken and presses the "Send" button in the application. The device sends the photo data to the server. When sending, the device compresses the image data to ensure it can be sent quickly.

[0232] Step 3:

[0233] The server receives the transmitted photo and runs an image recognition algorithm to analyze the received image data.

[0234] Step 4:

[0235] The server uses image recognition algorithms to identify ingredients in the received image, and lists the ingredients identified, for example, "chicken, tomatoes, and spices."

[0236] Step 5:

[0237] The server searches the database for the best recipe based on the identified ingredients, taking into account the user's nutritional needs (e.g., calorie restriction, vitamin intake, etc.) and expiration dates of the ingredients.

[0238] Step 6:

[0239] The server generates an optimal menu. For example, it might suggest "chicken steak with tomato salsa." Details of the suggested menu are stored in a database.

[0240] Step 7:

[0241] The server analyzes the nutritional value of the generated menu. The nutritional information includes calories, protein, fat, vitamins, minerals, etc. This information is obtained from a database.

[0242] Step 8:

[0243] The server quantifies the amount of each nutrient and calculates the total amount, then graphs the calculated nutritional information to generate data that can be displayed visually.

[0244] Step 9:

[0245] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For example, if there is a vitamin D deficiency, it will recommend additional foods and ingredients to supplement it.

[0246] Step 10:

[0247] The server transmits information including incomplete nutritional suggestions to the terminal, including menu details and nutritional value information.

[0248] Step 11:

[0249] The terminal displays the received information to the user in graph and list format so that the user can easily understand it.

[0250] Step 12:

[0251] Users can ask questions or get more information by speaking, for example, "How do I prepare this dish?"

[0252] Step 13:

[0253] The device converts the voice input into text and sends the text data to the server.

[0254] Step 14:

[0255] The server analyzes the question and generates an appropriate answer, such as "How to cook chicken steak..." and provides specific instructions.

[0256] Step 15:

[0257] The server generates a response and sends it to the terminal, which displays or plays it aloud to the user.

[0258] Step 16:

[0259] The device manages daily menus and nutritional intake in a calendar format, allowing users to check the calendar and view past data and upcoming plans.

[0260] Step 17:

[0261] The device activates an emotion engine that recognizes emotions from the user's facial expressions and voice. The user's emotional state is collected through the application. For example, facial expression analysis can determine whether the user is smiling, and voice analysis can determine whether the user is under stress.

[0262] Step 18:

[0263] The server analyzes the collected emotional data using an emotion engine. Based on the analysis results, it adjusts menu and nutrition suggestions based on the user's emotions. For example, if the user is feeling stressed, it will suggest ingredients and menus that have a relaxing effect.

[0264] Step 19:

[0265] The server analyzes the accumulated emotional data over the long term and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake. For example, if a particular ingredient or menu improves the user's mood, personalized suggestions will be made based on that information.

[0266] In this way, by performing specific actions at each step, the system efficiently supports food inventory management and nutritional management.In addition, personalized suggestions that take into account the user's emotional state can provide a more satisfying service.

[0267] Example 2

[0268] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0269] Conventional menu suggestion systems have difficulty effectively utilizing ingredients in the refrigerator and providing balanced nutritional management. Furthermore, they are unable to provide personalized suggestions that take the user's emotional state into account, resulting in low satisfaction. Furthermore, they lack the ability to provide appropriate answers to voice questions from users and to manage past nutritional intake history. By resolving these issues, it is necessary to improve the quality and satisfaction of users' dietary habits.

[0270] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for a user to take an image of ingredients; means for compressing the taken image at a terminal and sending it to the server; means for the server to analyze the received image and identify ingredients; means for generating a menu based on the identified ingredients, taking into consideration the user's nutritional needs and the expiration dates of the ingredients; means for analyzing the nutritional value of the generated menu and quantifying and graphing calorie and nutrient values; means for identifying missing nutrients based on the analysis results and suggesting substitute foods and additional ingredients; means for displaying information about the menu and nutritional value on a user terminal; means for the server to provide appropriate information in response to questions from the user via voice input, and for the user terminal to provide answers in text or voice; means for managing daily menus and nutritional intake status in a calendar format and providing the user with past and future information; means for analyzing emotions from the user's facial expressions and voice, and personalizing meal suggestions using emotional data; and means for suggesting ingredients and menus with a relaxing effect according to a specific emotional state, and optimizing individual meal suggestions by analyzing the relationship between past emotions and nutritional intake. This will allow for effective use of ingredients in the refrigerator, optimizing the user's nutritional management, and enabling personalized meal suggestions based on the user's emotional state.

[0271] "User" refers to a person who uses the system to take pictures of ingredients and receive menu suggestions and nutritional management.

[0272] "Terminal" refers to an electronic device such as a smartphone or tablet that takes and sends images of ingredients, communicates with the server, and displays information.

[0273] The "server" refers to a remote data processing device that analyzes images of ingredients, generates menus, analyzes nutritional values, suggests nutrient deficiencies, and so on.

[0274] "Image analysis" refers to the technical process of analyzing received image data of ingredients to automatically identify the ingredients.

[0275] "Ingredient identification" refers to the process of identifying and listing types of ingredients through image analysis.

[0276] "Menu generation" refers to the process of determining the optimal menu based on the identified ingredients, taking into account the user's nutritional needs and the expiration dates of the ingredients.

[0277] "Nutritional analysis" refers to the process of calculating the nutritional information of the generated menu and quantifying and graphing the calories and amount of each nutrient.

[0278] "Nutrient deficiency suggestion" refers to the process of identifying missing nutrients based on analyzed nutritional value data and suggesting appropriate substitute foods or additional ingredients.

[0279] "Voice input" refers to a means by which a user uses voice to ask questions or give instructions to a system.

[0280] "Emotion analysis" refers to the process of recognizing emotions from a user's facial expressions and voice and analyzing that data.

[0281] "Personalization" refers to providing information and services that are individually optimized based on a user's specific conditions and tendencies.

[0282] "Calendar management" refers to the process of recording daily menus and nutritional intake status in calendar format and providing users with past data and future schedules.

[0283] The present invention relates to a system that takes images of ingredients in a refrigerator, analyzes the images, creates optimal menus, and manages nutrition. Furthermore, the present invention can improve user satisfaction and health management by recognizing the user's emotions and providing personalized meal suggestions based on those emotions.

[0284] System Overview

[0285] This system is primarily operated through devices such as smartphones and tablets. The devices have the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on that information. It also has a function to analyze the user's emotional state using an emotion engine and adjust the menu and nutritional suggestions accordingly.

[0286] Hardware and software used

[0287] This system uses the following hardware and software:

[0288] Devices: smartphones, tablets

[0289] Server: High-performance data processing server

[0290] Image analysis algorithm: YOLO (You Only Look Once), TensorFlow

[0291] Emotion analysis engine: speech recognition software, facial recognition software

[0292] System operation flow

[0293] The system works as follows:

[0294] 1. Take a photo of the ingredients and send it

[0295] The user uses the camera on their smartphone or tablet to take a photo of the food in their refrigerator.

[0296] The device compresses the captured images and quickly transmits the data to the server.

[0297] 2. Image Analysis

[0298] The server decodes the received image and converts it into an analyzable format.

[0299] The server uses image recognition algorithms such as YOLO and TensorFlow to automatically identify ingredients in the image.

[0300] For example, ingredients such as chicken, tomatoes, and spices are identified.

[0301] 3. Menu generation

[0302] The server searches the database for the most suitable recipe based on the identified ingredient list.

[0303] The server generates optimal menus taking into account the user's nutritional needs and the expiration dates of ingredients.

[0304] For example, a menu suggestion might be "chicken steak with tomato salsa."

[0305] 4. Analysis and presentation of nutritional value

[0306] The server analyzes the nutritional value of the generated menu, using the nutritional information of each ingredient registered in the database.

[0307] The amount of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified.

[0308] This nutritional information is graphed and displayed visually to the user.

[0309] 5. Suggestions for nutrient deficiencies

[0310] Based on the analysis results, the server compares the nutrients needed per day with the nutritional value of the generated menu and identifies any nutrients that are lacking.

[0311] For those who are lacking in nutrients, we suggest alternative foods or additional ingredients. For example, if you are lacking in vitamin D, we suggest eating mushrooms or fish.

[0312] 6. Menu and nutrition information display

[0313] The terminal displays the menu and nutritional information received from the server to the user.

[0314] Users can practice balanced eating habits based on visually displayed graphs and list-style information.

[0315] 7. Voice question function

[0316] Users can use voice input to ask questions or for additional information.

[0317] For example, if you ask, "How do you cook this dish?", the server will analyze the voice data and provide the appropriate cooking method as an answer. The answer will be played back in text or audio.

[0318] 8. Calendar Management

[0319] The device has the function of managing daily menus and nutritional intake in calendar format.

[0320] Users can view past data and future plans and manage their daily eating habits.

[0321] 9. Incorporating an Emotional Engine

[0322] The server incorporates an emotion engine to recognize emotions from the user's facial expressions and voice.

[0323] Emotional data is reflected in meal suggestions. For example, if a user is feeling stressed, the system can suggest ingredients and menu items that have a relaxing effect.

[0324] 10. Emotional Adjustment

[0325] The server analyzes the accumulated emotional data and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake.

[0326] For example, if a particular ingredient or menu item improves a user's mood, personalized suggestions can be made based on that information.

[0327] Specific examples

[0328] For example, suppose a user takes a photo of their refrigerator, which includes "chicken, tomatoes, and spices." The device compresses this and sends it to the server. The server analyzes the photo and identifies these ingredients. The server then suggests a menu item such as "chicken steak with tomato salsa," and also considers the user's emotional state and suggests adding ingredients that have a relaxing effect. The server also analyzes and visualizes nutritional values ​​(calories, protein, vitamin C, etc.), suggesting adding mushrooms or fish to compensate for vitamin D deficiency. Users can check this information on their device and use it to plan their daily meals.

[0329] This invention allows users to make effective use of ingredients and efficiently manage their nutrition, and also allows users to receive personalized suggestions based on their emotions.

[0330] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0331] Step 1: Photograph the ingredients and send the image

[0332] The user uses a device (smartphone or tablet) to take a photo of the food in the refrigerator.

[0333] Input: Images of ingredients in the refrigerator.

[0334] The device compresses the images it takes, reducing the amount of data it takes to send them over the network.

[0335] The terminal transmits the compressed image data to the server.

[0336] Output: Compressed image data sent to the server.

[0337] Step 2: Image analysis and ingredient identification

[0338] The server decodes the received image data and converts it into an analyzable format.

[0339] Input: Received compressed image data.

[0340] The server uses image recognition algorithms such as YOLO (You Only Look Once) and TensorFlow to automatically identify ingredients.

[0341] Image analysis algorithms identify the type of ingredient (e.g., chicken, tomato, spices, etc.) from the image.

[0342] Output: A list of identified ingredients.

[0343] Step 3: Create a menu

[0344] The server searches the registered database for the most suitable recipe based on the identified ingredient list.

[0345] Input: Identified ingredient list, recipe information in database.

[0346] The server generates optimal menus taking into account the user's nutritional needs and the expiration dates of ingredients.

[0347] For example, a menu such as "chicken steak with tomato salsa" may be generated.

[0348] Output: The generated menu information.

[0349] Step 4: Nutritional analysis and presentation

[0350] The server analyzes the nutritional value of the generated menu.

[0351] Input: Generated menu information, nutrition information in the database.

[0352] The amount of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified.

[0353] This nutritional information is graphed and formatted in a way that can be visually presented to the user.

[0354] Output: Nutritional values ​​are displayed in graph and list format.

[0355] Step 5: Suggesting nutrient deficiencies

[0356] The server analyzes the nutritional value of the menu and identifies any nutrients that are lacking by comparing them with the daily required nutrients.

[0357] Input: Quantified nutritional value data, reference values ​​for nutrients needed per day.

[0358] The server will identify any nutrient deficiencies and suggest substitutions or additional ingredients.

[0359] For example, if you are deficient in vitamin D, it is suggested that you "consume additional mushrooms and fish."

[0360] Output: Suggested information on nutrient deficiencies.

[0361] Step 6: Display menu and nutrition information

[0362] The server sends the generated menu and nutritional information to the terminal.

[0363] Input: Menu information, nutritional information.

[0364] The information received by the terminal is visually displayed to the user.

[0365] The device displays the information in graph and list format, and users can use the information to practice a balanced diet.

[0366] Output: The visual information that is displayed to the user.

[0367] Step 7: Voice Question Function

[0368] Users use voice input to ask questions or for additional information.

[0369] Input: Audio data.

[0370] The device converts the voice data into text and sends it to the server.

[0371] The server analyzes the question and generates an appropriate answer.

[0372] The device will present the answer to the user in text or audio.

[0373] For example, if you ask, "How do you cook this dish?", the cooking instructions will be answered in text or voice.

[0374] Output: The answer information provided to the user.

[0375] Step 8: Calendar Management

[0376] The device manages daily menus and nutritional intake in calendar format.

[0377] Input: Menu information, nutritional intake data.

[0378] Users can view past data and future schedules.

[0379] The terminal displays information in calendar format, enabling long-term nutritional management.

[0380] Output: Information displayed in a calendar format.

[0381] Step 9: Incorporating the Emotion Engine

[0382] The server uses an emotion engine to recognize emotions from the user's facial expressions and voice data.

[0383] Input: User's facial expression data, voice data.

[0384] A sentiment analysis engine analyzes this data and infers the user's emotional state (e.g., stress, happiness, etc.).

[0385] Output: Emotional state data.

[0386] Step 10: Emotional Adjustment

[0387] The server analyzes the accumulated emotional data and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake.

[0388] Input: Emotional state data, nutritional intake data.

[0389] Depending on your specific emotional state, it will suggest foods and menus that have a relaxing effect.

[0390] For example, if a user is feeling stressed, a menu containing ingredients that have a relaxing effect will be suggested based on that information.

[0391] Output: Personalized meal suggestion information.

[0392] (Application example 2)

[0393] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0394] Conventional food management systems have limitations in efficiently managing ingredients in the refrigerator and suggesting appropriate menus. Furthermore, they make uniform suggestions without considering the user's emotions, resulting in insufficient personalization to meet individual needs. Furthermore, there has been a lack of systems that can utilize new devices such as smart glasses to recognize ingredients and emotions in real time.

[0395] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for capturing images of ingredients, a means for transmitting the captured images to the server, a means for the server to analyze the images and identify ingredients, a means for generating a menu based on the identified ingredients, a means for analyzing, graphing, and quantifying the nutritional value of the generated menu, a means for presenting missing nutrients based on the analysis results, a means for displaying the menu and nutritional value information on a user terminal, a means for recognizing the user's emotions and reflecting them in meal suggestions, and a means for using smart glasses to recognize ingredients and emotions in real time and suggest a menu. This enables efficient recognition of ingredients in the user's refrigerator and personalized menu suggestions based on the user's individual emotional state. Furthermore, the use of smart glasses enables real-time ingredient recognition and emotion recognition, enabling more immediate and accurate information provision.

[0396] "Means for taking images of ingredients" refers to functionality that includes devices and software for capturing images of ingredients in a refrigerator or other storage location.

[0397] The "means for transmitting the captured image to the server" is a function for transmitting the image captured from the smart glasses or other terminal to the server via data communication.

[0398] The "means for the server to analyze the image and identify ingredients" refers to a system that uses algorithms or techniques to analyze received image data and identify ingredients.

[0399] The "means for generating a menu based on the identified ingredients" is a function that searches a database for the most suitable recipe based on the identified ingredients and suggests a menu.

[0400] The "means for analyzing, graphing and quantifying the nutritional value of the generated menu" is a function for quantifying each nutrient in the generated menu and visually displaying it.

[0401] The "means for presenting missing nutrients based on the analysis results" is a function for identifying missing nutrients by comparing them with the calculated nutritional values ​​and suggesting appropriate ingredients and supplements.

[0402] The "means for displaying the menu and nutritional value information on a user terminal" is a function for displaying the generated menu and its nutritional value information on a user terminal such as smart glasses or a smartphone.

[0403] "Means for recognizing the user's emotions and reflecting them in meal suggestions" is a function that analyzes the user's facial expressions and voice to identify their emotional state and then makes personalized meal suggestions based on that information.

[0404] "Means for using smart glasses to recognize ingredients and emotions in real time and suggest menus" is a function that uses the cameras and sensors in smart glasses to recognize ingredients and the user's emotions in real time and suggests menus based on that information.

[0405] The present invention provides a system for efficiently managing ingredients in a refrigerator and proposing personalized menus based on the user's emotions. Here, specific embodiments of the present invention will be described.

[0406] System Overview

[0407] The main components of this system are smart glasses (or a user terminal such as a smartphone), a server, and software for recognizing images of ingredients in the refrigerator and the user's emotions.

[0408] 1. Food and emotion recognition

[0409] A user uses smart glasses to check the ingredients in the refrigerator. At this time, the camera in the smart glasses captures an image of the ingredients and sends the image data to the server. At the same time, the sensors in the smart glasses capture the user's facial expressions and voice to recognize their emotions. For example, if a user opens the refrigerator and says, "I want to relax today," that voice data is also sent to the server.

[0410] 2. Image analysis and emotion analysis on the server

[0411] The server first analyzes the received image data, using the YOLO (You Only Look Once) model and TensorFlow algorithms to automatically identify the ingredients in the refrigerator.

[0412] The received facial and voice data is then analyzed using an emotion engine, which is used to identify the user's current emotional state (e.g., relaxed, stressed, depressed, etc.).

[0413] 3. Menu Creation and Nutritional Analysis

[0414] Based on the identified ingredients and the user's emotional state, the server searches for the most suitable recipe from the database and generates a menu. For example, if the ingredients are "chicken, tomato, and spices" and the user is recognized as "feeling stressed," the server will suggest "chicken steak with tomato salsa" with "herbal tea," which has a relaxing effect, as a side dish.

[0415] To analyze the nutritional value of the generated menu, the system refers to a nutrition database and converts information such as total calories, protein, fat, vitamins, and minerals into numerical values ​​and graphs, which the user can visually check.

[0416] 4. Display on the user's device

[0417] The generated menu information and nutritional information are displayed on the user's device, such as smart glasses or a smartphone, allowing the user to efficiently use the ingredients in their refrigerator and practice nutritionally balanced meals.

[0418] 5. Examples of prompts

[0419] Specific examples of prompts include:

[0420] "It recognizes the ingredients in the refrigerator and suggests personalized meals based on the user's emotions. For example, if it analyzes an image containing "chicken, tomatoes, and spices" and determines that the user is feeling stressed, it will suggest a meal that includes ingredients that have a relaxing effect."

[0421] This system allows users to make the most of the ingredients in their refrigerator and receive personalized meal suggestions based on their emotions. It also recognizes ingredients and emotions in real time and provides appropriate information instantly.

[0422] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0423] Step 1:

[0424] A user uses smart glasses to check the ingredients in the refrigerator and captures the images with the camera. The input is the image of the ingredients in the refrigerator, and the output is the captured image data.

[0425] Step 2:

[0426] The smart glasses transmit the captured image data to a server. The input is the captured image data, and the output is the transmission of the image data to the server. In actual operation, the smart glasses use a communication module to compress the image data and transmit it to the server via the Internet.

[0427] Step 3:

[0428] The server analyzes the received image data and identifies the ingredients. The input is the image data sent to the server, and the output is a list of identified ingredients. Specifically, the server uses the YOLO model to detect ingredients in the image and create a list of ingredient names.

[0429] Step 4:

[0430] The smart glasses capture the user's facial expressions and voice and send the data to the server. The input is the user's facial expressions and voice data, and the output is data transmission to the server. The smart glasses' sensors and microphone are used to capture the user's facial expressions and voice in real time and send the data to the server.

[0431] Step 5:

[0432] The server analyzes the received facial and voice data to recognize the user's emotions. The input is the facial and voice data sent to the server, and the output is the recognized emotion data. Specifically, the emotion engine is used to analyze the facial and voice data and identify the user's emotional state.

[0433] Step 6:

[0434] The server generates an optimal menu based on the identified ingredients and the recognized emotion data. The input is the identified ingredient list and emotion data, and the output is the generated menu. The server searches for appropriate recipes from the database and selects meals that suit the user's emotions.

[0435] Step 7:

[0436] The server analyzes the nutritional value of the generated menu and graphs and quantifies it. The input is the generated menu, and the output is the nutritional value and graph. The server refers to a nutrition database, tallying the nutritional information of each ingredient and calculating the nutritional value.

[0437] Step 8:

[0438] The server then presents the nutritional deficiencies based on the analysis results. The input is the nutritional analysis results, and the output is suggestions regarding the nutrients that are lacking. It identifies the nutrients that are lacking and suggests additional ingredients or supplements to make up for them.

[0439] Step 9:

[0440] The generated menu and nutritional information are displayed on the user's device. The input is the generated menu and nutritional information, and the output is the display on the user's device. The menu and nutritional information are displayed on smart glasses or a smartphone, allowing the user to visually confirm them.

[0441] Step 10:

[0442] The user asks a question by voice input, and the server provides the corresponding information. The input is the user's voice question, and the output is the answer from the server. The server uses a voice recognition system to analyze the user's question and provide appropriate information and explanations.

[0443] Step 11:

[0444] The user terminal manages daily menus and nutritional intake status in a calendar format, providing users with past and future information. The input is past and current menu data, and the output is a visual display in calendar format. Users can manage their nutrition over the long term.

[0445] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0446] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0447] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0448] [Second embodiment]

[0449] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0450] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0451] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0452] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0453] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0454] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0455] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0456] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0457] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0458] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0459] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0460] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0461] The present invention relates to a system for nutritional management that takes images of ingredients in a refrigerator, analyzes the images, and generates optimal menus. This system provides users with a means for efficiently managing their daily nutrition and making effective use of ingredients.

[0462] System Overview

[0463] This system is operated through devices such as smartphones and tablets. The devices have the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on that information. The nutritional value of the generated menu is analyzed and visually displayed to the user.

[0464] Specific Embodiments of the System

[0465] 1. Take a photo of the ingredients and send it

[0466] A user takes a photo of the food in the refrigerator using a device such as a smartphone. The image is then sent to a server via an application. At this time, the device compresses the image data and processes it to enable quick transmission.

[0467] 2. Image Analysis

[0468] The server analyzes the received image. An image recognition algorithm is used for the analysis, and ingredients are automatically identified. For example, ingredients such as chicken, tomatoes, and spices are identified. The algorithm for analyzing the image uses technologies such as YOLO (You Only Look Once) and TensorFlow.

[0469] 3. Menu generation

[0470] The server then searches the database for the most suitable recipe based on the identified ingredients. It generates the optimal menu, taking into account the user's nutritional needs and the expiration dates of the ingredients. For example, it might suggest a menu item such as "chicken steak with tomato salsa."

[0471] 4. Analysis and presentation of nutritional value

[0472] The server analyzes the nutritional value of the generated menu. The analysis uses the nutritional information of each ingredient registered in the database. The amounts of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified. This nutritional information is then graphed and displayed visually to the user.

[0473] 5. Suggestions for nutrient deficiencies

[0474] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For those nutrients, it suggests alternative foods or additional ingredients. For example, if you are deficient in vitamin D, it suggests eating mushrooms or fish.

[0475] 6. Menu and nutrition information display

[0476] The device receives the menu and nutritional information from the server and displays it to the user, who can then practice a balanced diet based on the visually displayed graphs and list-style information.

[0477] 7. Voice question function

[0478] Users can use voice input to ask questions or get more detailed information. For example, if they ask, "How do I cook this dish?", the server will analyze the voice data and provide the appropriate cooking method. The answer will be played back as text or audio.

[0479] 8. Calendar Management

[0480] The device has a function to manage daily menus and nutritional intake status in a calendar format. Users can view past data and future plans and manage their daily eating habits. This allows for long-term nutritional management.

[0481] Specific Examples

[0482] For example, suppose a user takes a photo of their refrigerator and sees that it contains "chicken, tomatoes, and spices." The server analyzes the photo and identifies these ingredients. The server then suggests a meal such as "chicken steak with tomato salsa," and analyzes and visualizes its nutritional value (calories, protein, vitamin C, etc.). Furthermore, if the server determines that the user is lacking in vitamin D, it suggests adding mushrooms or fish. Users can check this information on their device and use it to plan their daily meals.

[0483] The above is an embodiment of the present invention. This system allows users to effectively use ingredients and efficiently manage nutrition.

[0484] The processing flow will be explained below.

[0485] Step 1:

[0486] When a user launches the smartphone application and takes a photo of the ingredients in the refrigerator, the device activates its camera function and displays a screen on which the user can take a photo of the ingredients.

[0487] Step 2:

[0488] The user checks the photo they have taken and presses the "Send" button in the application. The device sends the photo data to the server.

[0489] Step 3:

[0490] The server receives the transmitted photo and runs an image recognition algorithm to analyze the received image data.

[0491] Step 4:

[0492] The server uses image recognition algorithms to identify ingredients in the received image, and lists the ingredients identified, for example, "chicken, tomatoes, and spices."

[0493] Step 5:

[0494] The server searches the database for the best recipe based on the identified ingredients, taking into account the user's nutritional needs (e.g., calorie restriction, vitamin intake, etc.) and expiration dates of the ingredients.

[0495] Step 6:

[0496] The server generates an optimal menu. For example, it might suggest "chicken steak with tomato salsa." Details of the suggested menu are stored in a database.

[0497] Step 7:

[0498] The server analyzes the nutritional value of the generated menu. The nutritional information includes calories, protein, fat, vitamins, minerals, etc. This information is obtained from a database.

[0499] Step 8:

[0500] The server quantifies the amount of each nutrient and calculates the total amount, then graphs the calculated nutritional information to generate data that can be displayed visually.

[0501] Step 9:

[0502] The server compares the analysis results with the daily required nutrients and identifies any deficiencies. For example, if there is a vitamin D deficiency, it will recommend additional ingredients or foods to supplement it.

[0503] Step 10:

[0504] The server transmits information including incomplete nutritional suggestions to the terminal, including menu details and nutritional value information.

[0505] Step 11:

[0506] The terminal displays the received information to the user in graph and list format so that the user can easily understand it.

[0507] Step 12:

[0508] Users can ask questions or get more information by speaking, for example, "How do I prepare this dish?"

[0509] Step 13:

[0510] The device converts the voice input into text and sends the text data to the server.

[0511] Step 14:

[0512] The server analyzes the question and generates an appropriate answer, such as "How to cook chicken steak..." and provides specific instructions.

[0513] Step 15:

[0514] The server generates a response and sends it to the terminal, which displays or plays it aloud to the user.

[0515] Step 16:

[0516] The device manages daily menus and nutritional intake in a calendar format, allowing users to check the calendar and view past data and upcoming plans.

[0517] In this way, by performing specific actions at each step, the system efficiently supports food inventory management and nutritional management.

[0518] Example 1

[0519] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0520] Managing ingredients and planning nutritionally balanced meals is time-consuming and laborious, making it difficult for many people. It's particularly challenging to effectively utilize ingredients in the refrigerator and manage daily nutritional intake. Furthermore, it's difficult to instantly obtain specific cooking instructions and nutritional information for a menu. A system that can solve these problems is needed.

[0521] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0522] In this invention, the server includes means for analyzing and identifying images of ingredients, means for generating menus based on the identified ingredients, and means for analyzing and presenting the nutritional values ​​of the generated menus, thereby enabling users to efficiently use ingredients in their refrigerators and easily create balanced meal plans.

[0523] "Means for taking pictures of ingredients" is a function for taking pictures of ingredients in the refrigerator using a camera on a smartphone, tablet, or other device.

[0524] The "means for transmitting the captured image to the information processing device" is a function for appropriately compressing the captured image and transmitting it to an information processing device on a server or cloud.

[0525] "Means for the information processing device to analyze the image and identify ingredients" refers to a function that enables an information processing device located on a server or cloud to detect and identify ingredients in an image using image recognition algorithms such as YOLO or TensorFlow.

[0526] The "means for generating a menu based on the identified ingredients" is a function for automatically suggesting a menu by selecting the optimal recipe based on the identified ingredient information, taking into consideration the user's nutritional needs and the expiration date of the ingredients.

[0527] The "means for analyzing, graphing, and quantifying the nutritional value of the generated menu" is a function that uses the nutritional information of each ingredient registered in the database to analyze the total calories, protein, lipids, vitamins, minerals, etc. of the generated menu and visually display this information.

[0528] The "means for suggesting missing nutrients based on the analysis results" is a function that compares the results of the nutritional analysis of the menu with the amount of nutrients needed per day, identifies missing nutrients, and suggests ingredients or alternative foods to replenish them.

[0529] The "means for displaying the menu and nutritional value information on a user device" is a function for visually displaying the generated menu and its nutritional information in graph or list format on a device such as a smartphone or tablet used by the user.

[0530] "Means for managing daily menus and nutritional intake status in calendar format and providing users with past and future information" is a function that displays the user's daily food records and nutritional intake status in calendar format, allowing them to refer to past data and future plans.

[0531] "Means for an information processing device to provide information corresponding to a question from a user via voice input" is a function that enables an information processing device on a server or cloud to analyze the content of a question when the user asks it via voice input and provide an appropriate answer in voice or text.

[0532] The "means for compressing images of ingredients and quickly transmitting them to an information processing device" is a function for appropriately compressing large amounts of ingredient images, improving communication speed and data transmission efficiency, and transmitting them to a server or cloud.

[0533] The present invention is a system for nutritional management that takes images of ingredients in a refrigerator, analyzes the images, and generates optimal menus. This system provides users with a means for efficiently managing their daily nutrition and making effective use of ingredients. Specific embodiments of the system are described below.

[0534] System hardware and software configuration

[0535] This system is operated using a device such as a smartphone or tablet. The device has the function of taking photos of ingredients and sending them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on that information. The nutritional value of the generated menu is analyzed and visually displayed to the user.

[0536] The device must be equipped with a high-resolution camera and internet connection. The server must have a powerful computer and a large database. Image recognition algorithms such as YOLO (You Only Look Once) and TensorFlow are used for image recognition.

[0537] Software used

[0538] Image recognition algorithms: YOLO, TensorFlow

[0539] Database management systems: MySQL, PostgreSQL

[0540] Cloud platforms: AWS (Amazon Web Services), Google Cloud Platform

[0541] Specific examples

[0542] For example, a user can take a photo of "chicken, tomatoes, and spices" in the refrigerator. The device compresses the image and quickly sends it to the server. The server analyzes the received image and identifies these ingredients. The server then uses its database to search for the most suitable recipe and suggests a menu item such as "chicken steak with tomato salsa." The server also analyzes and visualizes the nutritional value of this menu item (total calories, protein, vitamin C, etc.). Furthermore, if it is determined that the user is lacking in vitamin D, the server will suggest adding mushrooms or fish. Users can check this information on their device and use it to plan their daily meals.

[0543] Prompt Sentence Examples

[0544] You can use prompts like the following for your generative AI model:

[0545] A user takes a picture of the ingredients in their refrigerator and sends it to the server. This image shows chicken, tomatoes, and spices. Suggest a meal plan using these ingredients, analyze their nutritional value, and suggest additional ingredients if certain nutrients are lacking.

[0546] As described above, this system allows users to efficiently utilize ingredients and manage their nutrition. Users can also easily ask questions via voice input and instantly obtain specific cooking methods and additional nutritional information. This makes it easier to plan and follow balanced meals.

[0547] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0548] Step 1:

[0549] The user takes a photo of the ingredients. The user uses the camera on their smartphone or tablet to take a photo of the ingredients in the refrigerator. For example, they take a photo of chicken, tomatoes, spices, etc. stored in the refrigerator. In this case, the input data is the image of the ingredients, and the output data is the image file stored in the device.

[0550] Step 2:

[0551] The device preprocesses the captured image. Specifically, it compresses the image data to reduce the file size and make it easier to send to the server. In this process, the input is a high-resolution food image, and the output is compressed image data. For example, the image is compressed using the JPEG format.

[0552] Step 3:

[0553] The terminal sends the preprocessed image data to the server. The compressed image data is uploaded to the server via the Internet. The input in this step is the compressed image data, and the output is the image data stored on the server.

[0554] Step 4:

[0555] The server performs image analysis. It uses image recognition algorithms such as YOLO and TensorFlow to analyze the uploaded image. The input in this step is the image data stored on the server, and the output is a list of identified ingredients. For example, chicken, tomatoes, spices, etc. in the image are automatically identified.

[0556] Step 5:

[0557] The server generates a menu based on the identified ingredient information. It searches recipe information stored in a database and proposes an optimal menu, taking into account the user's nutritional needs and the expiration dates of ingredients. The input in this step is the identified ingredient list, and the output is the generated menu information. For example, a specific menu such as "chicken steak with tomato salsa" is proposed.

[0558] Step 6:

[0559] The server analyzes the nutritional value of the generated menu. Using the nutritional information of each ingredient registered in the database, it quantifies and visualizes nutrients such as total calories, protein, fat, vitamins, and minerals. The input in this step is the generated menu information, and the output is numerical information on nutritional value and graphed data.

[0560] Step 7:

[0561] The server identifies any nutrient deficiencies based on the analysis results and suggests alternative foods or additional ingredients. For example, if it determines that a person is deficient in vitamin D, it will suggest adding mushrooms or fish. The input in this step is numerical information on nutritional value, and the output is suggested information on deficient nutrients.

[0562] Step 8:

[0563] The device displays the generated menu and nutritional information to the user. Specifically, the information is displayed in a visually easy-to-understand graph or list format. The input in this step is the menu information and nutritional information, and the output is the visualized data displayed on the user device.

[0564] Step 9:

[0565] The user uses the voice question function. They can ask questions about cooking methods and additional nutritional information by voice input. For example, if they ask, "How do I cook this dish?", the server analyzes it and responds by voice or text. The input in this step is the voice question data, and the output is the response data from the server.

[0566] Step 10:

[0567] The device manages daily menus and nutritional intake status in a calendar format. Daily food records and nutritional intake status are compiled in a calendar format, and past data and future plans can be viewed. The input in this step is daily food data, and the output is information presented in calendar format.

[0568] (Application example 1)

[0569] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0570] Conventional refrigerator food management systems have had issues with the effective use of ingredients and insufficient nutritional management. In addition, because they are unable to propose optimal delivery menus that take into account the ingredients in the refrigerator, users are likely to waste ingredients or eat meals that are nutritionally unbalanced.

[0571] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0572] In this invention, the server includes means for taking images of ingredients, means for transmitting the taken images to the server, means for the server to analyze the images and identify ingredients, means for generating a menu based on the identified ingredients, means for analyzing the nutritional value of the generated menu and graphing and quantifying it, means for presenting nutrients that are lacking based on the analysis results, means for displaying information on the menu and nutritional values ​​on a user terminal, and means for suggesting an optimal delivery menu via the user terminal. This allows users to make the most of the ingredients in their refrigerators and use delivery services while maintaining a nutritionally balanced diet.

[0573] "Means for taking images of ingredients" refers to a device or function that uses a camera to take pictures of ingredients in the refrigerator.

[0574] "Means for transmitting the captured images to a server" refers to a function or process for transmitting the captured images to a remote server via the Internet or other communication means.

[0575] "Means for the server to analyze the image and identify ingredients" refers to the function of the server analyzing the image received by the server using machine learning algorithms or image recognition technology to accurately identify ingredients in the image.

[0576] The "means for generating a menu based on the identified ingredients" refers to a function that automatically suggests appropriate recipes and menus based on the identified ingredient information.

[0577] "Means for analyzing, graphing, and quantifying the nutritional value of the generated menu" refers to a function that analyzes the nutritional information of the proposed menu, graphs the results for visual display, and provides them as specific numerical data.

[0578] "Means for suggesting nutrients that are lacking based on the analysis results" refers to a function that suggests nutrients that the user is lacking and additional ingredients that are needed based on the results of the nutritional value analysis of the menu.

[0579] "Means for displaying the menu and nutritional value information on the user terminal" refers to a function for displaying the proposed menu and its nutritional value information on the user's terminal such as a smartphone or tablet.

[0580] "Means for proposing the most suitable delivery menu via the user terminal" refers to a function that provides the most suitable delivery menu taking into consideration the ingredients in the user's refrigerator and their nutritional needs.

[0581] The present invention relates to a system that takes images of ingredients in a refrigerator, analyzes the images to generate optimal menus, and also suggests food delivery services. This system provides a means for users to effectively use ingredients in the refrigerator while achieving nutritionally balanced meals.

[0582] System Overview

[0583] The system is operated through a user device such as a smartphone or tablet. The user device has the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on them. The generated menu is then analyzed for nutritional value and visually displayed to the user. The system also suggests optimal food delivery menus based on the ingredients in the user's refrigerator and their nutritional needs.

[0584] Specific Embodiments of the System

[0585] Photograph and send ingredients

[0586] A user takes a photo of the food in the refrigerator using a device such as a smartphone. The image is then sent to a server via an application. At this time, the device compresses the image data and processes it to enable quick transmission.

[0587] Image analysis

[0588] The server analyzes the received image. It uses an image recognition algorithm to automatically identify ingredients. For example, ingredients such as chicken, tomatoes, and spices can be identified. Technologies such as YOLO (You Only Look Once) and TensorFlow are used for the analysis.

[0589] Menu generation

[0590] The server then searches the database for the most suitable recipe based on the identified ingredients. It generates the optimal menu, taking into account the user's nutritional needs and the expiration dates of the ingredients. For example, it might suggest a menu item such as "chicken steak with tomato salsa."

[0591] Nutritional analysis and presentation

[0592] The server analyzes the nutritional value of the generated menu. The analysis uses the nutritional information of each ingredient registered in the database. The amounts of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified. This nutritional information is then graphed and displayed visually to the user.

[0593] Suggestions for nutrient deficiencies

[0594] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For those nutrients, it suggests alternative foods or additional ingredients. For example, if you are deficient in vitamin D, it suggests eating mushrooms or fish.

[0595] Food delivery menu suggestions

[0596] The system proposes optimal food delivery menus via the user's device, taking into account the ingredients in the refrigerator and the user's nutritional needs, allowing users to easily select nutritionally balanced meals through the delivery service.

[0597] Nutritional Information

[0598] The user's device displays the menu and nutritional information received from the server. The user can practice a balanced diet based on the visually displayed graphs and list format information.

[0599] Specific examples

[0600] Let's say a user takes a photo of their refrigerator and it shows "chicken, tomatoes, and spices." The server analyzes it and identifies these ingredients. The server then suggests a meal such as "chicken steak with tomato salsa," and analyzes and visualizes its nutritional value (calories, protein, vitamin C, etc.). Furthermore, if the server determines that the user is lacking in vitamin D, it suggests adding mushrooms or fish. The user can check this information on their device and use it to plan their daily meals.

[0601] Prompt Sentence Examples

[0602] "You analyzed images of the inside of the refrigerator and identified chicken, tomatoes, and onions. Please use these to suggest the best delivery menu. Also, please show the nutritional value of the suggested menu."

[0603] The above is an embodiment of the present invention. This system allows users to effectively use ingredients and efficiently manage nutrition.

[0604] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0605] Step 1:

[0606] A user takes a picture of the food in the refrigerator with their smartphone. At this time, the device uses its camera function to capture the image and saves the image data in the application. The input is the image of the food in the refrigerator, and the output is the saved image data.

[0607] Step 2:

[0608] The image data captured by the device is compressed and sent to the server. This process reduces the size of the image data so that it can be sent quickly. The input is the saved image data, and the output is the compressed image data.

[0609] Step 3:

[0610] The server decodes the received compressed image data and applies image analysis algorithms to identify ingredients. The server uses YOLO or TensorFlow to identify each ingredient in the image. The input is the compressed image data, and the output is a list of identified ingredients.

[0611] Step 4:

[0612] The server searches the database for the best recipe based on the ingredient list and generates a menu. The generated menu takes into account the user's nutritional needs and the expiration dates of the ingredients. The input is the identified ingredient list, and the output is the generated menu.

[0613] Step 5:

[0614] The server analyzes the nutritional value of the generated menu, quantifies the amount of each nutrient, and visually graphs it. The input is the generated menu, and the output is the analyzed nutritional value data and its graphical representation.

[0615] Step 6:

[0616] The server compares daily nutritional intake with the analysis results, identifies nutrient deficiencies, and suggests additional ingredients. The input is the analyzed nutritional value data, and the output is the nutrient deficiencies and suggestions for supplementing them.

[0617] Step 7:

[0618] The server sends the generated menu, nutritional value data, and suggested information on nutrient deficiencies to the user terminal, which then displays them. The input is the nutritional value data, suggested nutrient deficiencies, and the generated menu, and the output is the information displayed on the user terminal.

[0619] Step 8:

[0620] The user terminal proposes an optimal food delivery menu that takes into account the ingredients in the refrigerator and the user's nutritional needs. The input is the list of ingredients in the refrigerator and the user's nutritional needs, and the output is the optimal food delivery menu.

[0621] The above are the specific processing steps of the system program that realizes this application example. This allows users to make effective use of ingredients, efficiently manage their nutrition, and use the optimal food delivery menu.

[0622] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0623] The present invention relates to a system that takes pictures of ingredients in a refrigerator, analyzes the pictures, and generates optimal menus for nutritional management. A feature of the present invention is that by combining it with an emotion engine that recognizes the user's emotions, it is possible to suggest more personalized meals.

[0624] System Overview

[0625] This system is primarily operated through devices such as smartphones and tablets. The devices have the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on them. It also has a function to analyze the user's emotional state using an emotion engine and adjust the menu and nutritional suggestions as described below.

[0626] Specific Embodiments of the System

[0627] 1. Take a photo of the ingredients and send it

[0628] A user takes a photo of the food in the refrigerator using a device such as a smartphone. The image is then sent to a server via an application. The device compresses the image data and processes it to ensure fast transmission.

[0629] 2. Image Analysis

[0630] The server analyzes the received image. An image recognition algorithm is used for the analysis, and ingredients are automatically identified. For example, ingredients such as chicken, tomatoes, and spices are identified. The algorithm for analyzing the image uses technologies such as YOLO (You Only Look Once) and TensorFlow.

[0631] 3. Menu generation

[0632] The server then searches the database for the most suitable recipe based on the identified ingredients. It generates the optimal menu, taking into account the user's nutritional needs and the expiration dates of the ingredients. For example, it might suggest a menu item such as "chicken steak with tomato salsa."

[0633] 4. Analysis and presentation of nutritional value

[0634] The server analyzes the nutritional value of the generated menu. The analysis uses the nutritional information of each ingredient registered in the database. The amounts of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified. This nutritional information is then graphed and displayed visually to the user.

[0635] 5. Suggestions for nutrient deficiencies

[0636] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For those nutrients, it suggests alternative foods or additional ingredients. For example, if you are deficient in vitamin D, it suggests eating mushrooms or fish.

[0637] 6. Menu and nutrition information display

[0638] The device receives the menu and nutritional information from the server and displays it to the user, who can then practice a balanced diet based on the visually displayed graphs and list-style information.

[0639] 7. Voice question function

[0640] Users can use voice input to ask questions or get more detailed information. For example, if they ask, "How do I cook this dish?", the server will analyze the voice data and provide the appropriate cooking method. The answer will be played back as text or audio.

[0641] 8. Calendar Management

[0642] The device has a function to manage daily menus and nutritional intake status in a calendar format. Users can view past data and future plans and manage their daily eating habits. This allows for long-term nutritional management.

[0643] 9. Incorporating an Emotional Engine

[0644] The server incorporates an emotion engine to recognize emotions from the user's facial expressions and voice. This emotional data is reflected in meal suggestions. For example, if the user is feeling stressed, the server can suggest ingredients and menus that have a relaxing effect.

[0645] 10. Emotional Adjustment

[0646] The server analyzes the accumulated emotional data and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake. For example, if a particular ingredient or menu improves the user's mood, personalized suggestions will be made based on that information.

[0647] Specific Examples

[0648] For example, suppose a user takes a photo of their refrigerator and sees that it contains "chicken, tomatoes, and spices." The server analyzes the photo and identifies these ingredients. The server then suggests a menu item such as "chicken steak with tomato salsa," and also considers the user's emotional state to suggest adding ingredients that have a relaxing effect. The server also analyzes and visualizes nutritional values ​​(calories, protein, vitamin C, etc.), suggesting adding mushrooms or fish to compensate for vitamin D deficiency. Users can check this information on their device and use it to plan their daily meals.

[0649] The above is an embodiment of the present invention. This system allows users to effectively use ingredients and efficiently manage nutrition, and also allows users to receive personalized suggestions based on their emotions.

[0650] The processing flow will be explained below.

[0651] Step 1:

[0652] When a user launches the smartphone application and takes a photo of the ingredients in the refrigerator, the device activates its camera function and displays a screen on which the user can take a photo of the ingredients.

[0653] Step 2:

[0654] The user checks the photo they have taken and presses the "Send" button in the application. The device sends the photo data to the server. When sending, the device compresses the image data to ensure it can be sent quickly.

[0655] Step 3:

[0656] The server receives the transmitted photo and runs an image recognition algorithm to analyze the received image data.

[0657] Step 4:

[0658] The server uses image recognition algorithms to identify ingredients in the received image, and lists the ingredients identified, for example, "chicken, tomatoes, and spices."

[0659] Step 5:

[0660] The server searches the database for the best recipe based on the identified ingredients, taking into account the user's nutritional needs (e.g., calorie restriction, vitamin intake, etc.) and expiration dates of the ingredients.

[0661] Step 6:

[0662] The server generates an optimal menu. For example, it might suggest "chicken steak with tomato salsa." Details of the suggested menu are stored in a database.

[0663] Step 7:

[0664] The server analyzes the nutritional value of the generated menu. The nutritional information includes calories, protein, fat, vitamins, minerals, etc. This information is obtained from a database.

[0665] Step 8:

[0666] The server quantifies the amount of each nutrient and calculates the total amount, then graphs the calculated nutritional information to generate data that can be displayed visually.

[0667] Step 9:

[0668] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For example, if there is a vitamin D deficiency, it will recommend additional foods and ingredients to supplement it.

[0669] Step 10:

[0670] The server transmits information including incomplete nutritional suggestions to the terminal, including menu details and nutritional value information.

[0671] Step 11:

[0672] The terminal displays the received information to the user in graph and list format so that the user can easily understand it.

[0673] Step 12:

[0674] Users can ask questions or get more information by speaking, for example, "How do I prepare this dish?"

[0675] Step 13:

[0676] The device converts the voice input into text and sends the text data to the server.

[0677] Step 14:

[0678] The server analyzes the question and generates an appropriate answer, such as "How to cook chicken steak..." and provides specific instructions.

[0679] Step 15:

[0680] The server generates a response and sends it to the terminal, which displays or plays it aloud to the user.

[0681] Step 16:

[0682] The device manages daily menus and nutritional intake in a calendar format, allowing users to check the calendar and view past data and upcoming plans.

[0683] Step 17:

[0684] The device activates an emotion engine that recognizes emotions from the user's facial expressions and voice. The user's emotional state is collected through the application. For example, facial expression analysis can determine whether the user is smiling, and voice analysis can determine whether the user is under stress.

[0685] Step 18:

[0686] The server analyzes the collected emotional data using an emotion engine. Based on the analysis results, it adjusts menu and nutrition suggestions based on the user's emotions. For example, if the user is feeling stressed, it will suggest ingredients and menus that have a relaxing effect.

[0687] Step 19:

[0688] The server analyzes the accumulated emotional data over the long term and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake. For example, if a particular ingredient or menu improves the user's mood, personalized suggestions will be made based on that information.

[0689] In this way, by performing specific actions at each step, the system efficiently supports food inventory management and nutritional management.In addition, personalized suggestions that take into account the user's emotional state can provide a more satisfying service.

[0690] Example 2

[0691] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0692] Conventional menu suggestion systems have difficulty effectively utilizing ingredients in the refrigerator and providing balanced nutritional management. Furthermore, they are unable to provide personalized suggestions that take the user's emotional state into account, resulting in low satisfaction. Furthermore, they lack the ability to provide appropriate answers to voice questions from users and to manage past nutritional intake history. By resolving these issues, it is necessary to improve the quality and satisfaction of users' dietary habits.

[0693] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for a user to take an image of ingredients; means for compressing the taken image at a terminal and sending it to the server; means for the server to analyze the received image and identify ingredients; means for generating a menu based on the identified ingredients, taking into consideration the user's nutritional needs and the expiration dates of the ingredients; means for analyzing the nutritional value of the generated menu and quantifying and graphing calorie and nutrient values; means for identifying missing nutrients based on the analysis results and suggesting substitute foods and additional ingredients; means for displaying information about the menu and nutritional value on a user terminal; means for the server to provide appropriate information in response to questions from the user via voice input, and for the user terminal to provide answers in text or voice; means for managing daily menus and nutritional intake status in a calendar format and providing the user with past and future information; means for analyzing emotions from the user's facial expressions and voice, and personalizing meal suggestions using emotional data; and means for suggesting ingredients and menus with a relaxing effect according to a specific emotional state, and optimizing individual meal suggestions by analyzing the relationship between past emotions and nutritional intake. This will allow for effective use of ingredients in the refrigerator, optimizing the user's nutritional management, and enabling personalized meal suggestions based on the user's emotional state.

[0694] "User" refers to a person who uses the system to take pictures of ingredients and receive menu suggestions and nutritional management.

[0695] "Terminal" refers to an electronic device such as a smartphone or tablet that takes and sends images of ingredients, communicates with the server, and displays information.

[0696] The "server" refers to a remote data processing device that analyzes images of ingredients, generates menus, analyzes nutritional values, suggests nutrient deficiencies, and so on.

[0697] "Image analysis" refers to the technical process of analyzing received image data of ingredients to automatically identify the ingredients.

[0698] "Ingredient identification" refers to the process of identifying and listing types of ingredients through image analysis.

[0699] "Menu generation" refers to the process of determining the optimal menu based on the identified ingredients, taking into account the user's nutritional needs and the expiration dates of the ingredients.

[0700] "Nutritional analysis" refers to the process of calculating the nutritional information of the generated menu and quantifying and graphing the calories and amount of each nutrient.

[0701] "Nutrient deficiency suggestion" refers to the process of identifying missing nutrients based on analyzed nutritional value data and suggesting appropriate substitute foods or additional ingredients.

[0702] "Voice input" refers to a means by which a user uses voice to ask questions or give instructions to a system.

[0703] "Emotion analysis" refers to the process of recognizing emotions from a user's facial expressions and voice and analyzing that data.

[0704] "Personalization" refers to providing information and services that are individually optimized based on a user's specific conditions and tendencies.

[0705] "Calendar management" refers to the process of recording daily menus and nutritional intake status in calendar format and providing users with past data and future schedules.

[0706] The present invention relates to a system that takes images of ingredients in a refrigerator, analyzes the images, creates optimal menus, and manages nutrition. Furthermore, the present invention can improve user satisfaction and health management by recognizing the user's emotions and providing personalized meal suggestions based on those emotions.

[0707] System Overview

[0708] This system is primarily operated through devices such as smartphones and tablets. The devices have the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on that information. It also has a function to analyze the user's emotional state using an emotion engine and adjust the menu and nutritional suggestions accordingly.

[0709] Hardware and software used

[0710] This system uses the following hardware and software:

[0711] Devices: smartphones, tablets

[0712] Server: High-performance data processing server

[0713] Image analysis algorithm: YOLO (You Only Look Once), TensorFlow

[0714] Emotion analysis engine: speech recognition software, facial recognition software

[0715] System operation flow

[0716] The system works as follows:

[0717] 1. Take a photo of the ingredients and send it

[0718] The user uses the camera on their smartphone or tablet to take a photo of the food in their refrigerator.

[0719] The device compresses the captured images and quickly transmits the data to the server.

[0720] 2. Image Analysis

[0721] The server decodes the received image and converts it into an analyzable format.

[0722] The server uses image recognition algorithms such as YOLO and TensorFlow to automatically identify ingredients in the image.

[0723] For example, ingredients such as chicken, tomatoes, and spices are identified.

[0724] 3. Menu generation

[0725] The server searches the database for the most suitable recipe based on the identified ingredient list.

[0726] The server generates optimal menus taking into account the user's nutritional needs and the expiration dates of ingredients.

[0727] For example, a menu suggestion might be "chicken steak with tomato salsa."

[0728] 4. Analysis and presentation of nutritional value

[0729] The server analyzes the nutritional value of the generated menu, using the nutritional information of each ingredient registered in the database.

[0730] The amount of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified.

[0731] This nutritional information is graphed and displayed visually to the user.

[0732] 5. Suggestions for nutrient deficiencies

[0733] Based on the analysis results, the server compares the nutrients needed per day with the nutritional value of the generated menu and identifies any nutrients that are lacking.

[0734] For those who are lacking in nutrients, we suggest alternative foods or additional ingredients. For example, if you are lacking in vitamin D, we suggest eating mushrooms or fish.

[0735] 6. Menu and nutrition information display

[0736] The terminal displays the menu and nutritional information received from the server to the user.

[0737] Users can practice balanced eating habits based on visually displayed graphs and list-style information.

[0738] 7. Voice question function

[0739] Users can use voice input to ask questions or for additional information.

[0740] For example, if you ask, "How do you cook this dish?", the server will analyze the voice data and provide the appropriate cooking method as an answer. The answer will be played back in text or audio.

[0741] 8. Calendar Management

[0742] The device has the function of managing daily menus and nutritional intake in calendar format.

[0743] Users can view past data and future plans and manage their daily eating habits.

[0744] 9. Incorporating an Emotional Engine

[0745] The server incorporates an emotion engine to recognize emotions from the user's facial expressions and voice.

[0746] Emotional data is reflected in meal suggestions. For example, if a user is feeling stressed, the system can suggest ingredients and menu items that have a relaxing effect.

[0747] 10. Emotional Adjustment

[0748] The server analyzes the accumulated emotional data and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake.

[0749] For example, if a particular ingredient or menu item improves a user's mood, personalized suggestions can be made based on that information.

[0750] Specific examples

[0751] For example, suppose a user takes a photo of their refrigerator, which includes "chicken, tomatoes, and spices." The device compresses this and sends it to the server. The server analyzes the photo and identifies these ingredients. The server then suggests a menu item such as "chicken steak with tomato salsa," and also considers the user's emotional state and suggests adding ingredients that have a relaxing effect. The server also analyzes and visualizes nutritional values ​​(calories, protein, vitamin C, etc.), suggesting adding mushrooms or fish to compensate for vitamin D deficiency. Users can check this information on their device and use it to plan their daily meals.

[0752] This invention allows users to make effective use of ingredients and efficiently manage their nutrition, and also allows users to receive personalized suggestions based on their emotions.

[0753] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0754] Step 1: Photograph the ingredients and send the image

[0755] The user uses a device (smartphone or tablet) to take a photo of the food in the refrigerator.

[0756] Input: Images of ingredients in the refrigerator.

[0757] The device compresses the images it takes, reducing the amount of data it takes to send them over the network.

[0758] The terminal transmits the compressed image data to the server.

[0759] Output: Compressed image data sent to the server.

[0760] Step 2: Image analysis and ingredient identification

[0761] The server decodes the received image data and converts it into an analyzable format.

[0762] Input: Received compressed image data.

[0763] The server uses image recognition algorithms such as YOLO (You Only Look Once) and TensorFlow to automatically identify ingredients.

[0764] Image analysis algorithms identify the type of ingredient (e.g., chicken, tomato, spices, etc.) from the image.

[0765] Output: A list of identified ingredients.

[0766] Step 3: Create a menu

[0767] The server searches the registered database for the most suitable recipe based on the identified ingredient list.

[0768] Input: Identified ingredient list, recipe information in database.

[0769] The server generates optimal menus taking into account the user's nutritional needs and the expiration dates of ingredients.

[0770] For example, a menu such as "chicken steak with tomato salsa" may be generated.

[0771] Output: The generated menu information.

[0772] Step 4: Nutritional analysis and presentation

[0773] The server analyzes the nutritional value of the generated menu.

[0774] Input: Generated menu information, nutrition information in the database.

[0775] The amount of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified.

[0776] This nutritional information is graphed and formatted in a way that can be visually presented to the user.

[0777] Output: Nutritional values ​​are displayed in graph and list format.

[0778] Step 5: Suggesting nutrient deficiencies

[0779] The server analyzes the nutritional value of the menu and identifies any nutrients that are lacking by comparing them with the daily required nutrients.

[0780] Input: Quantified nutritional value data, reference values ​​for nutrients needed per day.

[0781] The server will identify any nutrient deficiencies and suggest substitutions or additional ingredients.

[0782] For example, if you are deficient in vitamin D, it is suggested that you "consume additional mushrooms and fish."

[0783] Output: Suggested information on nutrient deficiencies.

[0784] Step 6: Display menu and nutrition information

[0785] The server sends the generated menu and nutritional information to the terminal.

[0786] Input: Menu information, nutritional information.

[0787] The information received by the terminal is visually displayed to the user.

[0788] The device displays the information in graph and list format, and users can use the information to practice a balanced diet.

[0789] Output: The visual information that is displayed to the user.

[0790] Step 7: Voice Question Function

[0791] Users use voice input to ask questions or for additional information.

[0792] Input: Audio data.

[0793] The device converts the voice data into text and sends it to the server.

[0794] The server analyzes the question and generates an appropriate answer.

[0795] The device will present the answer to the user in text or audio.

[0796] For example, if you ask, "How do you cook this dish?", the cooking instructions will be answered in text or voice.

[0797] Output: The answer information provided to the user.

[0798] Step 8: Calendar Management

[0799] The device manages daily menus and nutritional intake in calendar format.

[0800] Input: Menu information, nutritional intake data.

[0801] Users can view past data and future schedules.

[0802] The terminal displays information in calendar format, enabling long-term nutritional management.

[0803] Output: Information displayed in a calendar format.

[0804] Step 9: Incorporating the Emotion Engine

[0805] The server uses an emotion engine to recognize emotions from the user's facial expressions and voice data.

[0806] Input: User's facial expression data, voice data.

[0807] A sentiment analysis engine analyzes this data and infers the user's emotional state (e.g., stress, happiness, etc.).

[0808] Output: Emotional state data.

[0809] Step 10: Emotional Adjustment

[0810] The server analyzes the accumulated emotional data and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake.

[0811] Input: Emotional state data, nutritional intake data.

[0812] Depending on your specific emotional state, it will suggest foods and menus that have a relaxing effect.

[0813] For example, if a user is feeling stressed, a menu containing ingredients that have a relaxing effect will be suggested based on that information.

[0814] Output: Personalized meal suggestion information.

[0815] (Application example 2)

[0816] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0817] Conventional food management systems have limitations in efficiently managing ingredients in the refrigerator and suggesting appropriate menus. Furthermore, they make uniform suggestions without considering the user's emotions, resulting in insufficient personalization to meet individual needs. Furthermore, there has been a lack of systems that can utilize new devices such as smart glasses to recognize ingredients and emotions in real time.

[0818] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for capturing images of ingredients, a means for transmitting the captured images to the server, a means for the server to analyze the images and identify ingredients, a means for generating a menu based on the identified ingredients, a means for analyzing, graphing, and quantifying the nutritional value of the generated menu, a means for presenting missing nutrients based on the analysis results, a means for displaying the menu and nutritional value information on a user terminal, a means for recognizing the user's emotions and reflecting them in meal suggestions, and a means for using smart glasses to recognize ingredients and emotions in real time and suggest a menu. This enables efficient recognition of ingredients in the user's refrigerator and personalized menu suggestions based on the user's individual emotional state. Furthermore, the use of smart glasses enables real-time ingredient recognition and emotion recognition, enabling more immediate and accurate information provision.

[0819] "Means for taking images of ingredients" refers to functionality that includes devices and software for capturing images of ingredients in a refrigerator or other storage location.

[0820] The "means for transmitting the captured image to the server" is a function for transmitting the image captured from the smart glasses or other terminal to the server via data communication.

[0821] The "means for the server to analyze the image and identify ingredients" refers to a system that uses algorithms or techniques to analyze received image data and identify ingredients.

[0822] The "means for generating a menu based on the identified ingredients" is a function that searches a database for the most suitable recipe based on the identified ingredients and suggests a menu.

[0823] The "means for analyzing, graphing and quantifying the nutritional value of the generated menu" is a function for quantifying each nutrient in the generated menu and visually displaying it.

[0824] The "means for presenting missing nutrients based on the analysis results" is a function for identifying missing nutrients by comparing them with the calculated nutritional values ​​and suggesting appropriate ingredients and supplements.

[0825] The "means for displaying the menu and nutritional value information on a user terminal" is a function for displaying the generated menu and its nutritional value information on a user terminal such as smart glasses or a smartphone.

[0826] "Means for recognizing the user's emotions and reflecting them in meal suggestions" is a function that analyzes the user's facial expressions and voice to identify their emotional state and then makes personalized meal suggestions based on that information.

[0827] "Means for using smart glasses to recognize ingredients and emotions in real time and suggest menus" is a function that uses the cameras and sensors in smart glasses to recognize ingredients and the user's emotions in real time and suggests menus based on that information.

[0828] The present invention provides a system for efficiently managing ingredients in a refrigerator and proposing personalized menus based on the user's emotions. Here, specific embodiments of the present invention will be described.

[0829] System Overview

[0830] The main components of this system are smart glasses (or a user terminal such as a smartphone), a server, and software for recognizing images of ingredients in the refrigerator and the user's emotions.

[0831] 1. Food and emotion recognition

[0832] A user uses smart glasses to check the ingredients in the refrigerator. At this time, the camera in the smart glasses captures an image of the ingredients and sends the image data to the server. At the same time, the sensors in the smart glasses capture the user's facial expressions and voice to recognize their emotions. For example, if a user opens the refrigerator and says, "I want to relax today," that voice data is also sent to the server.

[0833] 2. Image analysis and emotion analysis on the server

[0834] The server first analyzes the received image data, using the YOLO (You Only Look Once) model and TensorFlow algorithms to automatically identify the ingredients in the refrigerator.

[0835] The received facial and voice data is then analyzed using an emotion engine, which is used to identify the user's current emotional state (e.g., relaxed, stressed, depressed, etc.).

[0836] 3. Menu Creation and Nutritional Analysis

[0837] Based on the identified ingredients and the user's emotional state, the server searches for the most suitable recipe from the database and generates a menu. For example, if the ingredients are "chicken, tomato, and spices" and the user is recognized as "feeling stressed," the server will suggest "chicken steak with tomato salsa" with "herbal tea," which has a relaxing effect, as a side dish.

[0838] To analyze the nutritional value of the generated menu, the system refers to a nutrition database and converts information such as total calories, protein, fat, vitamins, and minerals into numerical values ​​and graphs, which the user can visually check.

[0839] 4. Display on the user's device

[0840] The generated menu information and nutritional information are displayed on the user's device, such as smart glasses or a smartphone, allowing the user to efficiently use the ingredients in their refrigerator and practice nutritionally balanced meals.

[0841] 5. Examples of prompts

[0842] Specific examples of prompts include:

[0843] "It recognizes the ingredients in the refrigerator and suggests personalized meals based on the user's emotions. For example, if it analyzes an image containing "chicken, tomatoes, and spices" and determines that the user is feeling stressed, it will suggest a meal that includes ingredients that have a relaxing effect."

[0844] This system allows users to make the most of the ingredients in their refrigerator and receive personalized meal suggestions based on their emotions. It also recognizes ingredients and emotions in real time and provides appropriate information instantly.

[0845] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0846] Step 1:

[0847] A user uses smart glasses to check the ingredients in the refrigerator and captures the images with the camera. The input is the image of the ingredients in the refrigerator, and the output is the captured image data.

[0848] Step 2:

[0849] The smart glasses transmit the captured image data to a server. The input is the captured image data, and the output is the transmission of the image data to the server. In actual operation, the smart glasses use a communication module to compress the image data and transmit it to the server via the Internet.

[0850] Step 3:

[0851] The server analyzes the received image data and identifies the ingredients. The input is the image data sent to the server, and the output is a list of identified ingredients. Specifically, the server uses the YOLO model to detect ingredients in the image and create a list of ingredient names.

[0852] Step 4:

[0853] The smart glasses capture the user's facial expressions and voice and send the data to the server. The input is the user's facial expressions and voice data, and the output is data transmission to the server. The smart glasses' sensors and microphone are used to capture the user's facial expressions and voice in real time and send the data to the server.

[0854] Step 5:

[0855] The server analyzes the received facial and voice data to recognize the user's emotions. The input is the facial and voice data sent to the server, and the output is the recognized emotion data. Specifically, the emotion engine is used to analyze the facial and voice data and identify the user's emotional state.

[0856] Step 6:

[0857] The server generates an optimal menu based on the identified ingredients and the recognized emotion data. The input is the identified ingredient list and emotion data, and the output is the generated menu. The server searches for appropriate recipes from the database and selects meals that suit the user's emotions.

[0858] Step 7:

[0859] The server analyzes the nutritional value of the generated menu and graphs and quantifies it. The input is the generated menu, and the output is the nutritional value and graph. The server refers to a nutrition database, tallying the nutritional information of each ingredient and calculating the nutritional value.

[0860] Step 8:

[0861] The server then presents the nutritional deficiencies based on the analysis results. The input is the nutritional analysis results, and the output is suggestions regarding the nutrients that are lacking. It identifies the nutrients that are lacking and suggests additional ingredients or supplements to make up for them.

[0862] Step 9:

[0863] The generated menu and nutritional information are displayed on the user's device. The input is the generated menu and nutritional information, and the output is the display on the user's device. The menu and nutritional information are displayed on smart glasses or a smartphone, allowing the user to visually confirm them.

[0864] Step 10:

[0865] The user asks a question by voice input, and the server provides the corresponding information. The input is the user's voice question, and the output is the answer from the server. The server uses a voice recognition system to analyze the user's question and provide appropriate information and explanations.

[0866] Step 11:

[0867] The user terminal manages daily menus and nutritional intake status in a calendar format, providing users with past and future information. The input is past and current menu data, and the output is a visual display in calendar format. Users can manage their nutrition over the long term.

[0868] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0869] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0870] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0871] [Third embodiment]

[0872] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0873] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0874] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0875] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0876] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0877] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0878] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0879] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0880] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0881] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0882] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0883] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0884] The present invention relates to a system for nutritional management that takes images of ingredients in a refrigerator, analyzes the images, and generates optimal menus. This system provides users with a means for efficiently managing their daily nutrition and making effective use of ingredients.

[0885] System Overview

[0886] This system is operated through devices such as smartphones and tablets. The devices have the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on that information. The nutritional value of the generated menu is analyzed and visually displayed to the user.

[0887] Specific Embodiments of the System

[0888] 1. Take a photo of the ingredients and send it

[0889] A user takes a photo of the food in the refrigerator using a device such as a smartphone. The image is then sent to a server via an application. At this time, the device compresses the image data and processes it to enable quick transmission.

[0890] 2. Image Analysis

[0891] The server analyzes the received image. An image recognition algorithm is used for the analysis, and ingredients are automatically identified. For example, ingredients such as chicken, tomatoes, and spices are identified. The algorithm for analyzing the image uses technologies such as YOLO (You Only Look Once) and TensorFlow.

[0892] 3. Menu generation

[0893] The server then searches the database for the most suitable recipe based on the identified ingredients. It generates the optimal menu, taking into account the user's nutritional needs and the expiration dates of the ingredients. For example, it might suggest a menu item such as "chicken steak with tomato salsa."

[0894] 4. Analysis and presentation of nutritional value

[0895] The server analyzes the nutritional value of the generated menu. The analysis uses the nutritional information of each ingredient registered in the database. The amounts of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified. This nutritional information is then graphed and displayed visually to the user.

[0896] 5. Suggestions for nutrient deficiencies

[0897] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For those nutrients, it suggests alternative foods or additional ingredients. For example, if you are deficient in vitamin D, it suggests eating mushrooms or fish.

[0898] 6. Menu and nutrition information display

[0899] The device receives the menu and nutritional information from the server and displays it to the user, who can then practice a balanced diet based on the visually displayed graphs and list-style information.

[0900] 7. Voice question function

[0901] Users can use voice input to ask questions or get more detailed information. For example, if they ask, "How do I cook this dish?", the server will analyze the voice data and provide the appropriate cooking method. The answer will be played back as text or audio.

[0902] 8. Calendar Management

[0903] The device has a function to manage daily menus and nutritional intake status in a calendar format. Users can view past data and future plans and manage their daily eating habits. This allows for long-term nutritional management.

[0904] Specific Examples

[0905] For example, suppose a user takes a photo of their refrigerator and sees that it contains "chicken, tomatoes, and spices." The server analyzes the photo and identifies these ingredients. The server then suggests a meal such as "chicken steak with tomato salsa," and analyzes and visualizes its nutritional value (calories, protein, vitamin C, etc.). Furthermore, if the server determines that the user is lacking in vitamin D, it suggests adding mushrooms or fish. Users can check this information on their device and use it to plan their daily meals.

[0906] The above is an embodiment of the present invention. This system allows users to effectively use ingredients and efficiently manage nutrition.

[0907] The processing flow will be explained below.

[0908] Step 1:

[0909] When a user launches the smartphone application and takes a photo of the ingredients in the refrigerator, the device activates its camera function and displays a screen on which the user can take a photo of the ingredients.

[0910] Step 2:

[0911] The user checks the photo they have taken and presses the "Send" button in the application. The device sends the photo data to the server.

[0912] Step 3:

[0913] The server receives the transmitted photo and runs an image recognition algorithm to analyze the received image data.

[0914] Step 4:

[0915] The server uses image recognition algorithms to identify ingredients in the received image, and lists the ingredients identified, for example, "chicken, tomatoes, and spices."

[0916] Step 5:

[0917] The server searches the database for the best recipe based on the identified ingredients, taking into account the user's nutritional needs (e.g., calorie restriction, vitamin intake, etc.) and expiration dates of the ingredients.

[0918] Step 6:

[0919] The server generates an optimal menu. For example, it might suggest "chicken steak with tomato salsa." Details of the suggested menu are stored in a database.

[0920] Step 7:

[0921] The server analyzes the nutritional value of the generated menu. The nutritional information includes calories, protein, fat, vitamins, minerals, etc. This information is obtained from a database.

[0922] Step 8:

[0923] The server quantifies the amount of each nutrient and calculates the total amount, then graphs the calculated nutritional information to generate data that can be displayed visually.

[0924] Step 9:

[0925] The server compares the analysis results with the daily required nutrients and identifies any deficiencies. For example, if there is a vitamin D deficiency, it will recommend additional ingredients or foods to supplement it.

[0926] Step 10:

[0927] The server transmits information including incomplete nutritional suggestions to the terminal, including menu details and nutritional value information.

[0928] Step 11:

[0929] The terminal displays the received information to the user in graph and list format so that the user can easily understand it.

[0930] Step 12:

[0931] Users can ask questions or get more information by speaking, for example, "How do I prepare this dish?"

[0932] Step 13:

[0933] The device converts the voice input into text and sends the text data to the server.

[0934] Step 14:

[0935] The server analyzes the question and generates an appropriate answer, such as "How to cook chicken steak..." and provides specific instructions.

[0936] Step 15:

[0937] The server generates a response and sends it to the terminal, which displays or plays it aloud to the user.

[0938] Step 16:

[0939] The device manages daily menus and nutritional intake in a calendar format, allowing users to check the calendar and view past data and upcoming plans.

[0940] In this way, by performing specific actions at each step, the system efficiently supports food inventory management and nutritional management.

[0941] Example 1

[0942] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0943] Managing ingredients and planning nutritionally balanced meals is time-consuming and laborious, making it difficult for many people. It's particularly challenging to effectively utilize ingredients in the refrigerator and manage daily nutritional intake. Furthermore, it's difficult to instantly obtain specific cooking instructions and nutritional information for a menu. A system that can solve these problems is needed.

[0944] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0945] In this invention, the server includes means for analyzing and identifying images of ingredients, means for generating menus based on the identified ingredients, and means for analyzing and presenting the nutritional values ​​of the generated menus, thereby enabling users to efficiently use ingredients in their refrigerators and easily create balanced meal plans.

[0946] "Means for taking pictures of ingredients" is a function for taking pictures of ingredients in the refrigerator using a camera on a smartphone, tablet, or other device.

[0947] The "means for transmitting the captured image to the information processing device" is a function for appropriately compressing the captured image and transmitting it to an information processing device on a server or cloud.

[0948] "Means for the information processing device to analyze the image and identify ingredients" refers to a function that enables an information processing device located on a server or cloud to detect and identify ingredients in an image using image recognition algorithms such as YOLO or TensorFlow.

[0949] The "means for generating a menu based on the identified ingredients" is a function for automatically suggesting a menu by selecting the optimal recipe based on the identified ingredient information, taking into consideration the user's nutritional needs and the expiration date of the ingredients.

[0950] The "means for analyzing, graphing, and quantifying the nutritional value of the generated menu" is a function that uses the nutritional information of each ingredient registered in the database to analyze the total calories, protein, lipids, vitamins, minerals, etc. of the generated menu and visually display this information.

[0951] The "means for suggesting missing nutrients based on the analysis results" is a function that compares the results of the nutritional analysis of the menu with the amount of nutrients needed per day, identifies missing nutrients, and suggests ingredients or alternative foods to replenish them.

[0952] The "means for displaying the menu and nutritional value information on a user device" is a function for visually displaying the generated menu and its nutritional information in graph or list format on a device such as a smartphone or tablet used by the user.

[0953] "Means for managing daily menus and nutritional intake status in calendar format and providing users with past and future information" is a function that displays the user's daily food records and nutritional intake status in calendar format, allowing them to refer to past data and future plans.

[0954] "Means for an information processing device to provide information corresponding to a question from a user via voice input" is a function that enables an information processing device on a server or cloud to analyze the content of a question when the user asks it via voice input and provide an appropriate answer in voice or text.

[0955] The "means for compressing images of ingredients and quickly transmitting them to an information processing device" is a function for appropriately compressing large amounts of ingredient images, improving communication speed and data transmission efficiency, and transmitting them to a server or cloud.

[0956] The present invention is a system for nutritional management that takes images of ingredients in a refrigerator, analyzes the images, and generates optimal menus. This system provides users with a means for efficiently managing their daily nutrition and making effective use of ingredients. Specific embodiments of the system are described below.

[0957] System hardware and software configuration

[0958] This system is operated using a device such as a smartphone or tablet. The device has the function of taking photos of ingredients and sending them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on that information. The nutritional value of the generated menu is analyzed and visually displayed to the user.

[0959] The device must be equipped with a high-resolution camera and internet connection. The server must have a powerful computer and a large database. Image recognition algorithms such as YOLO (You Only Look Once) and TensorFlow are used for image recognition.

[0960] Software used

[0961] Image recognition algorithms: YOLO, TensorFlow

[0962] Database management systems: MySQL, PostgreSQL

[0963] Cloud platforms: AWS (Amazon Web Services), Google Cloud Platform

[0964] Specific examples

[0965] For example, a user can take a photo of "chicken, tomatoes, and spices" in the refrigerator. The device compresses the image and quickly sends it to the server. The server analyzes the received image and identifies these ingredients. The server then uses its database to search for the most suitable recipe and suggests a menu item such as "chicken steak with tomato salsa." The server also analyzes and visualizes the nutritional value of this menu item (total calories, protein, vitamin C, etc.). Furthermore, if it is determined that the user is lacking in vitamin D, the server will suggest adding mushrooms or fish. Users can check this information on their device and use it to plan their daily meals.

[0966] Prompt Sentence Examples

[0967] You can use prompts like the following for your generative AI model:

[0968] A user takes a picture of the ingredients in their refrigerator and sends it to the server. This image shows chicken, tomatoes, and spices. Suggest a meal plan using these ingredients, analyze their nutritional value, and suggest additional ingredients if certain nutrients are lacking.

[0969] As described above, this system allows users to efficiently utilize ingredients and manage their nutrition. Users can also easily ask questions via voice input and instantly obtain specific cooking methods and additional nutritional information. This makes it easier to plan and follow balanced meals.

[0970] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0971] Step 1:

[0972] The user takes a photo of the ingredients. The user uses the camera on their smartphone or tablet to take a photo of the ingredients in the refrigerator. For example, they take a photo of chicken, tomatoes, spices, etc. stored in the refrigerator. In this case, the input data is the image of the ingredients, and the output data is the image file stored in the device.

[0973] Step 2:

[0974] The device preprocesses the captured image. Specifically, it compresses the image data to reduce the file size and make it easier to send to the server. In this process, the input is a high-resolution food image, and the output is compressed image data. For example, the image is compressed using the JPEG format.

[0975] Step 3:

[0976] The terminal sends the preprocessed image data to the server. The compressed image data is uploaded to the server via the Internet. The input in this step is the compressed image data, and the output is the image data stored on the server.

[0977] Step 4:

[0978] The server performs image analysis. It uses image recognition algorithms such as YOLO and TensorFlow to analyze the uploaded image. The input in this step is the image data stored on the server, and the output is a list of identified ingredients. For example, chicken, tomatoes, spices, etc. in the image are automatically identified.

[0979] Step 5:

[0980] The server generates a menu based on the identified ingredient information. It searches recipe information stored in a database and proposes an optimal menu, taking into account the user's nutritional needs and the expiration dates of ingredients. The input in this step is the identified ingredient list, and the output is the generated menu information. For example, a specific menu such as "chicken steak with tomato salsa" is proposed.

[0981] Step 6:

[0982] The server analyzes the nutritional value of the generated menu. Using the nutritional information of each ingredient registered in the database, it quantifies and visualizes nutrients such as total calories, protein, fat, vitamins, and minerals. The input in this step is the generated menu information, and the output is numerical information on nutritional value and graphed data.

[0983] Step 7:

[0984] The server identifies any nutrient deficiencies based on the analysis results and suggests alternative foods or additional ingredients. For example, if it determines that a person is deficient in vitamin D, it will suggest adding mushrooms or fish. The input in this step is numerical information on nutritional value, and the output is suggested information on deficient nutrients.

[0985] Step 8:

[0986] The device displays the generated menu and nutritional information to the user. Specifically, the information is displayed in a visually easy-to-understand graph or list format. The input in this step is the menu information and nutritional information, and the output is the visualized data displayed on the user device.

[0987] Step 9:

[0988] The user uses the voice question function. They can ask questions about cooking methods and additional nutritional information by voice input. For example, if they ask, "How do I cook this dish?", the server analyzes it and responds by voice or text. The input in this step is the voice question data, and the output is the response data from the server.

[0989] Step 10:

[0990] The device manages daily menus and nutritional intake status in a calendar format. Daily food records and nutritional intake status are compiled in a calendar format, and past data and future plans can be viewed. The input in this step is daily food data, and the output is information presented in calendar format.

[0991] (Application example 1)

[0992] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0993] Conventional refrigerator food management systems have had issues with the effective use of ingredients and insufficient nutritional management. In addition, because they are unable to propose optimal delivery menus that take into account the ingredients in the refrigerator, users are likely to waste ingredients or eat meals that are nutritionally unbalanced.

[0994] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0995] In this invention, the server includes means for taking images of ingredients, means for transmitting the taken images to the server, means for the server to analyze the images and identify ingredients, means for generating a menu based on the identified ingredients, means for analyzing the nutritional value of the generated menu and graphing and quantifying it, means for presenting nutrients that are lacking based on the analysis results, means for displaying information on the menu and nutritional values ​​on a user terminal, and means for suggesting an optimal delivery menu via the user terminal. This allows users to make the most of the ingredients in their refrigerators and use delivery services while maintaining a nutritionally balanced diet.

[0996] "Means for taking images of ingredients" refers to a device or function that uses a camera to take pictures of ingredients in the refrigerator.

[0997] "Means for transmitting the captured images to a server" refers to a function or process for transmitting the captured images to a remote server via the Internet or other communication means.

[0998] "Means for the server to analyze the image and identify ingredients" refers to the function of the server analyzing the image received by the server using machine learning algorithms or image recognition technology to accurately identify ingredients in the image.

[0999] The "means for generating a menu based on the identified ingredients" refers to a function that automatically suggests appropriate recipes and menus based on the identified ingredient information.

[1000] "Means for analyzing, graphing, and quantifying the nutritional value of the generated menu" refers to a function that analyzes the nutritional information of the proposed menu, graphs the results for visual display, and provides them as specific numerical data.

[1001] "Means for suggesting nutrients that are lacking based on the analysis results" refers to a function that suggests nutrients that the user is lacking and additional ingredients that are needed based on the results of the nutritional value analysis of the menu.

[1002] "Means for displaying the menu and nutritional value information on the user terminal" refers to a function for displaying the proposed menu and its nutritional value information on the user's terminal such as a smartphone or tablet.

[1003] "Means for proposing the most suitable delivery menu via the user terminal" refers to a function that provides the most suitable delivery menu taking into consideration the ingredients in the user's refrigerator and their nutritional needs.

[1004] The present invention relates to a system that takes images of ingredients in a refrigerator, analyzes the images to generate optimal menus, and also suggests food delivery services. This system provides a means for users to effectively use ingredients in the refrigerator while achieving nutritionally balanced meals.

[1005] System Overview

[1006] The system is operated through a user device such as a smartphone or tablet. The user device has the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on them. The generated menu is then analyzed for nutritional value and visually displayed to the user. The system also suggests optimal food delivery menus based on the ingredients in the user's refrigerator and their nutritional needs.

[1007] Specific Embodiments of the System

[1008] Photograph and send ingredients

[1009] A user takes a photo of the food in the refrigerator using a device such as a smartphone. The image is then sent to a server via an application. At this time, the device compresses the image data and processes it to enable quick transmission.

[1010] Image analysis

[1011] The server analyzes the received image. It uses an image recognition algorithm to automatically identify ingredients. For example, ingredients such as chicken, tomatoes, and spices can be identified. Technologies such as YOLO (You Only Look Once) and TensorFlow are used for the analysis.

[1012] Menu generation

[1013] The server then searches the database for the most suitable recipe based on the identified ingredients. It generates the optimal menu, taking into account the user's nutritional needs and the expiration dates of the ingredients. For example, it might suggest a menu item such as "chicken steak with tomato salsa."

[1014] Nutritional analysis and presentation

[1015] The server analyzes the nutritional value of the generated menu. The analysis uses the nutritional information of each ingredient registered in the database. The amounts of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified. This nutritional information is then graphed and displayed visually to the user.

[1016] Suggestions for nutrient deficiencies

[1017] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For those nutrients, it suggests alternative foods or additional ingredients. For example, if you are deficient in vitamin D, it suggests eating mushrooms or fish.

[1018] Food delivery menu suggestions

[1019] The system proposes optimal food delivery menus via the user's device, taking into account the ingredients in the refrigerator and the user's nutritional needs, allowing users to easily select nutritionally balanced meals through the delivery service.

[1020] Nutritional Information

[1021] The user's device displays the menu and nutritional information received from the server. The user can practice a balanced diet based on the visually displayed graphs and list format information.

[1022] Specific examples

[1023] Let's say a user takes a photo of their refrigerator and it shows "chicken, tomatoes, and spices." The server analyzes it and identifies these ingredients. The server then suggests a meal such as "chicken steak with tomato salsa," and analyzes and visualizes its nutritional value (calories, protein, vitamin C, etc.). Furthermore, if the server determines that the user is lacking in vitamin D, it suggests adding mushrooms or fish. The user can check this information on their device and use it to plan their daily meals.

[1024] Prompt Sentence Examples

[1025] "You analyzed images of the inside of the refrigerator and identified chicken, tomatoes, and onions. Please use these to suggest the best delivery menu. Also, please show the nutritional value of the suggested menu."

[1026] The above is an embodiment of the present invention. This system allows users to effectively use ingredients and efficiently manage nutrition.

[1027] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1028] Step 1:

[1029] A user takes a picture of the food in the refrigerator with their smartphone. At this time, the device uses its camera function to capture the image and saves the image data in the application. The input is the image of the food in the refrigerator, and the output is the saved image data.

[1030] Step 2:

[1031] The image data captured by the device is compressed and sent to the server. This process reduces the size of the image data so that it can be sent quickly. The input is the saved image data, and the output is the compressed image data.

[1032] Step 3:

[1033] The server decodes the received compressed image data and applies image analysis algorithms to identify ingredients. The server uses YOLO or TensorFlow to identify each ingredient in the image. The input is the compressed image data, and the output is a list of identified ingredients.

[1034] Step 4:

[1035] The server searches the database for the best recipe based on the ingredient list and generates a menu. The generated menu takes into account the user's nutritional needs and the expiration dates of the ingredients. The input is the identified ingredient list, and the output is the generated menu.

[1036] Step 5:

[1037] The server analyzes the nutritional value of the generated menu, quantifies the amount of each nutrient, and visually graphs it. The input is the generated menu, and the output is the analyzed nutritional value data and its graphical representation.

[1038] Step 6:

[1039] The server compares daily nutritional intake with the analysis results, identifies nutrient deficiencies, and suggests additional ingredients. The input is the analyzed nutritional value data, and the output is the nutrient deficiencies and suggestions for supplementing them.

[1040] Step 7:

[1041] The server sends the generated menu, nutritional value data, and suggested information on nutrient deficiencies to the user terminal, which then displays them. The input is the nutritional value data, suggested nutrient deficiencies, and the generated menu, and the output is the information displayed on the user terminal.

[1042] Step 8:

[1043] The user terminal proposes an optimal food delivery menu that takes into account the ingredients in the refrigerator and the user's nutritional needs. The input is the list of ingredients in the refrigerator and the user's nutritional needs, and the output is the optimal food delivery menu.

[1044] The above are the specific processing steps of the system program that realizes this application example. This allows users to make effective use of ingredients, efficiently manage their nutrition, and use the optimal food delivery menu.

[1045] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1046] The present invention relates to a system that takes pictures of ingredients in a refrigerator, analyzes the pictures, and generates optimal menus for nutritional management. A feature of the present invention is that by combining it with an emotion engine that recognizes the user's emotions, it is possible to suggest more personalized meals.

[1047] System Overview

[1048] This system is primarily operated through devices such as smartphones and tablets. The devices have the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on them. It also has a function to analyze the user's emotional state using an emotion engine and adjust the menu and nutritional suggestions as described below.

[1049] Specific Embodiments of the System

[1050] 1. Take a photo of the ingredients and send it

[1051] A user takes a photo of the food in the refrigerator using a device such as a smartphone. The image is then sent to a server via an application. The device compresses the image data and processes it to ensure fast transmission.

[1052] 2. Image Analysis

[1053] The server analyzes the received image. An image recognition algorithm is used for the analysis, and ingredients are automatically identified. For example, ingredients such as chicken, tomatoes, and spices are identified. The algorithm for analyzing the image uses technologies such as YOLO (You Only Look Once) and TensorFlow.

[1054] 3. Menu generation

[1055] The server then searches the database for the most suitable recipe based on the identified ingredients. It generates the optimal menu, taking into account the user's nutritional needs and the expiration dates of the ingredients. For example, it might suggest a menu item such as "chicken steak with tomato salsa."

[1056] 4. Analysis and presentation of nutritional value

[1057] The server analyzes the nutritional value of the generated menu. The analysis uses the nutritional information of each ingredient registered in the database. The amounts of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified. This nutritional information is then graphed and displayed visually to the user.

[1058] 5. Suggestions for nutrient deficiencies

[1059] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For those nutrients, it suggests alternative foods or additional ingredients. For example, if you are deficient in vitamin D, it suggests eating mushrooms or fish.

[1060] 6. Menu and nutrition information display

[1061] The device receives the menu and nutritional information from the server and displays it to the user, who can then practice a balanced diet based on the visually displayed graphs and list-style information.

[1062] 7. Voice question function

[1063] Users can use voice input to ask questions or get more detailed information. For example, if they ask, "How do I cook this dish?", the server will analyze the voice data and provide the appropriate cooking method. The answer will be played back as text or audio.

[1064] 8. Calendar Management

[1065] The device has a function to manage daily menus and nutritional intake status in a calendar format. Users can view past data and future plans and manage their daily eating habits. This allows for long-term nutritional management.

[1066] 9. Incorporating an Emotional Engine

[1067] The server incorporates an emotion engine to recognize emotions from the user's facial expressions and voice. This emotional data is reflected in meal suggestions. For example, if the user is feeling stressed, the server can suggest ingredients and menus that have a relaxing effect.

[1068] 10. Emotional Adjustment

[1069] The server analyzes the accumulated emotional data and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake. For example, if a particular ingredient or menu improves the user's mood, personalized suggestions will be made based on that information.

[1070] Specific Examples

[1071] For example, suppose a user takes a photo of their refrigerator and sees that it contains "chicken, tomatoes, and spices." The server analyzes the photo and identifies these ingredients. The server then suggests a menu item such as "chicken steak with tomato salsa," and also considers the user's emotional state to suggest adding ingredients that have a relaxing effect. The server also analyzes and visualizes nutritional values ​​(calories, protein, vitamin C, etc.), suggesting adding mushrooms or fish to compensate for vitamin D deficiency. Users can check this information on their device and use it to plan their daily meals.

[1072] The above is an embodiment of the present invention. This system allows users to effectively use ingredients and efficiently manage nutrition, and also allows users to receive personalized suggestions based on their emotions.

[1073] The processing flow will be explained below.

[1074] Step 1:

[1075] When a user launches the smartphone application and takes a photo of the ingredients in the refrigerator, the device activates its camera function and displays a screen on which the user can take a photo of the ingredients.

[1076] Step 2:

[1077] The user checks the photo they have taken and presses the "Send" button in the application. The device sends the photo data to the server. When sending, the device compresses the image data to ensure it can be sent quickly.

[1078] Step 3:

[1079] The server receives the transmitted photo and runs an image recognition algorithm to analyze the received image data.

[1080] Step 4:

[1081] The server uses image recognition algorithms to identify ingredients in the received image, and lists the ingredients identified, for example, "chicken, tomatoes, and spices."

[1082] Step 5:

[1083] The server searches the database for the best recipe based on the identified ingredients, taking into account the user's nutritional needs (e.g., calorie restriction, vitamin intake, etc.) and expiration dates of the ingredients.

[1084] Step 6:

[1085] The server generates an optimal menu. For example, it might suggest "chicken steak with tomato salsa." Details of the suggested menu are stored in a database.

[1086] Step 7:

[1087] The server analyzes the nutritional value of the generated menu. The nutritional information includes calories, protein, fat, vitamins, minerals, etc. This information is obtained from a database.

[1088] Step 8:

[1089] The server quantifies the amount of each nutrient and calculates the total amount, then graphs the calculated nutritional information to generate data that can be displayed visually.

[1090] Step 9:

[1091] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For example, if there is a vitamin D deficiency, it will recommend additional foods and ingredients to supplement it.

[1092] Step 10:

[1093] The server transmits information including incomplete nutritional suggestions to the terminal, including menu details and nutritional value information.

[1094] Step 11:

[1095] The terminal displays the received information to the user in graph and list format so that the user can easily understand it.

[1096] Step 12:

[1097] Users can ask questions or get more information by speaking, for example, "How do I prepare this dish?"

[1098] Step 13:

[1099] The device converts the voice input into text and sends the text data to the server.

[1100] Step 14:

[1101] The server analyzes the question and generates an appropriate answer, such as "How to cook chicken steak..." and provides specific instructions.

[1102] Step 15:

[1103] The server generates a response and sends it to the terminal, which displays or plays it aloud to the user.

[1104] Step 16:

[1105] The device manages daily menus and nutritional intake in a calendar format, allowing users to check the calendar and view past data and upcoming plans.

[1106] Step 17:

[1107] The device activates an emotion engine that recognizes emotions from the user's facial expressions and voice. The user's emotional state is collected through the application. For example, facial expression analysis can determine whether the user is smiling, and voice analysis can determine whether the user is under stress.

[1108] Step 18:

[1109] The server analyzes the collected emotional data using an emotion engine. Based on the analysis results, it adjusts menu and nutrition suggestions based on the user's emotions. For example, if the user is feeling stressed, it will suggest ingredients and menus that have a relaxing effect.

[1110] Step 19:

[1111] The server analyzes the accumulated emotional data over the long term and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake. For example, if a particular ingredient or menu improves the user's mood, personalized suggestions will be made based on that information.

[1112] In this way, by performing specific actions at each step, the system efficiently supports food inventory management and nutritional management.In addition, personalized suggestions that take into account the user's emotional state can provide a more satisfying service.

[1113] Example 2

[1114] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1115] Conventional menu suggestion systems have difficulty effectively utilizing ingredients in the refrigerator and providing balanced nutritional management. Furthermore, they are unable to provide personalized suggestions that take the user's emotional state into account, resulting in low satisfaction. Furthermore, they lack the ability to provide appropriate answers to voice questions from users and to manage past nutritional intake history. By resolving these issues, it is necessary to improve the quality and satisfaction of users' dietary habits.

[1116] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for a user to take an image of ingredients; means for compressing the taken image at a terminal and sending it to the server; means for the server to analyze the received image and identify ingredients; means for generating a menu based on the identified ingredients, taking into consideration the user's nutritional needs and the expiration dates of the ingredients; means for analyzing the nutritional value of the generated menu and quantifying and graphing calorie and nutrient values; means for identifying missing nutrients based on the analysis results and suggesting substitute foods and additional ingredients; means for displaying information about the menu and nutritional value on a user terminal; means for the server to provide appropriate information in response to questions from the user via voice input, and for the user terminal to provide answers in text or voice; means for managing daily menus and nutritional intake status in a calendar format and providing the user with past and future information; means for analyzing emotions from the user's facial expressions and voice, and personalizing meal suggestions using emotional data; and means for suggesting ingredients and menus with a relaxing effect according to a specific emotional state, and optimizing individual meal suggestions by analyzing the relationship between past emotions and nutritional intake. This will allow for effective use of ingredients in the refrigerator, optimizing the user's nutritional management, and enabling personalized meal suggestions based on the user's emotional state.

[1117] "User" refers to a person who uses the system to take pictures of ingredients and receive menu suggestions and nutritional management.

[1118] "Terminal" refers to an electronic device such as a smartphone or tablet that takes and sends images of ingredients, communicates with the server, and displays information.

[1119] The "server" refers to a remote data processing device that analyzes images of ingredients, generates menus, analyzes nutritional values, suggests nutrient deficiencies, and so on.

[1120] "Image analysis" refers to the technical process of analyzing received image data of ingredients to automatically identify the ingredients.

[1121] "Ingredient identification" refers to the process of identifying and listing types of ingredients through image analysis.

[1122] "Menu generation" refers to the process of determining the optimal menu based on the identified ingredients, taking into account the user's nutritional needs and the expiration dates of the ingredients.

[1123] "Nutritional analysis" refers to the process of calculating the nutritional information of the generated menu and quantifying and graphing the calories and amount of each nutrient.

[1124] "Nutrient deficiency suggestion" refers to the process of identifying missing nutrients based on analyzed nutritional value data and suggesting appropriate substitute foods or additional ingredients.

[1125] "Voice input" refers to a means by which a user uses voice to ask questions or give instructions to a system.

[1126] "Emotion analysis" refers to the process of recognizing emotions from a user's facial expressions and voice and analyzing that data.

[1127] "Personalization" refers to providing information and services that are individually optimized based on a user's specific conditions and tendencies.

[1128] "Calendar management" refers to the process of recording daily menus and nutritional intake status in calendar format and providing users with past data and future schedules.

[1129] The present invention relates to a system that takes images of ingredients in a refrigerator, analyzes the images, creates optimal menus, and manages nutrition. Furthermore, the present invention can improve user satisfaction and health management by recognizing the user's emotions and providing personalized meal suggestions based on those emotions.

[1130] System Overview

[1131] This system is primarily operated through devices such as smartphones and tablets. The devices have the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on that information. It also has a function to analyze the user's emotional state using an emotion engine and adjust the menu and nutritional suggestions accordingly.

[1132] Hardware and software used

[1133] This system uses the following hardware and software:

[1134] Devices: smartphones, tablets

[1135] Server: High-performance data processing server

[1136] Image analysis algorithm: YOLO (You Only Look Once), TensorFlow

[1137] Emotion analysis engine: speech recognition software, facial recognition software

[1138] System operation flow

[1139] The system works as follows:

[1140] 1. Take a photo of the ingredients and send it

[1141] The user uses the camera on their smartphone or tablet to take a photo of the food in their refrigerator.

[1142] The device compresses the captured images and quickly transmits the data to the server.

[1143] 2. Image Analysis

[1144] The server decodes the received image and converts it into an analyzable format.

[1145] The server uses image recognition algorithms such as YOLO and TensorFlow to automatically identify ingredients in the image.

[1146] For example, ingredients such as chicken, tomatoes, and spices are identified.

[1147] 3. Menu generation

[1148] The server searches the database for the most suitable recipe based on the identified ingredient list.

[1149] The server generates optimal menus taking into account the user's nutritional needs and the expiration dates of ingredients.

[1150] For example, a menu suggestion might be "chicken steak with tomato salsa."

[1151] 4. Analysis and presentation of nutritional value

[1152] The server analyzes the nutritional value of the generated menu, using the nutritional information of each ingredient registered in the database.

[1153] The amount of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified.

[1154] This nutritional information is graphed and displayed visually to the user.

[1155] 5. Suggestions for nutrient deficiencies

[1156] Based on the analysis results, the server compares the nutrients needed per day with the nutritional value of the generated menu and identifies any nutrients that are lacking.

[1157] For those who are lacking in nutrients, we suggest alternative foods or additional ingredients. For example, if you are lacking in vitamin D, we suggest eating mushrooms or fish.

[1158] 6. Menu and nutrition information display

[1159] The terminal displays the menu and nutritional information received from the server to the user.

[1160] Users can practice balanced eating habits based on visually displayed graphs and list-style information.

[1161] 7. Voice question function

[1162] Users can use voice input to ask questions or for additional information.

[1163] For example, if you ask, "How do you cook this dish?", the server will analyze the voice data and provide the appropriate cooking method as an answer. The answer will be played back in text or audio.

[1164] 8. Calendar Management

[1165] The device has the function of managing daily menus and nutritional intake in calendar format.

[1166] Users can view past data and future plans and manage their daily eating habits.

[1167] 9. Incorporating an Emotional Engine

[1168] The server incorporates an emotion engine to recognize emotions from the user's facial expressions and voice.

[1169] Emotional data is reflected in meal suggestions. For example, if a user is feeling stressed, the system can suggest ingredients and menu items that have a relaxing effect.

[1170] 10. Emotional Adjustment

[1171] The server analyzes the accumulated emotional data and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake.

[1172] For example, if a particular ingredient or menu item improves a user's mood, personalized suggestions can be made based on that information.

[1173] Specific examples

[1174] For example, suppose a user takes a photo of their refrigerator, which includes "chicken, tomatoes, and spices." The device compresses this and sends it to the server. The server analyzes the photo and identifies these ingredients. The server then suggests a menu item such as "chicken steak with tomato salsa," and also considers the user's emotional state and suggests adding ingredients that have a relaxing effect. The server also analyzes and visualizes nutritional values ​​(calories, protein, vitamin C, etc.), suggesting adding mushrooms or fish to compensate for vitamin D deficiency. Users can check this information on their device and use it to plan their daily meals.

[1175] This invention allows users to make effective use of ingredients and efficiently manage their nutrition, and also allows users to receive personalized suggestions based on their emotions.

[1176] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1177] Step 1: Photograph the ingredients and send the image

[1178] The user uses a device (smartphone or tablet) to take a photo of the food in the refrigerator.

[1179] Input: Images of ingredients in the refrigerator.

[1180] The device compresses the images it takes, reducing the amount of data it takes to send them over the network.

[1181] The terminal transmits the compressed image data to the server.

[1182] Output: Compressed image data sent to the server.

[1183] Step 2: Image analysis and ingredient identification

[1184] The server decodes the received image data and converts it into an analyzable format.

[1185] Input: Received compressed image data.

[1186] The server uses image recognition algorithms such as YOLO (You Only Look Once) and TensorFlow to automatically identify ingredients.

[1187] Image analysis algorithms identify the type of ingredient (e.g., chicken, tomato, spices, etc.) from the image.

[1188] Output: A list of identified ingredients.

[1189] Step 3: Create a menu

[1190] The server searches the registered database for the most suitable recipe based on the identified ingredient list.

[1191] Input: Identified ingredient list, recipe information in database.

[1192] The server generates optimal menus taking into account the user's nutritional needs and the expiration dates of ingredients.

[1193] For example, a menu such as "chicken steak with tomato salsa" may be generated.

[1194] Output: The generated menu information.

[1195] Step 4: Nutritional analysis and presentation

[1196] The server analyzes the nutritional value of the generated menu.

[1197] Input: Generated menu information, nutrition information in the database.

[1198] The amount of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified.

[1199] This nutritional information is graphed and formatted in a way that can be visually presented to the user.

[1200] Output: Nutritional values ​​are displayed in graph and list format.

[1201] Step 5: Suggesting nutrient deficiencies

[1202] The server analyzes the nutritional value of the menu and identifies any nutrients that are lacking by comparing them with the daily required nutrients.

[1203] Input: Quantified nutritional value data, reference values ​​for nutrients needed per day.

[1204] The server will identify any nutrient deficiencies and suggest substitutions or additional ingredients.

[1205] For example, if you are deficient in vitamin D, it is suggested that you "consume additional mushrooms and fish."

[1206] Output: Suggested information on nutrient deficiencies.

[1207] Step 6: Display menu and nutrition information

[1208] The server sends the generated menu and nutritional information to the terminal.

[1209] Input: Menu information, nutritional information.

[1210] The information received by the terminal is visually displayed to the user.

[1211] The device displays the information in graph and list format, and users can use the information to practice a balanced diet.

[1212] Output: The visual information that is displayed to the user.

[1213] Step 7: Voice Question Function

[1214] Users use voice input to ask questions or for additional information.

[1215] Input: Audio data.

[1216] The device converts the voice data into text and sends it to the server.

[1217] The server analyzes the question and generates an appropriate answer.

[1218] The device will present the answer to the user in text or audio.

[1219] For example, if you ask, "How do you cook this dish?", the cooking instructions will be answered in text or voice.

[1220] Output: The answer information provided to the user.

[1221] Step 8: Calendar Management

[1222] The device manages daily menus and nutritional intake in calendar format.

[1223] Input: Menu information, nutritional intake data.

[1224] Users can view past data and future schedules.

[1225] The terminal displays information in calendar format, enabling long-term nutritional management.

[1226] Output: Information displayed in a calendar format.

[1227] Step 9: Incorporating the Emotion Engine

[1228] The server uses an emotion engine to recognize emotions from the user's facial expressions and voice data.

[1229] Input: User's facial expression data, voice data.

[1230] A sentiment analysis engine analyzes this data and infers the user's emotional state (e.g., stress, happiness, etc.).

[1231] Output: Emotional state data.

[1232] Step 10: Emotional Adjustment

[1233] The server analyzes the accumulated emotional data and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake.

[1234] Input: Emotional state data, nutritional intake data.

[1235] Depending on your specific emotional state, it will suggest foods and menus that have a relaxing effect.

[1236] For example, if a user is feeling stressed, a menu containing ingredients that have a relaxing effect will be suggested based on that information.

[1237] Output: Personalized meal suggestion information.

[1238] (Application example 2)

[1239] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1240] Conventional food management systems have limitations in efficiently managing ingredients in the refrigerator and suggesting appropriate menus. Furthermore, they make uniform suggestions without considering the user's emotions, resulting in insufficient personalization to meet individual needs. Furthermore, there has been a lack of systems that can utilize new devices such as smart glasses to recognize ingredients and emotions in real time.

[1241] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for capturing images of ingredients, a means for transmitting the captured images to the server, a means for the server to analyze the images and identify ingredients, a means for generating a menu based on the identified ingredients, a means for analyzing, graphing, and quantifying the nutritional value of the generated menu, a means for presenting missing nutrients based on the analysis results, a means for displaying the menu and nutritional value information on a user terminal, a means for recognizing the user's emotions and reflecting them in meal suggestions, and a means for using smart glasses to recognize ingredients and emotions in real time and suggest a menu. This enables efficient recognition of ingredients in the user's refrigerator and personalized menu suggestions based on the user's individual emotional state. Furthermore, the use of smart glasses enables real-time ingredient recognition and emotion recognition, enabling more immediate and accurate information provision.

[1242] "Means for taking images of ingredients" refers to functionality that includes devices and software for capturing images of ingredients in a refrigerator or other storage location.

[1243] The "means for transmitting the captured image to the server" is a function for transmitting the image captured from the smart glasses or other terminal to the server via data communication.

[1244] The "means for the server to analyze the image and identify ingredients" refers to a system that uses algorithms or techniques to analyze received image data and identify ingredients.

[1245] The "means for generating a menu based on the identified ingredients" is a function that searches a database for the most suitable recipe based on the identified ingredients and suggests a menu.

[1246] The "means for analyzing, graphing and quantifying the nutritional value of the generated menu" is a function for quantifying each nutrient in the generated menu and visually displaying it.

[1247] The "means for presenting missing nutrients based on the analysis results" is a function for identifying missing nutrients by comparing them with the calculated nutritional values ​​and suggesting appropriate ingredients and supplements.

[1248] The "means for displaying the menu and nutritional value information on a user terminal" is a function for displaying the generated menu and its nutritional value information on a user terminal such as smart glasses or a smartphone.

[1249] "Means for recognizing the user's emotions and reflecting them in meal suggestions" is a function that analyzes the user's facial expressions and voice to identify their emotional state and then makes personalized meal suggestions based on that information.

[1250] "Means for using smart glasses to recognize ingredients and emotions in real time and suggest menus" is a function that uses the cameras and sensors in smart glasses to recognize ingredients and the user's emotions in real time and suggests menus based on that information.

[1251] The present invention provides a system for efficiently managing ingredients in a refrigerator and proposing personalized menus based on the user's emotions. Here, specific embodiments of the present invention will be described.

[1252] System Overview

[1253] The main components of this system are smart glasses (or a user terminal such as a smartphone), a server, and software for recognizing images of ingredients in the refrigerator and the user's emotions.

[1254] 1. Food and emotion recognition

[1255] A user uses smart glasses to check the ingredients in the refrigerator. At this time, the camera in the smart glasses captures an image of the ingredients and sends the image data to the server. At the same time, the sensors in the smart glasses capture the user's facial expressions and voice to recognize their emotions. For example, if a user opens the refrigerator and says, "I want to relax today," that voice data is also sent to the server.

[1256] 2. Image analysis and emotion analysis on the server

[1257] The server first analyzes the received image data, using the YOLO (You Only Look Once) model and TensorFlow algorithms to automatically identify the ingredients in the refrigerator.

[1258] The received facial and voice data is then analyzed using an emotion engine, which is used to identify the user's current emotional state (e.g., relaxed, stressed, depressed, etc.).

[1259] 3. Menu Creation and Nutritional Analysis

[1260] Based on the identified ingredients and the user's emotional state, the server searches for the most suitable recipe from the database and generates a menu. For example, if the ingredients are "chicken, tomato, and spices" and the user is recognized as "feeling stressed," the server will suggest "chicken steak with tomato salsa" with "herbal tea," which has a relaxing effect, as a side dish.

[1261] To analyze the nutritional value of the generated menu, the system refers to a nutrition database and converts information such as total calories, protein, fat, vitamins, and minerals into numerical values ​​and graphs, which the user can visually check.

[1262] 4. Display on the user's device

[1263] The generated menu information and nutritional information are displayed on the user's device, such as smart glasses or a smartphone, allowing the user to efficiently use the ingredients in their refrigerator and practice nutritionally balanced meals.

[1264] 5. Examples of prompts

[1265] Specific examples of prompts include:

[1266] "It recognizes the ingredients in the refrigerator and suggests personalized meals based on the user's emotions. For example, if it analyzes an image containing "chicken, tomatoes, and spices" and determines that the user is feeling stressed, it will suggest a meal that includes ingredients that have a relaxing effect."

[1267] This system allows users to make the most of the ingredients in their refrigerator and receive personalized meal suggestions based on their emotions. It also recognizes ingredients and emotions in real time and provides appropriate information instantly.

[1268] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1269] Step 1:

[1270] A user uses smart glasses to check the ingredients in the refrigerator and captures the images with the camera. The input is the image of the ingredients in the refrigerator, and the output is the captured image data.

[1271] Step 2:

[1272] The smart glasses transmit the captured image data to a server. The input is the captured image data, and the output is the transmission of the image data to the server. In actual operation, the smart glasses use a communication module to compress the image data and transmit it to the server via the Internet.

[1273] Step 3:

[1274] The server analyzes the received image data and identifies the ingredients. The input is the image data sent to the server, and the output is a list of identified ingredients. Specifically, the server uses the YOLO model to detect ingredients in the image and create a list of ingredient names.

[1275] Step 4:

[1276] The smart glasses capture the user's facial expressions and voice and send the data to the server. The input is the user's facial expressions and voice data, and the output is data transmission to the server. The smart glasses' sensors and microphone are used to capture the user's facial expressions and voice in real time and send the data to the server.

[1277] Step 5:

[1278] The server analyzes the received facial and voice data to recognize the user's emotions. The input is the facial and voice data sent to the server, and the output is the recognized emotion data. Specifically, the emotion engine is used to analyze the facial and voice data and identify the user's emotional state.

[1279] Step 6:

[1280] The server generates an optimal menu based on the identified ingredients and the recognized emotion data. The input is the identified ingredient list and emotion data, and the output is the generated menu. The server searches for appropriate recipes from the database and selects meals that suit the user's emotions.

[1281] Step 7:

[1282] The server analyzes the nutritional value of the generated menu and graphs and quantifies it. The input is the generated menu, and the output is the nutritional value and graph. The server refers to a nutrition database, tallying the nutritional information of each ingredient and calculating the nutritional value.

[1283] Step 8:

[1284] The server then presents the nutritional deficiencies based on the analysis results. The input is the nutritional analysis results, and the output is suggestions regarding the nutrients that are lacking. It identifies the nutrients that are lacking and suggests additional ingredients or supplements to make up for them.

[1285] Step 9:

[1286] The generated menu and nutritional information are displayed on the user's device. The input is the generated menu and nutritional information, and the output is the display on the user's device. The menu and nutritional information are displayed on smart glasses or a smartphone, allowing the user to visually confirm them.

[1287] Step 10:

[1288] The user asks a question by voice input, and the server provides the corresponding information. The input is the user's voice question, and the output is the answer from the server. The server uses a voice recognition system to analyze the user's question and provide appropriate information and explanations.

[1289] Step 11:

[1290] The user terminal manages daily menus and nutritional intake status in a calendar format, providing users with past and future information. The input is past and current menu data, and the output is a visual display in calendar format. Users can manage their nutrition over the long term.

[1291] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1292] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1293] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1294] [Fourth embodiment]

[1295] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1296] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1297] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1298] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1299] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1300] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1301] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1302] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1303] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1304] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1305] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1306] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1307] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1308] The present invention relates to a system for nutritional management that takes images of ingredients in a refrigerator, analyzes the images, and generates optimal menus. This system provides users with a means for efficiently managing their daily nutrition and making effective use of ingredients.

[1309] System Overview

[1310] This system is operated through devices such as smartphones and tablets. The devices have the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on that information. The nutritional value of the generated menu is analyzed and visually displayed to the user.

[1311] Specific Embodiments of the System

[1312] 1. Take a photo of the ingredients and send it

[1313] A user takes a photo of the food in the refrigerator using a device such as a smartphone. The image is then sent to a server via an application. At this time, the device compresses the image data and processes it to enable quick transmission.

[1314] 2. Image Analysis

[1315] The server analyzes the received image. An image recognition algorithm is used for the analysis, and ingredients are automatically identified. For example, ingredients such as chicken, tomatoes, and spices are identified. The algorithm for analyzing the image uses technologies such as YOLO (You Only Look Once) and TensorFlow.

[1316] 3. Menu generation

[1317] The server then searches the database for the most suitable recipe based on the identified ingredients. It generates the optimal menu, taking into account the user's nutritional needs and the expiration dates of the ingredients. For example, it might suggest a menu item such as "chicken steak with tomato salsa."

[1318] 4. Analysis and presentation of nutritional value

[1319] The server analyzes the nutritional value of the generated menu. The analysis uses the nutritional information of each ingredient registered in the database. The amounts of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified. This nutritional information is then graphed and displayed visually to the user.

[1320] 5. Suggestions for nutrient deficiencies

[1321] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For those nutrients, it suggests alternative foods or additional ingredients. For example, if you are deficient in vitamin D, it suggests eating mushrooms or fish.

[1322] 6. Menu and nutrition information display

[1323] The device receives the menu and nutritional information from the server and displays it to the user, who can then practice a balanced diet based on the visually displayed graphs and list-style information.

[1324] 7. Voice question function

[1325] Users can use voice input to ask questions or get more detailed information. For example, if they ask, "How do I cook this dish?", the server will analyze the voice data and provide the appropriate cooking method. The answer will be played back as text or audio.

[1326] 8. Calendar Management

[1327] The device has a function to manage daily menus and nutritional intake status in a calendar format. Users can view past data and future plans and manage their daily eating habits. This allows for long-term nutritional management.

[1328] Specific Examples

[1329] For example, suppose a user takes a photo of their refrigerator and sees that it contains "chicken, tomatoes, and spices." The server analyzes the photo and identifies these ingredients. The server then suggests a meal such as "chicken steak with tomato salsa," and analyzes and visualizes its nutritional value (calories, protein, vitamin C, etc.). Furthermore, if the server determines that the user is lacking in vitamin D, it suggests adding mushrooms or fish. Users can check this information on their device and use it to plan their daily meals.

[1330] The above is an embodiment of the present invention. This system allows users to effectively use ingredients and efficiently manage nutrition.

[1331] The processing flow will be explained below.

[1332] Step 1:

[1333] When a user launches the smartphone application and takes a photo of the ingredients in the refrigerator, the device activates its camera function and displays a screen on which the user can take a photo of the ingredients.

[1334] Step 2:

[1335] The user checks the photo they have taken and presses the "Send" button in the application. The device sends the photo data to the server.

[1336] Step 3:

[1337] The server receives the transmitted photo and runs an image recognition algorithm to analyze the received image data.

[1338] Step 4:

[1339] The server uses image recognition algorithms to identify ingredients in the received image, and lists the ingredients identified, for example, "chicken, tomatoes, and spices."

[1340] Step 5:

[1341] The server searches the database for the best recipe based on the identified ingredients, taking into account the user's nutritional needs (e.g., calorie restriction, vitamin intake, etc.) and expiration dates of the ingredients.

[1342] Step 6:

[1343] The server generates an optimal menu. For example, it might suggest "chicken steak with tomato salsa." Details of the suggested menu are stored in a database.

[1344] Step 7:

[1345] The server analyzes the nutritional value of the generated menu. The nutritional information includes calories, protein, fat, vitamins, minerals, etc. This information is obtained from a database.

[1346] Step 8:

[1347] The server quantifies the amount of each nutrient and calculates the total amount, then graphs the calculated nutritional information to generate data that can be displayed visually.

[1348] Step 9:

[1349] The server compares the analysis results with the daily required nutrients and identifies any deficiencies. For example, if there is a vitamin D deficiency, it will recommend additional ingredients or foods to supplement it.

[1350] Step 10:

[1351] The server transmits information including incomplete nutritional suggestions to the terminal, including menu details and nutritional value information.

[1352] Step 11:

[1353] The terminal displays the received information to the user in graph and list format so that the user can easily understand it.

[1354] Step 12:

[1355] Users can ask questions or get more information by speaking, for example, "How do I prepare this dish?"

[1356] Step 13:

[1357] The device converts the voice input into text and sends the text data to the server.

[1358] Step 14:

[1359] The server analyzes the question and generates an appropriate answer, such as "How to cook chicken steak..." and provides specific instructions.

[1360] Step 15:

[1361] The server generates a response and sends it to the terminal, which displays or plays it aloud to the user.

[1362] Step 16:

[1363] The device manages daily menus and nutritional intake in a calendar format, allowing users to check the calendar and view past data and upcoming plans.

[1364] In this way, by performing specific actions at each step, the system efficiently supports food inventory management and nutritional management.

[1365] Example 1

[1366] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1367] Managing ingredients and planning nutritionally balanced meals is time-consuming and laborious, making it difficult for many people. It's particularly challenging to effectively utilize ingredients in the refrigerator and manage daily nutritional intake. Furthermore, it's difficult to instantly obtain specific cooking instructions and nutritional information for a menu. A system that can solve these problems is needed.

[1368] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1369] In this invention, the server includes means for analyzing and identifying images of ingredients, means for generating menus based on the identified ingredients, and means for analyzing and presenting the nutritional values ​​of the generated menus, thereby enabling users to efficiently use ingredients in their refrigerators and easily create balanced meal plans.

[1370] "Means for taking pictures of ingredients" is a function for taking pictures of ingredients in the refrigerator using a camera on a smartphone, tablet, or other device.

[1371] The "means for transmitting the captured image to the information processing device" is a function for appropriately compressing the captured image and transmitting it to an information processing device on a server or cloud.

[1372] "Means for the information processing device to analyze the image and identify ingredients" refers to a function that enables an information processing device located on a server or cloud to detect and identify ingredients in an image using image recognition algorithms such as YOLO or TensorFlow.

[1373] The "means for generating a menu based on the identified ingredients" is a function for automatically suggesting a menu by selecting the optimal recipe based on the identified ingredient information, taking into consideration the user's nutritional needs and the expiration date of the ingredients.

[1374] The "means for analyzing, graphing, and quantifying the nutritional value of the generated menu" is a function that uses the nutritional information of each ingredient registered in the database to analyze the total calories, protein, lipids, vitamins, minerals, etc. of the generated menu and visually display this information.

[1375] The "means for suggesting missing nutrients based on the analysis results" is a function that compares the results of the nutritional analysis of the menu with the amount of nutrients needed per day, identifies missing nutrients, and suggests ingredients or alternative foods to replenish them.

[1376] The "means for displaying the menu and nutritional value information on a user device" is a function for visually displaying the generated menu and its nutritional information in graph or list format on a device such as a smartphone or tablet used by the user.

[1377] "Means for managing daily menus and nutritional intake status in calendar format and providing users with past and future information" is a function that displays the user's daily food records and nutritional intake status in calendar format, allowing them to refer to past data and future plans.

[1378] "Means for an information processing device to provide information corresponding to a question from a user via voice input" is a function that enables an information processing device on a server or cloud to analyze the content of a question when the user asks it via voice input and provide an appropriate answer in voice or text.

[1379] The "means for compressing images of ingredients and quickly transmitting them to an information processing device" is a function for appropriately compressing large amounts of ingredient images, improving communication speed and data transmission efficiency, and transmitting them to a server or cloud.

[1380] The present invention is a system for nutritional management that takes images of ingredients in a refrigerator, analyzes the images, and generates optimal menus. This system provides users with a means for efficiently managing their daily nutrition and making effective use of ingredients. Specific embodiments of the system are described below.

[1381] System hardware and software configuration

[1382] This system is operated using a device such as a smartphone or tablet. The device has the function of taking photos of ingredients and sending them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on that information. The nutritional value of the generated menu is analyzed and visually displayed to the user.

[1383] The device must be equipped with a high-resolution camera and internet connection. The server must have a powerful computer and a large database. Image recognition algorithms such as YOLO (You Only Look Once) and TensorFlow are used for image recognition.

[1384] Software used

[1385] Image recognition algorithms: YOLO, TensorFlow

[1386] Database management systems: MySQL, PostgreSQL

[1387] Cloud platforms: AWS (Amazon Web Services), Google Cloud Platform

[1388] Specific examples

[1389] For example, a user can take a photo of "chicken, tomatoes, and spices" in the refrigerator. The device compresses the image and quickly sends it to the server. The server analyzes the received image and identifies these ingredients. The server then uses its database to search for the most suitable recipe and suggests a menu item such as "chicken steak with tomato salsa." The server also analyzes and visualizes the nutritional value of this menu item (total calories, protein, vitamin C, etc.). Furthermore, if it is determined that the user is lacking in vitamin D, the server will suggest adding mushrooms or fish. Users can check this information on their device and use it to plan their daily meals.

[1390] Prompt Sentence Examples

[1391] You can use prompts like the following for your generative AI model:

[1392] A user takes a picture of the ingredients in their refrigerator and sends it to the server. This image shows chicken, tomatoes, and spices. Suggest a meal plan using these ingredients, analyze their nutritional value, and suggest additional ingredients if certain nutrients are lacking.

[1393] As described above, this system allows users to efficiently utilize ingredients and manage their nutrition. Users can also easily ask questions via voice input and instantly obtain specific cooking methods and additional nutritional information. This makes it easier to plan and follow balanced meals.

[1394] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1395] Step 1:

[1396] The user takes a photo of the ingredients. The user uses the camera on their smartphone or tablet to take a photo of the ingredients in the refrigerator. For example, they take a photo of chicken, tomatoes, spices, etc. stored in the refrigerator. In this case, the input data is the image of the ingredients, and the output data is the image file stored in the device.

[1397] Step 2:

[1398] The device preprocesses the captured image. Specifically, it compresses the image data to reduce the file size and make it easier to send to the server. In this process, the input is a high-resolution food image, and the output is compressed image data. For example, the image is compressed using the JPEG format.

[1399] Step 3:

[1400] The terminal sends the preprocessed image data to the server. The compressed image data is uploaded to the server via the Internet. The input in this step is the compressed image data, and the output is the image data stored on the server.

[1401] Step 4:

[1402] The server performs image analysis. It uses image recognition algorithms such as YOLO and TensorFlow to analyze the uploaded image. The input in this step is the image data stored on the server, and the output is a list of identified ingredients. For example, chicken, tomatoes, spices, etc. in the image are automatically identified.

[1403] Step 5:

[1404] The server generates a menu based on the identified ingredient information. It searches recipe information stored in a database and proposes an optimal menu, taking into account the user's nutritional needs and the expiration dates of ingredients. The input in this step is the identified ingredient list, and the output is the generated menu information. For example, a specific menu such as "chicken steak with tomato salsa" is proposed.

[1405] Step 6:

[1406] The server analyzes the nutritional value of the generated menu. Using the nutritional information of each ingredient registered in the database, it quantifies and visualizes nutrients such as total calories, protein, fat, vitamins, and minerals. The input in this step is the generated menu information, and the output is numerical information on nutritional value and graphed data.

[1407] Step 7:

[1408] The server identifies any nutrient deficiencies based on the analysis results and suggests alternative foods or additional ingredients. For example, if it determines that a person is deficient in vitamin D, it will suggest adding mushrooms or fish. The input in this step is numerical information on nutritional value, and the output is suggested information on deficient nutrients.

[1409] Step 8:

[1410] The device displays the generated menu and nutritional information to the user. Specifically, the information is displayed in a visually easy-to-understand graph or list format. The input in this step is the menu information and nutritional information, and the output is the visualized data displayed on the user device.

[1411] Step 9:

[1412] The user uses the voice question function. They can ask questions about cooking methods and additional nutritional information by voice input. For example, if they ask, "How do I cook this dish?", the server analyzes it and responds by voice or text. The input in this step is the voice question data, and the output is the response data from the server.

[1413] Step 10:

[1414] The device manages daily menus and nutritional intake status in a calendar format. Daily food records and nutritional intake status are compiled in a calendar format, and past data and future plans can be viewed. The input in this step is daily food data, and the output is information presented in calendar format.

[1415] (Application example 1)

[1416] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1417] Conventional refrigerator food management systems have had issues with the effective use of ingredients and insufficient nutritional management. In addition, because they are unable to propose optimal delivery menus that take into account the ingredients in the refrigerator, users are likely to waste ingredients or eat meals that are nutritionally unbalanced.

[1418] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1419] In this invention, the server includes means for taking images of ingredients, means for transmitting the taken images to the server, means for the server to analyze the images and identify ingredients, means for generating a menu based on the identified ingredients, means for analyzing the nutritional value of the generated menu and graphing and quantifying it, means for presenting nutrients that are lacking based on the analysis results, means for displaying information on the menu and nutritional values ​​on a user terminal, and means for suggesting an optimal delivery menu via the user terminal. This allows users to make the most of the ingredients in their refrigerators and use delivery services while maintaining a nutritionally balanced diet.

[1420] "Means for taking images of ingredients" refers to a device or function that uses a camera to take pictures of ingredients in the refrigerator.

[1421] "Means for transmitting the captured images to a server" refers to a function or process for transmitting the captured images to a remote server via the Internet or other communication means.

[1422] "Means for the server to analyze the image and identify ingredients" refers to the function of the server analyzing the image received by the server using machine learning algorithms or image recognition technology to accurately identify ingredients in the image.

[1423] The "means for generating a menu based on the identified ingredients" refers to a function that automatically suggests appropriate recipes and menus based on the identified ingredient information.

[1424] "Means for analyzing, graphing, and quantifying the nutritional value of the generated menu" refers to a function that analyzes the nutritional information of the proposed menu, graphs the results for visual display, and provides them as specific numerical data.

[1425] "Means for suggesting nutrients that are lacking based on the analysis results" refers to a function that suggests nutrients that the user is lacking and additional ingredients that are needed based on the results of the nutritional value analysis of the menu.

[1426] "Means for displaying the menu and nutritional value information on the user terminal" refers to a function for displaying the proposed menu and its nutritional value information on the user's terminal such as a smartphone or tablet.

[1427] "Means for proposing the most suitable delivery menu via the user terminal" refers to a function that provides the most suitable delivery menu taking into consideration the ingredients in the user's refrigerator and their nutritional needs.

[1428] The present invention relates to a system that takes images of ingredients in a refrigerator, analyzes the images to generate optimal menus, and also suggests food delivery services. This system provides a means for users to effectively use ingredients in the refrigerator while achieving nutritionally balanced meals.

[1429] System Overview

[1430] The system is operated through a user device such as a smartphone or tablet. The user device has the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on them. The generated menu is then analyzed for nutritional value and visually displayed to the user. The system also suggests optimal food delivery menus based on the ingredients in the user's refrigerator and their nutritional needs.

[1431] Specific Embodiments of the System

[1432] Photograph and send ingredients

[1433] A user takes a photo of the food in the refrigerator using a device such as a smartphone. The image is then sent to a server via an application. At this time, the device compresses the image data and processes it to enable quick transmission.

[1434] Image analysis

[1435] The server analyzes the received image. It uses an image recognition algorithm to automatically identify ingredients. For example, ingredients such as chicken, tomatoes, and spices can be identified. Technologies such as YOLO (You Only Look Once) and TensorFlow are used for the analysis.

[1436] Menu generation

[1437] The server then searches the database for the most suitable recipe based on the identified ingredients. It generates the optimal menu, taking into account the user's nutritional needs and the expiration dates of the ingredients. For example, it might suggest a menu item such as "chicken steak with tomato salsa."

[1438] Nutritional analysis and presentation

[1439] The server analyzes the nutritional value of the generated menu. The analysis uses the nutritional information of each ingredient registered in the database. The amounts of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified. This nutritional information is then graphed and displayed visually to the user.

[1440] Suggestions for nutrient deficiencies

[1441] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For those nutrients, it suggests alternative foods or additional ingredients. For example, if you are deficient in vitamin D, it suggests eating mushrooms or fish.

[1442] Food delivery menu suggestions

[1443] The system proposes optimal food delivery menus via the user's device, taking into account the ingredients in the refrigerator and the user's nutritional needs, allowing users to easily select nutritionally balanced meals through the delivery service.

[1444] Nutritional Information

[1445] The user's device displays the menu and nutritional information received from the server. The user can practice a balanced diet based on the visually displayed graphs and list format information.

[1446] Specific examples

[1447] Let's say a user takes a photo of their refrigerator and it shows "chicken, tomatoes, and spices." The server analyzes it and identifies these ingredients. The server then suggests a meal such as "chicken steak with tomato salsa," and analyzes and visualizes its nutritional value (calories, protein, vitamin C, etc.). Furthermore, if the server determines that the user is lacking in vitamin D, it suggests adding mushrooms or fish. The user can check this information on their device and use it to plan their daily meals.

[1448] Prompt Sentence Examples

[1449] "You analyzed images of the inside of the refrigerator and identified chicken, tomatoes, and onions. Please use these to suggest the best delivery menu. Also, please show the nutritional value of the suggested menu."

[1450] The above is an embodiment of the present invention. This system allows users to effectively use ingredients and efficiently manage nutrition.

[1451] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1452] Step 1:

[1453] A user takes a picture of the food in the refrigerator with their smartphone. At this time, the device uses its camera function to capture the image and saves the image data in the application. The input is the image of the food in the refrigerator, and the output is the saved image data.

[1454] Step 2:

[1455] The image data captured by the device is compressed and sent to the server. This process reduces the size of the image data so that it can be sent quickly. The input is the saved image data, and the output is the compressed image data.

[1456] Step 3:

[1457] The server decodes the received compressed image data and applies image analysis algorithms to identify ingredients. The server uses YOLO or TensorFlow to identify each ingredient in the image. The input is the compressed image data, and the output is a list of identified ingredients.

[1458] Step 4:

[1459] The server searches the database for the best recipe based on the ingredient list and generates a menu. The generated menu takes into account the user's nutritional needs and the expiration dates of the ingredients. The input is the identified ingredient list, and the output is the generated menu.

[1460] Step 5:

[1461] The server analyzes the nutritional value of the generated menu, quantifies the amount of each nutrient, and visually graphs it. The input is the generated menu, and the output is the analyzed nutritional value data and its graphical representation.

[1462] Step 6:

[1463] The server compares daily nutritional intake with the analysis results, identifies nutrient deficiencies, and suggests additional ingredients. The input is the analyzed nutritional value data, and the output is the nutrient deficiencies and suggestions for supplementing them.

[1464] Step 7:

[1465] The server sends the generated menu, nutritional value data, and suggested information on nutrient deficiencies to the user terminal, which then displays them. The input is the nutritional value data, suggested nutrient deficiencies, and the generated menu, and the output is the information displayed on the user terminal.

[1466] Step 8:

[1467] The user terminal proposes an optimal food delivery menu that takes into account the ingredients in the refrigerator and the user's nutritional needs. The input is the list of ingredients in the refrigerator and the user's nutritional needs, and the output is the optimal food delivery menu.

[1468] The above are the specific processing steps of the system program that realizes this application example. This allows users to make effective use of ingredients, efficiently manage their nutrition, and use the optimal food delivery menu.

[1469] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1470] The present invention relates to a system that takes pictures of ingredients in a refrigerator, analyzes the pictures, and generates optimal menus for nutritional management. A feature of the present invention is that by combining it with an emotion engine that recognizes the user's emotions, it is possible to suggest more personalized meals.

[1471] System Overview

[1472] This system is primarily operated through devices such as smartphones and tablets. The devices have the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on them. It also has a function to analyze the user's emotional state using an emotion engine and adjust the menu and nutritional suggestions as described below.

[1473] Specific Embodiments of the System

[1474] 1. Take a photo of the ingredients and send it

[1475] A user takes a photo of the food in the refrigerator using a device such as a smartphone. The image is then sent to a server via an application. The device compresses the image data and processes it to ensure fast transmission.

[1476] 2. Image Analysis

[1477] The server analyzes the received image. An image recognition algorithm is used for the analysis, and ingredients are automatically identified. For example, ingredients such as chicken, tomatoes, and spices are identified. The algorithm for analyzing the image uses technologies such as YOLO (You Only Look Once) and TensorFlow.

[1478] 3. Menu generation

[1479] The server then searches the database for the most suitable recipe based on the identified ingredients. It generates the optimal menu, taking into account the user's nutritional needs and the expiration dates of the ingredients. For example, it might suggest a menu item such as "chicken steak with tomato salsa."

[1480] 4. Analysis and presentation of nutritional value

[1481] The server analyzes the nutritional value of the generated menu. The analysis uses the nutritional information of each ingredient registered in the database. The amounts of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified. This nutritional information is then graphed and displayed visually to the user.

[1482] 5. Suggestions for nutrient deficiencies

[1483] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For those nutrients, it suggests alternative foods or additional ingredients. For example, if you are deficient in vitamin D, it suggests eating mushrooms or fish.

[1484] 6. Menu and nutrition information display

[1485] The device receives the menu and nutritional information from the server and displays it to the user, who can then practice a balanced diet based on the visually displayed graphs and list-style information.

[1486] 7. Voice question function

[1487] Users can use voice input to ask questions or get more detailed information. For example, if they ask, "How do I cook this dish?", the server will analyze the voice data and provide the appropriate cooking method. The answer will be played back as text or audio.

[1488] 8. Calendar Management

[1489] The device has a function to manage daily menus and nutritional intake status in a calendar format. Users can view past data and future plans and manage their daily eating habits. This allows for long-term nutritional management.

[1490] 9. Incorporating an Emotional Engine

[1491] The server incorporates an emotion engine to recognize emotions from the user's facial expressions and voice. This emotional data is reflected in meal suggestions. For example, if the user is feeling stressed, the server can suggest ingredients and menus that have a relaxing effect.

[1492] 10. Emotional Adjustment

[1493] The server analyzes the accumulated emotional data and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake. For example, if a particular ingredient or menu improves the user's mood, personalized suggestions will be made based on that information.

[1494] Specific Examples

[1495] For example, suppose a user takes a photo of their refrigerator and sees that it contains "chicken, tomatoes, and spices." The server analyzes the photo and identifies these ingredients. The server then suggests a menu item such as "chicken steak with tomato salsa," and also considers the user's emotional state to suggest adding ingredients that have a relaxing effect. The server also analyzes and visualizes nutritional values ​​(calories, protein, vitamin C, etc.), suggesting adding mushrooms or fish to compensate for vitamin D deficiency. Users can check this information on their device and use it to plan their daily meals.

[1496] The above is an embodiment of the present invention. This system allows users to effectively use ingredients and efficiently manage nutrition, and also allows users to receive personalized suggestions based on their emotions.

[1497] The processing flow will be explained below.

[1498] Step 1:

[1499] When a user launches the smartphone application and takes a photo of the ingredients in the refrigerator, the device activates its camera function and displays a screen on which the user can take a photo of the ingredients.

[1500] Step 2:

[1501] The user checks the photo they have taken and presses the "Send" button in the application. The device sends the photo data to the server. When sending, the device compresses the image data to ensure it can be sent quickly.

[1502] Step 3:

[1503] The server receives the transmitted photo and runs an image recognition algorithm to analyze the received image data.

[1504] Step 4:

[1505] The server uses image recognition algorithms to identify ingredients in the received image, and lists the ingredients identified, for example, "chicken, tomatoes, and spices."

[1506] Step 5:

[1507] The server searches the database for the best recipe based on the identified ingredients, taking into account the user's nutritional needs (e.g., calorie restriction, vitamin intake, etc.) and expiration dates of the ingredients.

[1508] Step 6:

[1509] The server generates an optimal menu. For example, it might suggest "chicken steak with tomato salsa." Details of the suggested menu are stored in a database.

[1510] Step 7:

[1511] The server analyzes the nutritional value of the generated menu. The nutritional information includes calories, protein, fat, vitamins, minerals, etc. This information is obtained from a database.

[1512] Step 8:

[1513] The server quantifies the amount of each nutrient and calculates the total amount, then graphs the calculated nutritional information to generate data that can be displayed visually.

[1514] Step 9:

[1515] The server compares the analysis results with the daily required nutrients and identifies any nutrient deficiencies. For example, if there is a vitamin D deficiency, it will recommend additional foods and ingredients to supplement it.

[1516] Step 10:

[1517] The server transmits information including incomplete nutritional suggestions to the terminal, including menu details and nutritional value information.

[1518] Step 11:

[1519] The terminal displays the received information to the user in graph and list format so that the user can easily understand it.

[1520] Step 12:

[1521] Users can ask questions or get more information by speaking, for example, "How do I prepare this dish?"

[1522] Step 13:

[1523] The device converts the voice input into text and sends the text data to the server.

[1524] Step 14:

[1525] The server analyzes the question and generates an appropriate answer, such as "How to cook chicken steak..." and provides specific instructions.

[1526] Step 15:

[1527] The server generates a response and sends it to the terminal, which displays or plays it aloud to the user.

[1528] Step 16:

[1529] The device manages daily menus and nutritional intake in a calendar format, allowing users to check the calendar and view past data and upcoming plans.

[1530] Step 17:

[1531] The device activates an emotion engine that recognizes emotions from the user's facial expressions and voice. The user's emotional state is collected through the application. For example, facial expression analysis can determine whether the user is smiling, and voice analysis can determine whether the user is under stress.

[1532] Step 18:

[1533] The server analyzes the collected emotional data using an emotion engine. Based on the analysis results, it adjusts menu and nutrition suggestions based on the user's emotions. For example, if the user is feeling stressed, it will suggest ingredients and menus that have a relaxing effect.

[1534] Step 19:

[1535] The server analyzes the accumulated emotional data over the long term and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake. For example, if a particular ingredient or menu improves the user's mood, personalized suggestions will be made based on that information.

[1536] In this way, by performing specific actions at each step, the system efficiently supports food inventory management and nutritional management.In addition, personalized suggestions that take into account the user's emotional state can provide a more satisfying service.

[1537] Example 2

[1538] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1539] Conventional menu suggestion systems have difficulty effectively utilizing ingredients in the refrigerator and providing balanced nutritional management. Furthermore, they are unable to provide personalized suggestions that take the user's emotional state into account, resulting in low satisfaction. Furthermore, they lack the ability to provide appropriate answers to voice questions from users and to manage past nutritional intake history. By resolving these issues, it is necessary to improve the quality and satisfaction of users' dietary habits.

[1540] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes: means for a user to take an image of ingredients; means for compressing the taken image at a terminal and sending it to the server; means for the server to analyze the received image and identify ingredients; means for generating a menu based on the identified ingredients, taking into consideration the user's nutritional needs and the expiration dates of the ingredients; means for analyzing the nutritional value of the generated menu and quantifying and graphing calorie and nutrient values; means for identifying missing nutrients based on the analysis results and suggesting substitute foods and additional ingredients; means for displaying information about the menu and nutritional value on a user terminal; means for the server to provide appropriate information in response to questions from the user via voice input, and for the user terminal to provide answers in text or voice; means for managing daily menus and nutritional intake status in a calendar format and providing the user with past and future information; means for analyzing emotions from the user's facial expressions and voice, and personalizing meal suggestions using emotional data; and means for suggesting ingredients and menus with a relaxing effect according to a specific emotional state, and optimizing individual meal suggestions by analyzing the relationship between past emotions and nutritional intake. This will allow for effective use of ingredients in the refrigerator, optimizing the user's nutritional management, and enabling personalized meal suggestions based on the user's emotional state.

[1541] "User" refers to a person who uses the system to take pictures of ingredients and receive menu suggestions and nutritional management.

[1542] "Terminal" refers to an electronic device such as a smartphone or tablet that takes and sends images of ingredients, communicates with the server, and displays information.

[1543] The "server" refers to a remote data processing device that analyzes images of ingredients, generates menus, analyzes nutritional values, suggests nutrient deficiencies, and so on.

[1544] "Image analysis" refers to the technical process of analyzing received image data of ingredients to automatically identify the ingredients.

[1545] "Ingredient identification" refers to the process of identifying and listing types of ingredients through image analysis.

[1546] "Menu generation" refers to the process of determining the optimal menu based on the identified ingredients, taking into account the user's nutritional needs and the expiration dates of the ingredients.

[1547] "Nutritional analysis" refers to the process of calculating the nutritional information of the generated menu and quantifying and graphing the calories and amount of each nutrient.

[1548] "Nutrient deficiency suggestion" refers to the process of identifying missing nutrients based on analyzed nutritional value data and suggesting appropriate substitute foods or additional ingredients.

[1549] "Voice input" refers to a means by which a user uses voice to ask questions or give instructions to a system.

[1550] "Emotion analysis" refers to the process of recognizing emotions from a user's facial expressions and voice and analyzing that data.

[1551] "Personalization" refers to providing information and services that are individually optimized based on a user's specific conditions and tendencies.

[1552] "Calendar management" refers to the process of recording daily menus and nutritional intake status in calendar format and providing users with past data and future schedules.

[1553] The present invention relates to a system that takes images of ingredients in a refrigerator, analyzes the images, creates optimal menus, and manages nutrition. Furthermore, the present invention can improve user satisfaction and health management by recognizing the user's emotions and providing personalized meal suggestions based on those emotions.

[1554] System Overview

[1555] This system is primarily operated through devices such as smartphones and tablets. The devices have the ability to take photos of ingredients and send them to a server. The server analyzes the received images to identify the ingredients and generate a menu based on that information. It also has a function to analyze the user's emotional state using an emotion engine and adjust the menu and nutritional suggestions accordingly.

[1556] Hardware and software used

[1557] This system uses the following hardware and software:

[1558] Devices: smartphones, tablets

[1559] Server: High-performance data processing server

[1560] Image analysis algorithm: YOLO (You Only Look Once), TensorFlow

[1561] Emotion analysis engine: speech recognition software, facial recognition software

[1562] System operation flow

[1563] The system works as follows:

[1564] 1. Take a photo of the ingredients and send it

[1565] The user uses the camera on their smartphone or tablet to take a photo of the food in their refrigerator.

[1566] The device compresses the captured images and quickly transmits the data to the server.

[1567] 2. Image Analysis

[1568] The server decodes the received image and converts it into an analyzable format.

[1569] The server uses image recognition algorithms such as YOLO and TensorFlow to automatically identify ingredients in the image.

[1570] For example, ingredients such as chicken, tomatoes, and spices are identified.

[1571] 3. Menu generation

[1572] The server searches the database for the most suitable recipe based on the identified ingredient list.

[1573] The server generates optimal menus taking into account the user's nutritional needs and the expiration dates of ingredients.

[1574] For example, a menu suggestion might be "chicken steak with tomato salsa."

[1575] 4. Analysis and presentation of nutritional value

[1576] The server analyzes the nutritional value of the generated menu, using the nutritional information of each ingredient registered in the database.

[1577] The amount of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified.

[1578] This nutritional information is graphed and displayed visually to the user.

[1579] 5. Suggestions for nutrient deficiencies

[1580] Based on the analysis results, the server compares the nutrients needed per day with the nutritional value of the generated menu and identifies any nutrients that are lacking.

[1581] For those who are lacking in nutrients, we suggest alternative foods or additional ingredients. For example, if you are lacking in vitamin D, we suggest eating mushrooms or fish.

[1582] 6. Menu and nutrition information display

[1583] The terminal displays the menu and nutritional information received from the server to the user.

[1584] Users can practice balanced eating habits based on visually displayed graphs and list-style information.

[1585] 7. Voice question function

[1586] Users can use voice input to ask questions or for additional information.

[1587] For example, if you ask, "How do you cook this dish?", the server will analyze the voice data and provide the appropriate cooking method as an answer. The answer will be played back in text or audio.

[1588] 8. Calendar Management

[1589] The device has the function of managing daily menus and nutritional intake in calendar format.

[1590] Users can view past data and future plans and manage their daily eating habits.

[1591] 9. Incorporating an Emotional Engine

[1592] The server incorporates an emotion engine to recognize emotions from the user's facial expressions and voice.

[1593] Emotional data is reflected in meal suggestions. For example, if a user is feeling stressed, the system can suggest ingredients and menu items that have a relaxing effect.

[1594] 10. Emotional Adjustment

[1595] The server analyzes the accumulated emotional data and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake.

[1596] For example, if a particular ingredient or menu item improves a user's mood, personalized suggestions can be made based on that information.

[1597] Specific examples

[1598] For example, suppose a user takes a photo of their refrigerator, which includes "chicken, tomatoes, and spices." The device compresses this and sends it to the server. The server analyzes the photo and identifies these ingredients. The server then suggests a menu item such as "chicken steak with tomato salsa," and also considers the user's emotional state and suggests adding ingredients that have a relaxing effect. The server also analyzes and visualizes nutritional values ​​(calories, protein, vitamin C, etc.), suggesting adding mushrooms or fish to compensate for vitamin D deficiency. Users can check this information on their device and use it to plan their daily meals.

[1599] This invention allows users to make effective use of ingredients and efficiently manage their nutrition, and also allows users to receive personalized suggestions based on their emotions.

[1600] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1601] Step 1: Photograph the ingredients and send the image

[1602] The user uses a device (smartphone or tablet) to take a photo of the food in the refrigerator.

[1603] Input: Images of ingredients in the refrigerator.

[1604] The device compresses the images it takes, reducing the amount of data it takes to send them over the network.

[1605] The terminal transmits the compressed image data to the server.

[1606] Output: Compressed image data sent to the server.

[1607] Step 2: Image analysis and ingredient identification

[1608] The server decodes the received image data and converts it into an analyzable format.

[1609] Input: Received compressed image data.

[1610] The server uses image recognition algorithms such as YOLO (You Only Look Once) and TensorFlow to automatically identify ingredients.

[1611] Image analysis algorithms identify the type of ingredient (e.g., chicken, tomato, spices, etc.) from the image.

[1612] Output: A list of identified ingredients.

[1613] Step 3: Create a menu

[1614] The server searches the registered database for the most suitable recipe based on the identified ingredient list.

[1615] Input: Identified ingredient list, recipe information in database.

[1616] The server generates optimal menus taking into account the user's nutritional needs and the expiration dates of ingredients.

[1617] For example, a menu such as "chicken steak with tomato salsa" may be generated.

[1618] Output: The generated menu information.

[1619] Step 4: Nutritional analysis and presentation

[1620] The server analyzes the nutritional value of the generated menu.

[1621] Input: Generated menu information, nutrition information in the database.

[1622] The amount of nutrients such as total calories, protein, fat, vitamins, and minerals are quantified.

[1623] This nutritional information is graphed and formatted in a way that can be visually presented to the user.

[1624] Output: Nutritional values ​​are displayed in graph and list format.

[1625] Step 5: Suggesting nutrient deficiencies

[1626] The server analyzes the nutritional value of the menu and identifies any nutrients that are lacking by comparing them with the daily required nutrients.

[1627] Input: Quantified nutritional value data, reference values ​​for nutrients needed per day.

[1628] The server will identify any nutrient deficiencies and suggest substitutions or additional ingredients.

[1629] For example, if you are deficient in vitamin D, it is suggested that you "consume additional mushrooms and fish."

[1630] Output: Suggested information on nutrient deficiencies.

[1631] Step 6: Display menu and nutrition information

[1632] The server sends the generated menu and nutritional information to the terminal.

[1633] Input: Menu information, nutritional information.

[1634] The information received by the terminal is visually displayed to the user.

[1635] The device displays the information in graph and list format, and users can use the information to practice a balanced diet.

[1636] Output: The visual information that is displayed to the user.

[1637] Step 7: Voice Question Function

[1638] Users use voice input to ask questions or for additional information.

[1639] Input: Audio data.

[1640] The device converts the voice data into text and sends it to the server.

[1641] The server analyzes the question and generates an appropriate answer.

[1642] The device will present the answer to the user in text or audio.

[1643] For example, if you ask, "How do you cook this dish?", the cooking instructions will be answered in text or voice.

[1644] Output: The answer information provided to the user.

[1645] Step 8: Calendar Management

[1646] The device manages daily menus and nutritional intake in calendar format.

[1647] Input: Menu information, nutritional intake data.

[1648] Users can view past data and future schedules.

[1649] The terminal displays information in calendar format, enabling long-term nutritional management.

[1650] Output: Information displayed in a calendar format.

[1651] Step 9: Incorporating the Emotion Engine

[1652] The server uses an emotion engine to recognize emotions from the user's facial expressions and voice data.

[1653] Input: User's facial expression data, voice data.

[1654] A sentiment analysis engine analyzes this data and infers the user's emotional state (e.g., stress, happiness, etc.).

[1655] Output: Emotional state data.

[1656] Step 10: Emotional Adjustment

[1657] The server analyzes the accumulated emotional data and optimizes individual meal suggestions by taking into account the relationship between past emotional states and nutritional intake.

[1658] Input: Emotional state data, nutritional intake data.

[1659] Depending on your specific emotional state, it will suggest foods and menus that have a relaxing effect.

[1660] For example, if a user is feeling stressed, a menu containing ingredients that have a relaxing effect will be suggested based on that information.

[1661] Output: Personalized meal suggestion information.

[1662] (Application example 2)

[1663] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1664] Conventional food management systems have limitations in efficiently managing ingredients in the refrigerator and suggesting appropriate menus. Furthermore, they make uniform suggestions without considering the user's emotions, resulting in insufficient personalization to meet individual needs. Furthermore, there has been a lack of systems that can utilize new devices such as smart glasses to recognize ingredients and emotions in real time.

[1665] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a means for capturing images of ingredients, a means for transmitting the captured images to the server, a means for the server to analyze the images and identify ingredients, a means for generating a menu based on the identified ingredients, a means for analyzing, graphing, and quantifying the nutritional value of the generated menu, a means for presenting missing nutrients based on the analysis results, a means for displaying the menu and nutritional value information on a user terminal, a means for recognizing the user's emotions and reflecting them in meal suggestions, and a means for using smart glasses to recognize ingredients and emotions in real time and suggest a menu. This enables efficient recognition of ingredients in the user's refrigerator and personalized menu suggestions based on the user's individual emotional state. Furthermore, the use of smart glasses enables real-time ingredient recognition and emotion recognition, enabling more immediate and accurate information provision.

[1666] "Means for taking images of ingredients" refers to functionality that includes devices and software for capturing images of ingredients in a refrigerator or other storage location.

[1667] The "means for transmitting the captured image to the server" is a function for transmitting the image captured from the smart glasses or other terminal to the server via data communication.

[1668] The "means for the server to analyze the image and identify ingredients" refers to a system that uses algorithms or techniques to analyze received image data and identify ingredients.

[1669] The "means for generating a menu based on the identified ingredients" is a function that searches a database for the most suitable recipe based on the identified ingredients and suggests a menu.

[1670] The "means for analyzing, graphing and quantifying the nutritional value of the generated menu" is a function for quantifying each nutrient in the generated menu and visually displaying it.

[1671] The "means for presenting missing nutrients based on the analysis results" is a function for identifying missing nutrients by comparing them with the calculated nutritional values ​​and suggesting appropriate ingredients and supplements.

[1672] The "means for displaying the menu and nutritional value information on a user terminal" is a function for displaying the generated menu and its nutritional value information on a user terminal such as smart glasses or a smartphone.

[1673] "Means for recognizing the user's emotions and reflecting them in meal suggestions" is a function that analyzes the user's facial expressions and voice to identify their emotional state and then makes personalized meal suggestions based on that information.

[1674] "Means for using smart glasses to recognize ingredients and emotions in real time and suggest menus" is a function that uses the cameras and sensors in smart glasses to recognize ingredients and the user's emotions in real time and suggests menus based on that information.

[1675] The present invention provides a system for efficiently managing ingredients in a refrigerator and proposing personalized menus based on the user's emotions. Here, specific embodiments of the present invention will be described.

[1676] System Overview

[1677] The main components of this system are smart glasses (or a user terminal such as a smartphone), a server, and software for recognizing images of ingredients in the refrigerator and the user's emotions.

[1678] 1. Food and emotion recognition

[1679] A user uses smart glasses to check the ingredients in the refrigerator. At this time, the camera in the smart glasses captures an image of the ingredients and sends the image data to the server. At the same time, the sensors in the smart glasses capture the user's facial expressions and voice to recognize their emotions. For example, if a user opens the refrigerator and says, "I want to relax today," that voice data is also sent to the server.

[1680] 2. Image analysis and emotion analysis on the server

[1681] The server first analyzes the received image data, using the YOLO (You Only Look Once) model and TensorFlow algorithms to automatically identify the ingredients in the refrigerator.

[1682] The received facial and voice data is then analyzed using an emotion engine, which is used to identify the user's current emotional state (e.g., relaxed, stressed, depressed, etc.).

[1683] 3. Menu Creation and Nutritional Analysis

[1684] Based on the identified ingredients and the user's emotional state, the server searches for the most suitable recipe from the database and generates a menu. For example, if the ingredients are "chicken, tomato, and spices" and the user is recognized as "feeling stressed," the server will suggest "chicken steak with tomato salsa" with "herbal tea," which has a relaxing effect, as a side dish.

[1685] To analyze the nutritional value of the generated menu, the system refers to a nutrition database and converts information such as total calories, protein, fat, vitamins, and minerals into numerical values ​​and graphs, which the user can visually check.

[1686] 4. Display on the user's device

[1687] The generated menu information and nutritional information are displayed on the user's device, such as smart glasses or a smartphone, allowing the user to efficiently use the ingredients in their refrigerator and practice nutritionally balanced meals.

[1688] 5. Examples of prompts

[1689] Specific examples of prompts include:

[1690] "It recognizes the ingredients in the refrigerator and suggests personalized meals based on the user's emotions. For example, if it analyzes an image containing "chicken, tomatoes, and spices" and determines that the user is feeling stressed, it will suggest a meal that includes ingredients that have a relaxing effect."

[1691] This system allows users to make the most of the ingredients in their refrigerator and receive personalized meal suggestions based on their emotions. It also recognizes ingredients and emotions in real time and provides appropriate information instantly.

[1692] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1693] Step 1:

[1694] A user uses smart glasses to check the ingredients in the refrigerator and captures the images with the camera. The input is the image of the ingredients in the refrigerator, and the output is the captured image data.

[1695] Step 2:

[1696] The smart glasses transmit the captured image data to a server. The input is the captured image data, and the output is the transmission of the image data to the server. In actual operation, the smart glasses use a communication module to compress the image data and transmit it to the server via the Internet.

[1697] Step 3:

[1698] The server analyzes the received image data and identifies the ingredients. The input is the image data sent to the server, and the output is a list of identified ingredients. Specifically, the server uses the YOLO model to detect ingredients in the image and create a list of ingredient names.

[1699] Step 4:

[1700] The smart glasses capture the user's facial expressions and voice and send the data to the server. The input is the user's facial expressions and voice data, and the output is data transmission to the server. The smart glasses' sensors and microphone are used to capture the user's facial expressions and voice in real time and send the data to the server.

[1701] Step 5:

[1702] The server analyzes the received facial and voice data to recognize the user's emotions. The input is the facial and voice data sent to the server, and the output is the recognized emotion data. Specifically, the emotion engine is used to analyze the facial and voice data and identify the user's emotional state.

[1703] Step 6:

[1704] The server generates an optimal menu based on the identified ingredients and the recognized emotion data. The input is the identified ingredient list and emotion data, and the output is the generated menu. The server searches for appropriate recipes from the database and selects meals that suit the user's emotions.

[1705] Step 7:

[1706] The server analyzes the nutritional value of the generated menu and graphs and quantifies it. The input is the generated menu, and the output is the nutritional value and graph. The server refers to a nutrition database, tallying the nutritional information of each ingredient and calculating the nutritional value.

[1707] Step 8:

[1708] The server then presents the nutritional deficiencies based on the analysis results. The input is the nutritional analysis results, and the output is suggestions regarding the nutrients that are lacking. It identifies the nutrients that are lacking and suggests additional ingredients or supplements to make up for them.

[1709] Step 9:

[1710] The generated menu and nutritional information are displayed on the user's device. The input is the generated menu and nutritional information, and the output is the display on the user's device. The menu and nutritional information are displayed on smart glasses or a smartphone, allowing the user to visually confirm them.

[1711] Step 10:

[1712] The user asks a question by voice input, and the server provides the corresponding information. The input is the user's voice question, and the output is the answer from the server. The server uses a voice recognition system to analyze the user's question and provide appropriate information and explanations.

[1713] Step 11:

[1714] The user terminal manages daily menus and nutritional intake status in a calendar format, providing users with past and future information. The input is past and current menu data, and the output is a visual display in calendar format. Users can manage their nutrition over the long term.

[1715] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1716] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1717] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1718] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1719] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1720] These emotions are distributed in the 3 o'clock direction on emotion map 40...

Claims

1. a means for taking an image of the food ingredient; means for transmitting the captured image to a server; means for the server to analyze the image and identify ingredients; A means for generating a menu based on the identified ingredients; A means for analyzing, graphing, and quantifying the nutritional value of the generated menu; A means for indicating nutrients that are lacking based on the analysis results; a means for displaying the menu and nutritional value information on a user terminal; A system including:

2. 2. The system according to claim 1, further comprising means for the server to provide information corresponding to a question from the user via voice input.

3. 2. The system according to claim 1, further comprising means for managing daily menus and nutritional intake status in a calendar format and providing the user with past and future information.

Images