system

Abstract

Provide a system. A means for receiving a video and recognizing an object in the video; means for generating a plurality of combinations based on the recognized items; a means of identifying the missing items; means for ordering the missing items from a predetermined source; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, busy daily lives have made it difficult to prepare nutritionally balanced meals, especially for those in their prime working years and those with families. Even if they have the time, the effort required to plan a menu and go shopping for ingredients can result in meals that are nutritionally unbalanced. Due to the impact of the pandemic, the importance of a healthy diet to boost immunity has increased, but there is a problem of not being able to prepare an appropriate menu due to a lack of time. Furthermore, when there is a lack of time to cook, or when time and energy are limited due to work fatigue, childcare, or elderly care, people tend to eat out or rely on convenient foods. A method is needed to efficiently solve these problems and provide nutritionally balanced meals. [Means for solving the problem]

[0005] In order to solve the above problems, the present invention proposes the following means.

[0006] means for receiving the video and recognizing the object in the video;

[0007] means for generating a plurality of combinations based on the recognized items;

[0008] a means of identifying the missing items;

[0009] means for ordering the missing items from a predetermined source;

[0010] The system includes means for acquiring a scheduled delivery time of the item from a predetermined supply source and matching it with the time the user will be home.

[0011] Furthermore, we propose a system that includes a means for retrieving cooked meals from designated providers and delivering them to users when they are unable to find time to cook. This allows users to automatically receive nutritionally balanced meals by simply taking a video of the ingredients in their refrigerator or storage. Missing ingredients and seasonings are ordered through a local online supermarket and delivered in time for the user's return home, eliminating the need for grocery shopping. Furthermore, if users are unable to find time to cook, they can have cooked meals delivered from a local diner or restaurant. This allows users to provide healthy meals to their families while still having the time to spare.

[0012] "Video" refers to moving image data captured on devices such as smartphones and tablets.

[0013] "Receiving" refers to receiving data transmitted via a communication means.

[0014] "Items" refers to ingredients and seasonings stored in a refrigerator or storage room.

[0015] "Recognizing" refers to using AI-generated models and image analysis to identify objects in a video.

[0016] "Combination" refers to a meal consisting of a staple food and three side dishes (soup, main dish, and garnish) that takes nutritional balance into consideration.

[0017] "Missing items" refer to ingredients or seasonings that are needed to realize the generated combination but are not currently in storage.

[0018] "Predetermined sources" refers to local online supermarkets and food supply services.

[0019] "Placing an order" refers to ordering the missing item from a predetermined source.

[0020] "Scheduled Delivery Time" refers to the time an item is scheduled to be delivered from a given source.

[0021] "Return home time" refers to the time when the user plans to return home.

[0022] "Prepared meals" refers to meals prepared by local diner or restaurant.

[0023] "Specified provider" refers to a food service provider such as a diner or restaurant.

[0024] The term "system" refers to a series of devices and software that combine the above means to provide users with nutritionally balanced meals. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0026] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0027] First, the terms used in the following description will be explained.

[0028] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0029] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0030] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0031] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0033] [First embodiment]

[0034] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0035] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0036] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0038] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0041] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0043] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0044] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0045] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0046] This invention is a system that allows users to take videos of ingredients in their refrigerators or storage areas, analyzes the videos, and automatically creates nutritionally balanced menus. This system operates in cooperation with a server, terminals, and users to provide convenience and health to users.

[0047] First, the user uses a device such as a smartphone or tablet to record video of the inside of the refrigerator or storage unit. A dedicated app is installed on the device, and the video is sent to the server using this app. The video is sent via a communication line.

[0048] The server has a program that analyzes the received video. The server identifies and lists the items (ingredients and seasonings) in the video. Based on this list, the server generates multiple menus that take nutritional balance into consideration. Specifically, it generates combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also added to each menu.

[0049] The server then sends the generated menu information to the terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to select. The user selects the appropriate menu from the multiple menus presented and checks its details. If there are any ingredients or seasonings required for the menu that are missing from the refrigerator or storage, the server identifies them and lists them.

[0050] With the user's permission, the server orders missing ingredients and seasonings from a designated source (such as a local online supermarket). When the order is placed, it schedules delivery to coincide with the user's return home time. When the user returns home, the necessary ingredients have been delivered in a timely manner.

[0051] In addition, if the user does not have time to cook or is too tired from work to cook, the server will arrange for pre-cooked meals to be delivered from affiliated restaurants, allowing the user to provide healthy meals for themselves and their family.

[0052] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via the app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Any "miso" that is missing is listed and ordered from the online supermarket in time for the user to return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0053] As described above, the system of the present invention provides comprehensive support for users to efficiently prepare or obtain nutritionally balanced meals, allowing even busy users to easily achieve healthy meals.

[0054] The processing flow will be explained below.

[0055] Step 1:

[0056] Users take videos of the ingredients in their refrigerators or storage areas using a smartphone or tablet, adjusting the camera angle and brightness to ensure all ingredients are clearly visible.

[0057] Step 2:

[0058] The device sends the captured video to a server via a dedicated app, where it performs correction processing to appropriately compress the video data while maintaining image quality.

[0059] Step 3:

[0060] The server receives the video sent from the device and stores it in a database, where it is processed by an analysis program.

[0061] Step 4:

[0062] The server uses an AI-generated model to identify ingredients in the video, detecting items (ingredients and seasonings) in each frame and listing them.

[0063] Step 5:

[0064] Based on the listed ingredient information, the server retrieves nutritional information for each ingredient and available recipes from the database.

[0065] Step 6:

[0066] The server runs a menu generation algorithm that takes nutritional balance into consideration based on the acquired ingredient information and recipe data, generating five combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish).

[0067] Step 7:

[0068] The server adds cooking time and calorie information to each menu item and also generates a detailed list of the ingredients and seasonings required.

[0069] Step 8:

[0070] The server sends the generated menu information to the user's device using a secure communication method.

[0071] Step 9:

[0072] The device displays the menu information received from the server in a dedicated app. The user can check multiple menu options and view detailed information (cooking time, calories, and a list of necessary ingredients).

[0073] Step 10:

[0074] The user selects an appropriate menu from the presented menu options, checks the details, and then lists any missing ingredients and seasonings. The user then authorizes the ordering of missing ingredients within the app.

[0075] Step 11:

[0076] With the user's permission, the server automatically orders the listed ingredients from a designated supplier (such as a local online supermarket) and schedules delivery to coincide with the user's return home time.

[0077] Step 12:

[0078] The server obtains the estimated delivery time of the ordered ingredients from the supply source and notifies the user.

[0079] Step 13:

[0080] When the user returns home, the user checks that the necessary ingredients have been delivered and begins cooking based on the presented menu.

[0081] Step 14:

[0082] If the user cannot find time to cook, the user notifies the server that he / she would like to have cooked food delivered from a partner restaurant or diner.

[0083] Step 15:

[0084] The server receives the user's notification, obtains prepared food from affiliated restaurants and other establishments in the area, and arranges for delivery at the time specified by the user.

[0085] Through the above specific processing steps, the present invention helps users to efficiently prepare nutritionally balanced meals.

[0086] Example 1

[0087] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0088] In today's busy lifestyles, it is difficult for users to easily decide on nutritionally balanced meals and procure the necessary ingredients in a timely manner. In particular, users who are busy with work or housework cannot afford to take the time to identify shortages of ingredients and replenish them, making it difficult to prepare healthy meals. Furthermore, if they cannot find time to cook, they have to go through the hassle of arranging pre-cooked meals as an alternative.

[0089] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0090] In this invention, the server includes a means for receiving a video and recognizing items in the video, a means for generating multiple combinations based on the recognized items, and a means for identifying missing items. It also includes a means for automatically analyzing each frame of the captured video and identifying items, a means for automatically generating a nutritionally balanced menu based on the analyzed list of items, and a means for displaying the generated menu information and allowing the user to select from it. This allows the user to easily decide on a nutritionally balanced menu and procure the necessary ingredients in a timely manner. Furthermore, if cooking time is unavailable, pre-cooked meals can be quickly arranged.

[0091] "Means for receiving video and recognizing objects in the video" refers to a technology that sends a video taken by a user to a server and automatically recognizes objects that appear in the video through analysis.

[0092] The "means for generating a plurality of combinations based on recognized items" is a technology for automatically generating a plurality of nutritionally balanced menus based on a list of recognized items.

[0093] The "means for identifying missing items" is a technology for automatically finding items required for the generated menu that are not in the refrigerator or storage.

[0094] "Means for ordering from a specified source of supply" refers to technology that automatically places an order for the missing items with a specific supplier (for example, an online supermarket or a retail store).

[0095] "Means for analyzing each frame and automatically identifying items" refers to a technology for breaking down a video into individual frames and analyzing the items shown in each frame.

[0096] The "means for automatically generating menus that take nutritional balance into consideration" is a technology for generating multiple menus that take nutritional balance into consideration based on an analyzed item list.

[0097] "Means for displaying menu information and allowing the user to select" refers to technology that displays automatically generated menu information on the user's terminal and allows the user to select their preferred menu from the information.

[0098] This invention is a system that allows users to take videos of ingredients in their refrigerators or storage areas, analyzes the videos, and automatically creates nutritionally balanced menus. This system operates in cooperation with a server, terminals, and users to provide convenience and health to users.

[0099] First, the user uses a device such as a smartphone or tablet to take a video of the inside of the refrigerator or storage unit. A dedicated app is installed on the device, and the taken video is sent to the server using this app. The video is sent via a communication line. For example, the user takes a video of the inside of the refrigerator and sends it to the server via the dedicated app.

[0100] The server has a program that analyzes the received video. Using a video analysis program (e.g., Azure® Video Analyzer or Google® Cloud Video Intelligence), it analyzes the captured video frame by frame and automatically identifies items (ingredients and seasonings) in the video. Based on the analyzed list of items, the server generates a menu that takes nutritional balance into consideration. Specifically, it uses a generative AI model (e.g., OpenAI® GPT-4®) to combine a staple food (rice, bread, noodles) with three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also included with each menu.

[0101] Next, the server sends the generated menu information to the terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to make a selection. The user selects the appropriate menu from the multiple menus presented and checks its details. For example, the terminal may display "Main dish: rice, Main: teriyaki chicken thigh, Soup: miso soup, Side dish: stir-fried cabbage and carrots," with each menu item provided along with calorie information and cooking time.

[0102] If there are any ingredients or seasonings needed for a meal that are missing from the refrigerator or storage, the server will identify them and list them. With the user's permission, the server will order the missing ingredients or seasonings from a designated source (such as a local online supermarket). When ordering, the server will schedule delivery to coincide with the user's return home time. For example, the server will list the missing "miso" and order it from the online supermarket to coincide with the user's return home time.

[0103] In addition, if the user cannot find time to cook or is too tired from work to cook, the server will arrange for pre-cooked meals to be delivered from affiliated restaurants. The user can notify the app that they are having difficulty cooking and select pre-cooked meals as an alternative. For example, if the user inputs into the app, "I'm tired today, so I'd like pre-cooked meals delivered," the server will arrange for the above menu to be delivered from an affiliated restaurant.

[0104] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via a dedicated app. The server analyzes the received video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Any "miso" that is missing is listed and ordered from the online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0105] In this way, the system of the present invention provides comprehensive support for users to efficiently prepare or obtain nutritionally balanced meals, allowing even busy users to easily achieve healthy meals.

[0106] Example prompt sentence:

[0107] "Create a system that allows users to take a video of the ingredients in their refrigerator, analyzes the video, and automatically generates a nutritionally balanced meal plan."

[0108] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0109] Step 1: Record and send video

[0110] Subject: User

[0111] Users can launch a dedicated app on their smartphone or tablet and take video of the inside of their refrigerator or storage unit. Once the video is complete, the user can send the video to the server via the app.

[0112] Input: Video data taken inside a refrigerator or storage facility

[0113] Output: Video data sent to the server

[0114] Specific operation: The user opens the refrigerator door, takes sequential photos of the inside with their smartphone camera, and taps the "Send" button to upload the video data to the server.

[0115] Step 2: Receiving and analyzing the video

[0116] Subject: Server

[0117] The server receives the video sent by the user and uses a video analysis program to analyze the video frame by frame and identify the items (ingredients and seasonings) that appear in the video.

[0118] Input: Video data sent by the user

[0119] Output: List of recognized objects

[0120] How it works: The server runs a video analysis program, breaks down the video into frames, and automatically identifies and lists the ingredients and seasonings. For example, a list of items such as "chicken thighs, cabbage, and carrots" is obtained.

[0121] Step 3: Menu Creation

[0122] Subject: Server

[0123] The server generates a menu that takes nutritional balance into consideration based on the list of items obtained through the analysis. Using a generative AI model, multiple menu plans are created. Each menu includes a main dish and three side dishes (soup, main course, and garnish), and cooking time and calorie information are also included.

[0124] Input: A list of recognized items (e.g., "chicken thigh, cabbage, carrot")

[0125] Output: Multiple menu plans that take nutritional balance into consideration

[0126] Specific operation: The server runs the AI ​​model and generates a menu suggestion such as "rice, teriyaki chicken thighs, miso soup, and stir-fried cabbage and carrots."

[0127] Step 4: Display menu information

[0128] Subject: Terminal

[0129] The terminal receives the menu information sent from the server and displays it on the screen in an easy-to-read format. The user can then select an appropriate menu from multiple suggested menus.

[0130] Input: Menu information sent from the server

[0131] Output: Menu information displayed on the device

[0132] Specific operation: The device displays menu suggestions such as "Main dish: rice, Main course: teriyaki chicken thigh, Soup: miso soup, Side dish: stir-fried cabbage and carrots," and the user can choose from them.

[0133] Step 5: Identify and order missing ingredients

[0134] Subject: Server

[0135] The server automatically lists any missing items based on the selected menu, orders missing ingredients and seasonings from a designated supplier (such as an online supermarket), and schedules delivery to coincide with the user's return time.

[0136] Input: Selected menu plan, user's current ingredients list

[0137] Output: List of ingredients to order, order information

[0138] Specific operation: The server lists ingredients such as miso that are in short supply, and after receiving confirmation from the user, automatically places an order with the online supermarket. The delivery time is set to coincide with the user's return home.

[0139] Step 6: Arrange for prepared meals

[0140] Subject: Server

[0141] If the user cannot find time to cook, the server will arrange for the delivery of cooked food from affiliated restaurants or diner.

[0142] Input: User notification of cooking difficulty

[0143] Output: Arrange for delivery of cooked food

[0144] Specific operation: The user enters "I can't cook today" through the app, and the server arranges for the following meal from a partner restaurant: "Main dish: rice, Main course: teriyaki chicken thigh, Soup: miso soup, Side dish: stir-fried cabbage and carrots" and has it delivered.

[0145] (Application example 1)

[0146] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0147] In today's busy lifestyles, it is difficult for users to prepare healthy, nutritionally balanced meals. Managing the contents of the refrigerator and storage areas and planning menus based on that information is also a time-consuming task. Furthermore, if necessary ingredients are in short supply, it is inefficient to go shopping every time. To solve these problems, a system is needed that allows users to easily manage ingredients, create menus, and order missing ingredients.

[0148] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0149] In this invention, the server includes a means for receiving a video and recognizing items in the video, a means for generating a plurality of combinations based on the recognized items, and a means for identifying missing items. This automatically lists ingredients in the refrigerator or storage area from the video taken by the user, and generates a nutritionally balanced menu based on that list. Furthermore, the automatic identification of missing items allows online ordering from a virtual store, and delivery of ingredients or cooked meals can be arranged to coincide with the user's return home time. This allows users to prepare and obtain healthy meals without hassle.

[0150] The "means for receiving video and recognizing items in the video" is a technology for receiving video shot by a user and analyzing and identifying items that appear in the video.

[0151] The "means for generating multiple combinations based on recognized items" is a technology for creating multiple menus based on information about identified items, taking into consideration nutritional balance, cooking methods, etc.

[0152] The "means for identifying missing items" is a technology for comparing the recognized items with the items required for the generated menu and listing the missing items.

[0153] The "means for ordering the missing items from a predetermined supply source" is a technique for ordering the identified missing items from a related supply source (for example, an online supermarket or an affiliated store).

[0154] The "means for generating nutritionally balanced menus" is a technology for creating multiple balanced menus by taking into consideration the nutritional components of recognized items.

[0155] "Means for sending the generated menu information to the user terminal and displaying it so that the user can select" refers to a technology for sending the generated menu data to the user's device and displaying it so that the user can select the menu they desire.

[0156] "Means for placing online orders at a virtual store" refers to technology that allows users to easily place orders at a virtual store (e.g., an online shopping site) based on a list of items they are missing.

[0157] This system allows users to take videos of ingredients in their refrigerators or storage areas, analyzes the videos, and automatically creates nutritionally balanced meals. The system operates in cooperation with the server, terminals, and users to provide convenience and health to users.

[0158] First, the user uses a device such as a smartphone or tablet to record video of the inside of the refrigerator or storage unit. A dedicated application is installed on the device, and the video is sent to the server using this application. The video is sent via a communication line.

[0159] The server has a program that analyzes the received video. This analysis uses the OpenCV library for image analysis and TENSORFLOW (registered trademark) for ingredient recognition. The server identifies and lists the items (ingredients and seasonings) in the video. Based on this list, the server uses a generative AI model to generate multiple menus that take nutritional balance into consideration. Specifically, it generates combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also included with each menu.

[0160] The server then sends the generated menu information to the terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to select. The user selects the appropriate menu from the multiple menus presented and checks its details. If there are any ingredients or seasonings required for the menu that are missing from the refrigerator or storage, the server identifies them and lists them.

[0161] With the user's permission, the server orders missing ingredients and seasonings from a designated supply source (such as a local online supermarket). When the order is placed, it schedules delivery to coincide with the user's return time. This ensures that the necessary ingredients have arrived in a timely manner when the user returns home. In addition, if the user is unable to find time to cook or is too tired from work to cook, the server will arrange for pre-cooked food to be delivered from an affiliated restaurant.

[0162] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via the app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Any "miso" that is missing is listed and ordered from the online supermarket in time for the user to return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0163] An example of a prompt for a generative AI model is as follows:

[0164] "Analyze the video of the contents of the refrigerator and create a list of recognized ingredients. Furthermore, use that data to generate a nutritionally balanced meal plan, identifying and listing any missing ingredients. Display the generated meal plan in the following format: 'Main dish: Teriyaki chicken thigh, Side dish: Cabbage salad, Another side dish: Kinpira carrots'. If any ingredients are missing, provide a function that allows the user to order them from a virtual store."

[0165] The system allows users to prepare and consume healthy meals hassle-free.

[0166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0167] Step 1:

[0168] Users use devices such as smartphones or tablets to record video of the inside of refrigerators or storage units. A dedicated application is installed on the device, and the video recorded using this application is sent to a server. The input is video data recording the inside of the refrigerator or storage unit, and the output is the video data sent to the server. The device sends the video data via a communication line.

[0169] Step 2:

[0170] To analyze the received video, the server uses the OpenCV library to break down the video data into frames and perform image analysis on each frame. Based on the recognition results, TensorFlow is used to identify ingredients. The input is the image data broken down into frames, and the output is a list of recognized ingredients. Specifically, the server resizes the images of each frame to 224 x 224 pixels and inputs them into the ingredient recognition model.

[0171] Step 3:

[0172] The server uses a generative AI model to generate multiple nutritionally balanced menus based on the list of recognized ingredients. The input is the list of recognized ingredients, and the output is a list of generated menus. Specifically, it takes into account the combination of staple food, side dish, and soup, and also adds cooking time and calorie information to each menu.

[0173] Step 4:

[0174] The server sends the generated menu information to the user terminal. The input is a list of the generated menus, and the output is the menu information sent to the user terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to select an appropriate menu from it.

[0175] Step 5:

[0176] The user selects the appropriate menu from the multiple menus presented and checks its details. The input is the menu information sent from the server, and the output is the details of the menu selected by the user. The selected menu includes the necessary ingredients and seasonings.

[0177] Step 6:

[0178] The server identifies and lists the items that are missing from the refrigerator or storage based on the selected menu. The input is the selected menu and the current status of the refrigerator or storage, and the output is a list of the missing items.

[0179] Step 7:

[0180] With the user's permission, the server orders missing ingredients and seasonings from a designated source (e.g., a local online supermarket). The input is a list of missing items, and the output is order confirmation information. When placing an order, the server schedules delivery to coincide with the user's return home time.

[0181] Step 8:

[0182] If the user cannot find the time to cook, the server will arrange for the delivery of cooked food from a partner restaurant. The inputs are the user's cooking time and the partner restaurant's menu, and the output is confirmation of the delivery of the cooked food.

[0183] This allows users to prepare and consume healthy meals without any hassle.

[0184] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0185] This invention is a system that allows users to take videos of ingredients in their refrigerators and storage areas, analyzes the videos, and automatically generates nutritionally balanced menus. By combining this invention with an emotion engine that recognizes the user's emotions, it is possible to adjust the menu suggestions according to the user's emotional state. This allows for meal suggestions that are tailored to the user's mental state and preferences.

[0186] First, the user uses a device such as a smartphone or tablet to record video of the inside of the refrigerator or storage unit. A dedicated app is installed on the device, and the video is sent to the server using this app. The video is sent via a communication line.

[0187] The server has a program that analyzes the received video. The server identifies and lists the items (ingredients and seasonings) in the video. Based on this list, the server generates multiple menus that take nutritional balance into consideration. Specifically, it generates combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also added to each menu.

[0188] The server then analyzes the user's current emotional state using an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions based on, for example, the text the user types into the device, the tone of voice, or changes in facial expressions. This emotion information is used to adjust the menu suggestions. For example, if the user is feeling stressed, the server can prioritize easy-to-prepare menus.

[0189] The server optimizes the priority of the menu suggestions based on the generated menu information and emotion information and sends them to the device. The device then displays the received menu information in an easy-to-read format, allowing the user to select. The user selects an appropriate menu from the multiple menus presented and checks its details. If there are any ingredients or seasonings required for the menu that are missing from the refrigerator or storage, the server identifies them and lists them.

[0190] With the user's permission, the server orders missing ingredients and seasonings from a designated source (such as a local online supermarket). When the order is placed, it schedules delivery to coincide with the user's return home time. When the user returns home, the necessary ingredients have been delivered in a timely manner.

[0191] In addition, if the user does not have time to cook or is too tired from work to cook, the server will arrange for pre-cooked meals to be delivered from affiliated restaurants, allowing the user to provide healthy meals for themselves and their family.

[0192] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via the app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Furthermore, an emotion engine recognizes the user's stress level and prioritizes menu options that are easy to prepare. Any "miso" that is in short supply is listed and ordered from an online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0193] As described above, the system of the present invention provides comprehensive support for users to efficiently prepare or obtain nutritionally balanced meals, enabling users to easily and effectively achieve healthy eating habits even in busy daily lives.

[0194] The processing flow will be explained below.

[0195] Step 1:

[0196] Users take videos of the ingredients in their refrigerators or storage areas using a smartphone or tablet, adjusting the camera angle and brightness to ensure all ingredients are clearly visible.

[0197] Step 2:

[0198] The device sends the captured video to the server via a dedicated app. Before sending, the device encodes the video and compresses it appropriately.

[0199] Step 3:

[0200] The server receives the video sent from the device and stores it in a database. The received video is then processed by an analysis program.

[0201] Step 4:

[0202] The server uses an AI-generated model to identify ingredients in the video, detecting them in each frame and listing them.

[0203] Step 5:

[0204] Based on the listed ingredient information, the server retrieves nutritional information and available recipes for each ingredient from the database.

[0205] Step 6:

[0206] The server runs a menu generation algorithm that takes nutritional balance into consideration based on the acquired ingredient information and recipe data, generating five combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish).

[0207] Step 7:

[0208] The server adds cooking time and calorie information to each menu, and also generates a detailed list of ingredients and seasonings required.The server also uses an emotion engine to analyze the user's emotional state.

[0209] Step 8:

[0210] The server adjusts its menu suggestions based on the user's emotional state: for example, if the user is feeling stressed, it will prioritize easy-to-prepare meals.

[0211] Step 9:

[0212] The server then transmits the optimized menu information, taking into account the user's emotional state, to the user's device using secure communication methods.

[0213] Step 10:

[0214] The device displays the menu information received from the server in a dedicated app. The user can check multiple menu options and view detailed information (cooking time, calories, and a list of necessary ingredients).

[0215] Step 11:

[0216] The user selects an appropriate menu from the presented options and checks the details. After making a selection, the system lists any missing ingredients and seasonings.

[0217] Step 12:

[0218] The user can authorize the app to order missing ingredients by tapping the order authorization button.

[0219] Step 13:

[0220] With the user's permission, the server automatically orders the missing ingredients from a designated supply source (such as a local online supermarket). When placing an order, the server sets a delivery schedule to match the user's return time.

[0221] Step 14:

[0222] The server obtains the estimated delivery time of the ordered ingredients from the supply source and notifies the user.

[0223] Step 15:

[0224] When the user returns home, the user checks that the necessary ingredients have been delivered and begins cooking based on the presented menu.

[0225] Step 16:

[0226] If the user cannot find time to cook, the user notifies the server that he / she would like to have cooked food delivered from a partner restaurant or diner.

[0227] Step 17:

[0228] The server receives the user's notification, obtains prepared food from affiliated restaurants and other establishments in the area, and arranges for delivery at the time specified by the user.

[0229] Through the above specific processing steps, the present invention supports users in efficiently preparing nutritionally balanced meals, and further utilizes an emotion engine to make optimal suggestions based on the user's emotional state.

[0230] Example 2

[0231] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0232] In today's busy lifestyles, people often lack the time and effort to prepare balanced meals. Furthermore, there are no systems that suggest meals that take into account stress and fatigue, making it difficult for users to easily find menus that suit their individual circumstances. Furthermore, if necessary ingredients or seasonings are missing, there is no efficient way to supplement them, which is a time-consuming process.

[0233] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a video and recognizing items in the video, means for generating a food list based on the recognized items, means for generating a menu taking nutritional balance into consideration, means for analyzing the user's emotions and adjusting the menu based on the emotion information, means for identifying missing items, and means for ordering the missing items from a predetermined supply source. This allows users to efficiently prepare nutritionally balanced meals and easily find appropriate menus based on their emotional state, even in the midst of a busy lifestyle. Furthermore, the ordering process for missing ingredients and seasonings can be automated, significantly reducing the effort required for meal preparation.

[0234] The "means for receiving video" is a function for transmitting video data taken by the user to a server via a dedicated application and for the server to receive the video data.

[0235] "Means for recognizing items" is a function that uses video analysis technology to identify and list items such as ingredients and seasonings that appear in the received video data.

[0236] The "means for generating a food list" is a function for creating a list of ingredients currently stored in the refrigerator or storage area based on information about items recognized through video analysis.

[0237] The "means for generating menus" is a function for automatically generating multiple menu plans based on the recognized food list, taking into consideration nutritional balance.

[0238] The "means for analyzing emotions" is a function for analyzing emotional information in text, audio, or video input by the user and identifying the user's current emotional state.

[0239] The "means for adjusting the menu" is a function for adapting and optimizing the temporarily generated menu plan to the user's emotional state based on the analyzed emotional information.

[0240] The "means for identifying missing items" is a function for identifying the ingredients and seasonings required for the generated menu that are currently missing from the refrigerator or storage.

[0241] The "means for ordering from a predetermined supply source" is a function for automatically ordering missing items from a supply source such as an online supermarket after identifying the items that are in short supply.

[0242] This system allows users to take videos of ingredients in their refrigerators and storage areas, analyzes the videos, and automatically generates nutritionally balanced menus. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, this system can adjust the menu suggestions according to the user's emotional state.

[0243] System configuration

[0244] This system starts when the user uses a device such as a smartphone or tablet to record video of the inside of a refrigerator or storage unit. A dedicated app is installed on the device, and the video recorded using this app is sent to a server. The video is sent via an internet communication line.

[0245] Video Analysis

[0246] The server has a program for analyzing the received video. In this case, OpenCV or TensorFlow can be used as video analysis software. The server identifies items in the video, specifically ingredients and seasonings. This identification can be performed using image recognition technology or machine learning models.

[0247] Ingredient list generation and menu suggestions

[0248] The server generates a food list based on the identified ingredients. This list is stored in a database, and nutritional calculation software is used to automatically generate multiple balanced meal plans. These plans include a combination of a main dish and three side dishes (soup, main course, and side dish). Cooking time and calorie information are also included for each plan.

[0249] Emotion analysis

[0250] The server then uses an emotion engine to analyze the user's emotional state. This analysis can be performed using, for example, IBM Watson® Tone Analyzer or Microsoft® Azure Emotion API. The emotion engine analyzes the text and voice data the user enters into the device, or facial expressions captured using the camera. The optimal menu is then selected from the menu suggestions based on the user's emotional state. For example, if the user is feeling stressed, a dish that is easy to prepare and requires little cooking time will be suggested.

[0251] Sending and displaying menu information

[0252] The server sends the optimized menu information to the terminal. The terminal displays the received information in an easy-to-read format, allowing the user to easily select their preferred menu. The user selects the desired menu from the suggested menu and checks its details. If there are any items missing from the refrigerator or storage at this point, the server automatically lists them and places an order with a designated supply source (for example, a local online supermarket).

[0253] Automated ordering and delivery

[0254] When placing an order, the delivery is scheduled to coincide with the user's return time, and when the user returns home, the necessary ingredients are delivered in a timely manner.

[0255] Prepared food delivery

[0256] In addition, if the user does not have time to cook or is too tired from work to cook, the server will arrange for cooked food to be delivered from affiliated restaurants, allowing the user to provide nutritionally balanced meals for themselves and their family.

[0257] Specific examples

[0258] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via a dedicated app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as the side dish. Furthermore, an emotion engine recognizes the user's stress level and prioritizes menu options that are easy to prepare. Any "miso" that is in short supply is listed and ordered from an online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0259] Prompt Sentence Examples

[0260] "Please suggest a nutritionally balanced meal using the ingredients in your refrigerator. Currently, the refrigerator contains chicken thighs, cabbage, and carrots. The user is currently feeling stressed, so would like a recipe that is easy to prepare."

[0261] As described above, the system of the present invention allows users to efficiently prepare nutritionally balanced meals and easily find appropriate menus according to their emotional state. Furthermore, by automating the ordering process for ingredients and seasonings that are in short supply, it is possible to significantly reduce the effort required for meal preparation in users' daily lives.

[0262] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0263] Step 1:

[0264] The user takes a video of the refrigerator using the device.

[0265] Input: A user uses a smartphone or tablet to record a video of the inside of a refrigerator.

[0266] Output: Video data of the inside of the refrigerator.

[0267] Specific operation: The user launches the dedicated app and shoots a video that captures all the food in the refrigerator.

[0268] Step 2:

[0269] The video data is sent to the server.

[0270] Input: Video data taken inside a refrigerator.

[0271] Output: Video data sent to the server.

[0272] Specific operation: After the user has finished shooting, they use a dedicated app to send the video they have shot to a server via internet communication.

[0273] Step 3:

[0274] The server analyzes the video and generates a list of ingredients.

[0275] Input: Video data sent to the server.

[0276] Output: A list of recognized ingredients.

[0277] How it works: The server uses video analysis software to apply image recognition technology to identify ingredients in the video. Specifically, it uses OpenCV and TensorFlow to identify ingredients and generate a food list.

[0278] Step 4:

[0279] The server automatically generates the menu.

[0280] Input: A list of recognized ingredients.

[0281] Output: A list of generated menus.

[0282] What it does: The server uses nutrition calculation software to generate multiple nutritionally balanced meal plans based on the recognized ingredients, using the nutrition information and recipes stored in the database for this process.

[0283] Step 5:

[0284] Recognize user emotions with an emotion engine.

[0285] Input: User text, voice, or facial expression data.

[0286] Output: Parsed emotional state.

[0287] How it works: The server uses an emotion recognition engine to analyze the text, voice data, and facial expressions entered by the user on the device. Examples include IBM Watson Tone Analyzer and Microsoft Azure Emotion API.

[0288] Step 6:

[0289] Optimize the menu suggested by the server.

[0290] Input: Generated menu list and parsed emotional state.

[0291] Output: A list of optimized menus that match the user's emotions.

[0292] Specific operation: The server proposes menus that suit the user's emotions based on the generated menu information and emotional information. For example, if the user is feeling stressed, it will prioritize suggestions for easy-to-prepare dishes.

[0293] Step 7:

[0294] Optimized menu information is sent to the device.

[0295] Input: A list of optimized menu items.

[0296] Output: Menu information sent to the device.

[0297] Specific operation: The server sends the optimized menu information to the terminal in real time. This sending process is performed in data formats such as JSON and XML.

[0298] Step 8:

[0299] The user checks and selects the menu.

[0300] Input: Multiple menu items displayed on the terminal.

[0301] Output: The selected menu.

[0302] Specific operation: The user checks the menu information received on the device and intuitively swipes and taps to select the desired menu.

[0303] Step 9:

[0304] The server identifies the ingredients that are in short supply and processes the order.

[0305] Input: Selected menu and list of ingredients in the refrigerator.

[0306] Output: A list of ingredients that are in short supply and have been ordered from the online supermarket.

[0307] Specific operation: Based on the selected menu, the server identifies the ingredients that are missing and automatically processes the order via the online supermarket API.

[0308] Step 10:

[0309] Arrange for delivery of prepared meals as needed.

[0310] Input: Information about the user's cooking time and status.

[0311] Output: Cooked food ordered from partner restaurant.

[0312] Specific operation: If the user does not have time to cook or is too tired to cook, the server will use the online ordering system of a partner restaurant to send the user's address and order details and arrange for delivery.

[0313] By following these steps, the user can prepare a nutritionally balanced meal without much effort.

[0314] (Application example 2)

[0315] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0316] While conventional meal suggestion systems could automatically generate menus that took into account the ingredients and nutritional balance of the user's inventory, they were unable to suggest menus that reflected the user's emotional state. Furthermore, there was no centralized system for purchasing ingredients when the user was short of them or when they could not find the time to cook, which often caused stress for users. Another issue was that the suggested menu options were visually difficult to understand.

[0317] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0318] In this invention, the server includes means for receiving a video and recognizing items in the video, means for generating a plurality of combinations based on the recognized items, means for recognizing the user's emotional state, means for adjusting the priority of the combinations to be proposed based on the emotional state, means for identifying missing items, means for ordering the missing items from a predetermined supply source, and means for displaying the generated menu information in an easy-to-view format to allow the user to select. This allows the server to propose optimal menus based on the user's emotional state and available ingredients, and also allows for centralized management of purchasing missing ingredients and delivery of cooked meals, enabling the user to easily prepare or obtain nutritionally balanced meals without stress.

[0319] "Means for receiving video and recognizing items in the video" refers to a technology that acquires video data sent from the user's device and analyzes and identifies the items (ingredients and seasonings) that appear in the video.

[0320] The "means for generating multiple combinations based on recognized items" is a technology that automatically generates menu combinations that take into account diverse nutritional balances based on analyzed item information.

[0321] "Means for recognizing the user's emotional state" refers to technology that analyzes emotions from the user's input, tone of voice, changes in facial expression, etc., and identifies that state.

[0322] The "means for adjusting the priority of proposed combinations based on the emotional state" is a technology that dynamically changes the display priority of the most suitable combination from among the generated menus, taking into account the user's emotional state.

[0323] The "means for identifying missing items" is a technology that automatically detects items that are necessary for the generated menu but are not present in the user's inventory of ingredients.

[0324] The "means for ordering the missing items from a predetermined supply source" is a technology for automatically ordering the detected missing ingredients from a supply source such as an online store or supermarket.

[0325] "Means for displaying generated menu information in an easy-to-see format and allowing users to make selections" refers to a technology that displays a list of suggested menus in a visually easy-to-understand format on the user's terminal, providing an interface that allows users to easily make selections.

[0326] The present invention is a system that allows users to take videos of ingredients in their refrigerators and storage areas, analyzes the videos, and automatically generates nutritionally balanced menus. Furthermore, by adjusting the menu suggestions according to the user's emotional state, it is possible to provide meal suggestions that take into account the user's mental state and preferences.

[0327] First, a user uses a device such as a smartphone or tablet to record video of the inside of a refrigerator or storage unit. A dedicated application is installed on the device, and the video is sent to a cloud server using this application. The video is sent using a communication line such as the Internet.

[0328] Next, a video analysis program runs on the cloud server. Specifically, it uses video analysis engines such as OpenCV and TensorFlow to identify and list items (ingredients and seasonings) in the video. Based on the results of this analysis, multiple menus are generated that take nutritional balance into consideration. Combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, garnish) are generated, and cooking time and calorie information is also added to each menu.

[0329] After ingredient recognition is complete, an emotion engine is activated to recognize the user's emotional state. This emotion engine uses, for example, the Google Cloud Natural Language API or the Microsoft Azure Emotion API to analyze emotions based on the text the user enters into their device, their tone of voice, or changes in facial expressions. Based on this emotional information, the suggested meal plans are adjusted. For example, if the user is feeling stressed, the system can prioritize easy-to-prepare meals.

[0330] Next, the cloud server optimizes the priority of the menu suggestions based on the generated menu information and emotion information, and sends the results to the user's device. At this time, the generated menu information is displayed visually in an easy-to-understand user interface, allowing the user to easily select a menu. For example, a list of menus may be displayed in card format, with each card showing cooking time, calories, and icons of the ingredients used.

[0331] Furthermore, if any of the ingredients or seasonings required for the user's selected meal are missing from the refrigerator or storage, the cloud server will identify them and automatically list them. With the user's permission, the cloud server will order the missing items from a designated supplier (such as a local online supermarket). When the order is placed, it will be scheduled to be delivered by the user's return home time.

[0332] In addition, if the user cannot find time to cook or is too tired from work to cook, the cloud server can also arrange for cooked food to be delivered from affiliated food providers (such as set meal restaurants).

[0333] As a specific example, a user takes a video of the contents of their refrigerator with their smartphone and sends it to a cloud server via an application. The cloud server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Furthermore, an emotion engine recognizes the user's stress level and prioritizes menu options that are easy to prepare. Any "miso" that is in short supply is listed and ordered from an online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0334] Example prompt sentence:

[0335] "What should I make for dinner today? I'm busy and stressed at work, so I'd like something easy to make."

[0336] In this way, the system of the present invention proposes optimal menus based on the user's emotional state and the ingredients they have on hand, and can also centrally manage the purchase of missing ingredients and the delivery of cooked meals, allowing users to easily prepare or obtain nutritionally balanced meals without feeling stressed.

[0337] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0338] Step 1:

[0339] The user uses a device to record video of the food in the refrigerator or storage area. Once the video is complete, the user uses a dedicated application to send the video data to a cloud server. The input is high-resolution video data recorded by the device, and the output is this video data sent to the cloud server.

[0340] Step 2:

[0341] The cloud server analyzes the received video data and identifies the ingredients in the video. This process is performed using OpenCV and TensorFlow. Specifically, the video is broken down into frames, and the items (ingredients and seasonings) in each frame are recognized based on a model. The input is the video data sent by the user, and the output is a list of identified ingredients.

[0342] Step 3:

[0343] The cloud server generates multiple nutritionally balanced menus based on the identified ingredient list, including information such as the main dish, several side dishes, total calories, and cooking time. The input is the identified ingredient list, and the output is the generated menu list.

[0344] Step 4:

[0345] Next, the server recognizes the user's emotional state. Emotion recognition is achieved using the user's text input, voice input, or facial expression recognition. Technologies used include Google Cloud Natural Language API and Microsoft Azure Emotion API. The input includes text, audio, and video data that indicate the user's emotions, and the output is the analyzed emotional state.

[0346] Step 5:

[0347] The cloud server adjusts the priority of the generated menu items based on the recognized emotional state. For example, for a user who is under stress, adjustments are made such as giving priority to displaying easy-to-prepare menu items. The inputs are the emotional state and the generated menu list, and the output is the menu list with adjusted priority.

[0348] Step 6:

[0349] The cloud server sends the menu list with adjusted priorities to the user's device. A dedicated application displays the menu list in a visually easy-to-understand format. For example, cooking time and calorie information may be presented to the user in card format. The input is the menu list with adjusted priorities, and the output is the device screen that displays it.

[0350] Step 7:

[0351] Based on the menu selected by the user, the cloud server generates a list of ingredients that are missing. This list is generated by comparing it with the ingredients the user owns. The input is the list of ingredients the user owns and the selected menu, and the output is the list of ingredients that are missing.

[0352] Step 8:

[0353] The cloud server automatically places orders with a specified supply source (e.g., an online supermarket) based on the list of ingredients that are in short supply. This order is scheduled to coincide with the user's return time. The inputs are the list of ingredients that are in short supply and the user's return time, and the output is the order data sent to the supply source.

[0354] Step 9:

[0355] If the user does not have time to cook or is tired from work, the cloud server arranges for cooked food from affiliated food service providers. The input is the user's emotional state and cooking availability, and the output is the order and delivery request data for the cooked food.

[0356] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0357] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0358] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0359] [Second embodiment]

[0360] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0361] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0362] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0363] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0364] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0365] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0366] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0367] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0368] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0369] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0370] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0371] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0372] This invention is a system that allows users to take videos of ingredients in their refrigerators or storage areas, analyzes the videos, and automatically creates nutritionally balanced menus. This system operates in cooperation with a server, terminals, and users to provide convenience and health to users.

[0373] First, the user uses a device such as a smartphone or tablet to record video of the inside of the refrigerator or storage unit. A dedicated app is installed on the device, and the video is sent to the server using this app. The video is sent via a communication line.

[0374] The server has a program that analyzes the received video. The server identifies and lists the items (ingredients and seasonings) in the video. Based on this list, the server generates multiple menus that take nutritional balance into consideration. Specifically, it generates combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also added to each menu.

[0375] The server then sends the generated menu information to the terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to select. The user selects the appropriate menu from the multiple menus presented and checks its details. If there are any ingredients or seasonings required for the menu that are missing from the refrigerator or storage, the server identifies them and lists them.

[0376] With the user's permission, the server orders missing ingredients and seasonings from a designated source (such as a local online supermarket). When the order is placed, it schedules delivery to coincide with the user's return home time. When the user returns home, the necessary ingredients have been delivered in a timely manner.

[0377] In addition, if the user does not have time to cook or is too tired from work to cook, the server will arrange for pre-cooked meals to be delivered from affiliated restaurants, allowing the user to provide healthy meals for themselves and their family.

[0378] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via the app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Any "miso" that is missing is listed and ordered from the online supermarket in time for the user to return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0379] As described above, the system of the present invention provides comprehensive support for users to efficiently prepare or obtain nutritionally balanced meals, allowing even busy users to easily achieve healthy meals.

[0380] The processing flow will be explained below.

[0381] Step 1:

[0382] Users take videos of the ingredients in their refrigerators or storage areas using a smartphone or tablet, adjusting the camera angle and brightness to ensure all ingredients are clearly visible.

[0383] Step 2:

[0384] The device sends the captured video to a server via a dedicated app, where it performs correction processing to appropriately compress the video data while maintaining image quality.

[0385] Step 3:

[0386] The server receives the video sent from the device and stores it in a database, where it is processed by an analysis program.

[0387] Step 4:

[0388] The server uses an AI-generated model to identify ingredients in the video, detecting items (ingredients and seasonings) in each frame and listing them.

[0389] Step 5:

[0390] Based on the listed ingredient information, the server retrieves nutritional information for each ingredient and available recipes from the database.

[0391] Step 6:

[0392] The server runs a menu generation algorithm that takes nutritional balance into consideration based on the acquired ingredient information and recipe data, generating five combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish).

[0393] Step 7:

[0394] The server adds cooking time and calorie information to each menu item and also generates a detailed list of the ingredients and seasonings required.

[0395] Step 8:

[0396] The server sends the generated menu information to the user's device using a secure communication method.

[0397] Step 9:

[0398] The device displays the menu information received from the server in a dedicated app. The user can check multiple menu options and view detailed information (cooking time, calories, and a list of necessary ingredients).

[0399] Step 10:

[0400] The user selects an appropriate menu from the presented menu options, checks the details, and then lists any missing ingredients and seasonings. The user then authorizes the ordering of missing ingredients within the app.

[0401] Step 11:

[0402] With the user's permission, the server automatically orders the listed ingredients from a designated supplier (such as a local online supermarket) and schedules delivery to coincide with the user's return home time.

[0403] Step 12:

[0404] The server obtains the estimated delivery time of the ordered ingredients from the supply source and notifies the user.

[0405] Step 13:

[0406] When the user returns home, the user checks that the necessary ingredients have been delivered and begins cooking based on the presented menu.

[0407] Step 14:

[0408] If the user cannot find time to cook, the user notifies the server that he / she would like to have cooked food delivered from a partner restaurant or diner.

[0409] Step 15:

[0410] The server receives the user's notification, obtains prepared food from affiliated restaurants and other establishments in the area, and arranges for delivery at the time specified by the user.

[0411] Through the above specific processing steps, the present invention helps users to efficiently prepare nutritionally balanced meals.

[0412] Example 1

[0413] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0414] In today's busy lifestyles, it is difficult for users to easily decide on nutritionally balanced meals and procure the necessary ingredients in a timely manner. In particular, users who are busy with work or housework cannot afford to take the time to identify shortages of ingredients and replenish them, making it difficult to prepare healthy meals. Furthermore, if they cannot find time to cook, they have to go through the hassle of arranging pre-cooked meals as an alternative.

[0415] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0416] In this invention, the server includes a means for receiving a video and recognizing items in the video, a means for generating multiple combinations based on the recognized items, and a means for identifying missing items. It also includes a means for automatically analyzing each frame of the captured video and identifying items, a means for automatically generating a nutritionally balanced menu based on the analyzed list of items, and a means for displaying the generated menu information and allowing the user to select from it. This allows the user to easily decide on a nutritionally balanced menu and procure the necessary ingredients in a timely manner. Furthermore, if cooking time is unavailable, pre-cooked meals can be quickly arranged.

[0417] "Means for receiving video and recognizing objects in the video" refers to a technology that sends a video taken by a user to a server and automatically recognizes objects that appear in the video through analysis.

[0418] The "means for generating a plurality of combinations based on recognized items" is a technology for automatically generating a plurality of nutritionally balanced menus based on a list of recognized items.

[0419] The "means for identifying missing items" is a technology for automatically finding items required for the generated menu that are not in the refrigerator or storage.

[0420] "Means for ordering from a specified source of supply" refers to technology that automatically places an order for the missing items with a specific supplier (for example, an online supermarket or a retail store).

[0421] "Means for analyzing each frame and automatically identifying items" refers to a technology for breaking down a video into individual frames and analyzing the items shown in each frame.

[0422] The "means for automatically generating menus that take nutritional balance into consideration" is a technology for generating multiple menus that take nutritional balance into consideration based on an analyzed item list.

[0423] "Means for displaying menu information and allowing the user to select" refers to technology that displays automatically generated menu information on the user's terminal and allows the user to select their preferred menu from the information.

[0424] This invention is a system that allows users to take videos of ingredients in their refrigerators or storage areas, analyzes the videos, and automatically creates nutritionally balanced menus. This system operates in cooperation with a server, terminals, and users to provide convenience and health to users.

[0425] First, the user uses a device such as a smartphone or tablet to take a video of the inside of the refrigerator or storage unit. A dedicated app is installed on the device, and the taken video is sent to the server using this app. The video is sent via a communication line. For example, the user takes a video of the inside of the refrigerator and sends it to the server via the dedicated app.

[0426] The server has a program that analyzes the received video. Using a video analysis program (e.g., Azure Video Analyzer or Google Cloud Video Intelligence), it analyzes the captured video frame by frame and automatically identifies items (ingredients and seasonings) in the video. Based on the analyzed list of items, the server generates a menu that takes nutritional balance into consideration. Specifically, it uses a generative AI model (e.g., OpenAI GPT-4) to combine a staple food (rice, bread, noodles) with three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also included with each menu.

[0427] Next, the server sends the generated menu information to the terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to make a selection. The user selects the appropriate menu from the multiple menus presented and checks its details. For example, the terminal may display "Main dish: rice, Main: teriyaki chicken thigh, Soup: miso soup, Side dish: stir-fried cabbage and carrots," with each menu item provided along with calorie information and cooking time.

[0428] If there are any ingredients or seasonings needed for a meal that are missing from the refrigerator or storage, the server will identify them and list them. With the user's permission, the server will order the missing ingredients or seasonings from a designated source (such as a local online supermarket). When ordering, the server will schedule delivery to coincide with the user's return home time. For example, the server will list the missing "miso" and order it from the online supermarket to coincide with the user's return home time.

[0429] In addition, if the user cannot find time to cook or is too tired from work to cook, the server will arrange for pre-cooked meals to be delivered from affiliated restaurants. The user can notify the app that they are having difficulty cooking and select pre-cooked meals as an alternative. For example, if the user inputs into the app, "I'm tired today, so I'd like pre-cooked meals delivered," the server will arrange for the above menu to be delivered from an affiliated restaurant.

[0430] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via a dedicated app. The server analyzes the received video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Any "miso" that is missing is listed and ordered from the online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0431] In this way, the system of the present invention provides comprehensive support for users to efficiently prepare or obtain nutritionally balanced meals, allowing even busy users to easily achieve healthy meals.

[0432] Example prompt sentence:

[0433] "Create a system that allows users to take a video of the ingredients in their refrigerator, analyzes the video, and automatically generates a nutritionally balanced meal plan."

[0434] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0435] Step 1: Record and send video

[0436] Subject: User

[0437] Users can launch a dedicated app on their smartphone or tablet and take video of the inside of their refrigerator or storage unit. Once the video is complete, the user can send the video to the server via the app.

[0438] Input: Video data taken inside a refrigerator or storage facility

[0439] Output: Video data sent to the server

[0440] Specific operation: The user opens the refrigerator door, takes sequential photos of the inside with their smartphone camera, and taps the "Send" button to upload the video data to the server.

[0441] Step 2: Receiving and analyzing the video

[0442] Subject: Server

[0443] The server receives the video sent by the user and uses a video analysis program to analyze the video frame by frame and identify the items (ingredients and seasonings) that appear in the video.

[0444] Input: Video data sent by the user

[0445] Output: List of recognized objects

[0446] How it works: The server runs a video analysis program, breaks down the video into frames, and automatically identifies and lists the ingredients and seasonings. For example, a list of items such as "chicken thighs, cabbage, and carrots" is obtained.

[0447] Step 3: Menu Creation

[0448] Subject: Server

[0449] The server generates a menu that takes nutritional balance into consideration based on the list of items obtained through the analysis. Using a generative AI model, multiple menu plans are created. Each menu includes a main dish and three side dishes (soup, main course, and garnish), and cooking time and calorie information are also included.

[0450] Input: A list of recognized items (e.g., "chicken thigh, cabbage, carrot")

[0451] Output: Multiple menu plans that take nutritional balance into consideration

[0452] Specific operation: The server runs the AI ​​model and generates a menu suggestion such as "rice, teriyaki chicken thighs, miso soup, and stir-fried cabbage and carrots."

[0453] Step 4: Display menu information

[0454] Subject: Terminal

[0455] The terminal receives the menu information sent from the server and displays it on the screen in an easy-to-read format. The user can then select an appropriate menu from multiple suggested menus.

[0456] Input: Menu information sent from the server

[0457] Output: Menu information displayed on the device

[0458] Specific operation: The device displays menu suggestions such as "Main dish: rice, Main course: teriyaki chicken thigh, Soup: miso soup, Side dish: stir-fried cabbage and carrots," and the user can choose from them.

[0459] Step 5: Identify and order missing ingredients

[0460] Subject: Server

[0461] The server automatically lists any missing items based on the selected menu, orders missing ingredients and seasonings from a designated supplier (such as an online supermarket), and schedules delivery to coincide with the user's return time.

[0462] Input: Selected menu plan, user's current ingredients list

[0463] Output: List of ingredients to order, order information

[0464] Specific operation: The server lists ingredients such as miso that are in short supply, and after receiving confirmation from the user, automatically places an order with the online supermarket. The delivery time is set to coincide with the user's return home.

[0465] Step 6: Arrange for prepared meals

[0466] Subject: Server

[0467] If the user cannot find time to cook, the server will arrange for the delivery of cooked food from affiliated restaurants or diner.

[0468] Input: User notification of cooking difficulty

[0469] Output: Arrange for delivery of cooked food

[0470] Specific operation: The user enters "I can't cook today" through the app, and the server arranges for the following meal from a partner restaurant: "Main dish: rice, Main course: teriyaki chicken thigh, Soup: miso soup, Side dish: stir-fried cabbage and carrots" and has it delivered.

[0471] (Application example 1)

[0472] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0473] In today's busy lifestyles, it is difficult for users to prepare healthy, nutritionally balanced meals. Managing the contents of the refrigerator and storage areas and planning menus based on that information is also a time-consuming task. Furthermore, if necessary ingredients are in short supply, it is inefficient to go shopping every time. To solve these problems, a system is needed that allows users to easily manage ingredients, create menus, and order missing ingredients.

[0474] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0475] In this invention, the server includes a means for receiving a video and recognizing items in the video, a means for generating a plurality of combinations based on the recognized items, and a means for identifying missing items. This automatically lists ingredients in the refrigerator or storage area from the video taken by the user, and generates a nutritionally balanced menu based on that list. Furthermore, the automatic identification of missing items allows online ordering from a virtual store, and delivery of ingredients or cooked meals can be arranged to coincide with the user's return home time. This allows users to prepare and obtain healthy meals without hassle.

[0476] The "means for receiving video and recognizing items in the video" is a technology for receiving video shot by a user and analyzing and identifying items that appear in the video.

[0477] The "means for generating multiple combinations based on recognized items" is a technology for creating multiple menus based on information about identified items, taking into consideration nutritional balance, cooking methods, etc.

[0478] The "means for identifying missing items" is a technology for comparing the recognized items with the items required for the generated menu and listing the missing items.

[0479] The "means for ordering the missing items from a predetermined supply source" is a technique for ordering the identified missing items from a related supply source (for example, an online supermarket or an affiliated store).

[0480] The "means for generating nutritionally balanced menus" is a technology for creating multiple balanced menus by taking into consideration the nutritional components of recognized items.

[0481] "Means for sending the generated menu information to the user terminal and displaying it so that the user can select" refers to a technology for sending the generated menu data to the user's device and displaying it so that the user can select the menu they desire.

[0482] "Means for placing online orders at a virtual store" refers to technology that allows users to easily place orders at a virtual store (e.g., an online shopping site) based on a list of items they are missing.

[0483] This system allows users to take videos of ingredients in their refrigerators or storage areas, analyzes the videos, and automatically creates nutritionally balanced meals. The system operates in cooperation with the server, terminals, and users to provide convenience and health to users.

[0484] First, the user uses a device such as a smartphone or tablet to record video of the inside of the refrigerator or storage unit. A dedicated application is installed on the device, and the video is sent to the server using this application. The video is sent via a communication line.

[0485] The server has a program that analyzes the received video. This analysis uses the OpenCV library for image analysis and TensorFlow for ingredient recognition. The server identifies and lists the items (ingredients and seasonings) in the video. Based on this list, the server uses a generative AI model to generate multiple menus that take nutritional balance into consideration. Specifically, it generates combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also included with each menu.

[0486] The server then sends the generated menu information to the terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to select. The user selects the appropriate menu from the multiple menus presented and checks its details. If there are any ingredients or seasonings required for the menu that are missing from the refrigerator or storage, the server identifies them and lists them.

[0487] With the user's permission, the server orders missing ingredients and seasonings from a designated supply source (such as a local online supermarket). When the order is placed, it schedules delivery to coincide with the user's return time. This ensures that the necessary ingredients have arrived in a timely manner when the user returns home. In addition, if the user is unable to find time to cook or is too tired from work to cook, the server will arrange for pre-cooked food to be delivered from an affiliated restaurant.

[0488] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via the app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Any "miso" that is missing is listed and ordered from the online supermarket in time for the user to return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0489] An example of a prompt for a generative AI model is as follows:

[0490] "Analyze the video of the contents of the refrigerator and create a list of recognized ingredients. Furthermore, use that data to generate a nutritionally balanced meal plan, identifying and listing any missing ingredients. Display the generated meal plan in the following format: 'Main dish: Teriyaki chicken thigh, Side dish: Cabbage salad, Another side dish: Kinpira carrots'. If any ingredients are missing, provide a function that allows the user to order them from a virtual store."

[0491] The system allows users to prepare and consume healthy meals hassle-free.

[0492] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0493] Step 1:

[0494] Users use devices such as smartphones or tablets to record video of the inside of refrigerators or storage units. A dedicated application is installed on the device, and the video recorded using this application is sent to a server. The input is video data recording the inside of the refrigerator or storage unit, and the output is the video data sent to the server. The device sends the video data via a communication line.

[0495] Step 2:

[0496] To analyze the received video, the server uses the OpenCV library to break down the video data into frames and perform image analysis on each frame. Based on the recognition results, TensorFlow is used to identify ingredients. The input is the image data broken down into frames, and the output is a list of recognized ingredients. Specifically, the server resizes the images of each frame to 224 x 224 pixels and inputs them into the ingredient recognition model.

[0497] Step 3:

[0498] The server uses a generative AI model to generate multiple nutritionally balanced menus based on the list of recognized ingredients. The input is the list of recognized ingredients, and the output is a list of generated menus. Specifically, it takes into account the combination of staple food, side dish, and soup, and also adds cooking time and calorie information to each menu.

[0499] Step 4:

[0500] The server sends the generated menu information to the user terminal. The input is a list of the generated menus, and the output is the menu information sent to the user terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to select an appropriate menu from it.

[0501] Step 5:

[0502] The user selects the appropriate menu from the multiple menus presented and checks its details. The input is the menu information sent from the server, and the output is the details of the menu selected by the user. The selected menu includes the necessary ingredients and seasonings.

[0503] Step 6:

[0504] The server identifies and lists the items that are missing from the refrigerator or storage based on the selected menu. The input is the selected menu and the current status of the refrigerator or storage, and the output is a list of the missing items.

[0505] Step 7:

[0506] With the user's permission, the server orders missing ingredients and seasonings from a designated source (e.g., a local online supermarket). The input is a list of missing items, and the output is order confirmation information. When placing an order, the server schedules delivery to coincide with the user's return home time.

[0507] Step 8:

[0508] If the user cannot find the time to cook, the server will arrange for the delivery of cooked food from a partner restaurant. The inputs are the user's cooking time and the partner restaurant's menu, and the output is confirmation of the delivery of the cooked food.

[0509] This allows users to prepare and consume healthy meals without any hassle.

[0510] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0511] This invention is a system that allows users to take videos of ingredients in their refrigerators and storage areas, analyzes the videos, and automatically generates nutritionally balanced menus. By combining this invention with an emotion engine that recognizes the user's emotions, it is possible to adjust the menu suggestions according to the user's emotional state. This allows for meal suggestions that are tailored to the user's mental state and preferences.

[0512] First, the user uses a device such as a smartphone or tablet to record video of the inside of the refrigerator or storage unit. A dedicated app is installed on the device, and the video is sent to the server using this app. The video is sent via a communication line.

[0513] The server has a program that analyzes the received video. The server identifies and lists the items (ingredients and seasonings) in the video. Based on this list, the server generates multiple menus that take nutritional balance into consideration. Specifically, it generates combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also added to each menu.

[0514] The server then analyzes the user's current emotional state using an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions based on, for example, the text the user types into the device, the tone of voice, or changes in facial expressions. This emotion information is used to adjust the menu suggestions. For example, if the user is feeling stressed, the server can prioritize easy-to-prepare menus.

[0515] The server optimizes the priority of the menu suggestions based on the generated menu information and emotion information and sends them to the device. The device then displays the received menu information in an easy-to-read format, allowing the user to select. The user selects an appropriate menu from the multiple menus presented and checks its details. If there are any ingredients or seasonings required for the menu that are missing from the refrigerator or storage, the server identifies them and lists them.

[0516] With the user's permission, the server orders missing ingredients and seasonings from a designated source (such as a local online supermarket). When the order is placed, it schedules delivery to coincide with the user's return home time. When the user returns home, the necessary ingredients have been delivered in a timely manner.

[0517] In addition, if the user does not have time to cook or is too tired from work to cook, the server will arrange for pre-cooked meals to be delivered from affiliated restaurants, allowing the user to provide healthy meals for themselves and their family.

[0518] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via the app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Furthermore, an emotion engine recognizes the user's stress level and prioritizes menu options that are easy to prepare. Any "miso" that is in short supply is listed and ordered from an online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0519] As described above, the system of the present invention provides comprehensive support for users to efficiently prepare or obtain nutritionally balanced meals, enabling users to easily and effectively achieve healthy eating habits even in busy daily lives.

[0520] The processing flow will be explained below.

[0521] Step 1:

[0522] Users take videos of the ingredients in their refrigerators or storage areas using a smartphone or tablet, adjusting the camera angle and brightness to ensure all ingredients are clearly visible.

[0523] Step 2:

[0524] The device sends the captured video to the server via a dedicated app. Before sending, the device encodes the video and compresses it appropriately.

[0525] Step 3:

[0526] The server receives the video sent from the device and stores it in a database. The received video is then processed by an analysis program.

[0527] Step 4:

[0528] The server uses an AI-generated model to identify ingredients in the video, detecting them in each frame and listing them.

[0529] Step 5:

[0530] Based on the listed ingredient information, the server retrieves nutritional information and available recipes for each ingredient from the database.

[0531] Step 6:

[0532] The server runs a menu generation algorithm that takes nutritional balance into consideration based on the acquired ingredient information and recipe data, generating five combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish).

[0533] Step 7:

[0534] The server adds cooking time and calorie information to each menu, and also generates a detailed list of ingredients and seasonings required.The server also uses an emotion engine to analyze the user's emotional state.

[0535] Step 8:

[0536] The server adjusts its menu suggestions based on the user's emotional state: for example, if the user is feeling stressed, it will prioritize easy-to-prepare meals.

[0537] Step 9:

[0538] The server then transmits the optimized menu information, taking into account the user's emotional state, to the user's device using secure communication methods.

[0539] Step 10:

[0540] The device displays the menu information received from the server in a dedicated app. The user can check multiple menu options and view detailed information (cooking time, calories, and a list of necessary ingredients).

[0541] Step 11:

[0542] The user selects an appropriate menu from the presented options and checks the details. After making a selection, the system lists any missing ingredients and seasonings.

[0543] Step 12:

[0544] The user can authorize the app to order missing ingredients by tapping the order authorization button.

[0545] Step 13:

[0546] With the user's permission, the server automatically orders the missing ingredients from a designated supply source (such as a local online supermarket). When placing an order, the server sets a delivery schedule to match the user's return time.

[0547] Step 14:

[0548] The server obtains the estimated delivery time of the ordered ingredients from the supply source and notifies the user.

[0549] Step 15:

[0550] When the user returns home, the user checks that the necessary ingredients have been delivered and begins cooking based on the presented menu.

[0551] Step 16:

[0552] If the user cannot find time to cook, the user notifies the server that he / she would like to have cooked food delivered from a partner restaurant or diner.

[0553] Step 17:

[0554] The server receives the user's notification, obtains prepared food from affiliated restaurants and other establishments in the area, and arranges for delivery at the time specified by the user.

[0555] Through the above specific processing steps, the present invention supports users in efficiently preparing nutritionally balanced meals, and further utilizes an emotion engine to make optimal suggestions based on the user's emotional state.

[0556] Example 2

[0557] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0558] In today's busy lifestyles, people often lack the time and effort to prepare balanced meals. Furthermore, there are no systems that suggest meals that take into account stress and fatigue, making it difficult for users to easily find menus that suit their individual circumstances. Furthermore, if necessary ingredients or seasonings are missing, there is no efficient way to supplement them, which is a time-consuming process.

[0559] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a video and recognizing items in the video, means for generating a food list based on the recognized items, means for generating a menu taking nutritional balance into consideration, means for analyzing the user's emotions and adjusting the menu based on the emotion information, means for identifying missing items, and means for ordering the missing items from a predetermined supply source. This allows users to efficiently prepare nutritionally balanced meals and easily find appropriate menus based on their emotional state, even in the midst of a busy lifestyle. Furthermore, the ordering process for missing ingredients and seasonings can be automated, significantly reducing the effort required for meal preparation.

[0560] The "means for receiving video" is a function for transmitting video data taken by the user to a server via a dedicated application and for the server to receive the video data.

[0561] "Means for recognizing items" is a function that uses video analysis technology to identify and list items such as ingredients and seasonings that appear in the received video data.

[0562] The "means for generating a food list" is a function for creating a list of ingredients currently stored in the refrigerator or storage area based on information about items recognized through video analysis.

[0563] The "means for generating menus" is a function for automatically generating multiple menu plans based on the recognized food list, taking into consideration nutritional balance.

[0564] The "means for analyzing emotions" is a function for analyzing emotional information in text, audio, or video input by the user and identifying the user's current emotional state.

[0565] The "means for adjusting the menu" is a function for adapting and optimizing the temporarily generated menu plan to the user's emotional state based on the analyzed emotional information.

[0566] The "means for identifying missing items" is a function for identifying the ingredients and seasonings required for the generated menu that are currently missing from the refrigerator or storage.

[0567] The "means for ordering from a predetermined supply source" is a function for automatically ordering missing items from a supply source such as an online supermarket after identifying the items that are in short supply.

[0568] This system allows users to take videos of ingredients in their refrigerators and storage areas, analyzes the videos, and automatically generates nutritionally balanced menus. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, this system can adjust the menu suggestions according to the user's emotional state.

[0569] System configuration

[0570] This system starts when the user uses a device such as a smartphone or tablet to record video of the inside of a refrigerator or storage unit. A dedicated app is installed on the device, and the video recorded using this app is sent to a server. The video is sent via an internet communication line.

[0571] Video Analysis

[0572] The server has a program for analyzing the received video. In this case, OpenCV or TensorFlow can be used as video analysis software. The server identifies items in the video, specifically ingredients and seasonings. This identification can be performed using image recognition technology or machine learning models.

[0573] Ingredient list generation and menu suggestions

[0574] The server generates a food list based on the identified ingredients. This list is stored in a database, and nutritional calculation software is used to automatically generate multiple balanced meal plans. These plans include a combination of a main dish and three side dishes (soup, main course, and side dish). Cooking time and calorie information are also included for each plan.

[0575] Emotion analysis

[0576] The server then uses an emotion engine to analyze the user's emotional state. This analysis can be performed using, for example, IBM Watson Tone Analyzer or Microsoft Azure Emotion API. The emotion engine analyzes the text and voice data entered by the user into the device, or changes in facial expressions captured using the camera. The optimal menu is then selected from the suggested menus based on the user's emotional state. For example, if the user is feeling stressed, it will suggest dishes that are easy to prepare and require little time to prepare.

[0577] Sending and displaying menu information

[0578] The server sends the optimized menu information to the terminal. The terminal displays the received information in an easy-to-read format, allowing the user to easily select their preferred menu. The user selects the desired menu from the suggested menu and checks its details. If there are any items missing from the refrigerator or storage at this point, the server automatically lists them and places an order with a designated supply source (for example, a local online supermarket).

[0579] Automated ordering and delivery

[0580] When placing an order, the delivery is scheduled to coincide with the user's return time, and when the user returns home, the necessary ingredients are delivered in a timely manner.

[0581] Prepared food delivery

[0582] In addition, if the user does not have time to cook or is too tired from work to cook, the server will arrange for cooked food to be delivered from affiliated restaurants, allowing the user to provide nutritionally balanced meals for themselves and their family.

[0583] Specific examples

[0584] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via a dedicated app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as the side dish. Furthermore, an emotion engine recognizes the user's stress level and prioritizes menu options that are easy to prepare. Any "miso" that is in short supply is listed and ordered from an online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0585] Prompt Sentence Examples

[0586] "Please suggest a nutritionally balanced meal using the ingredients in your refrigerator. Currently, the refrigerator contains chicken thighs, cabbage, and carrots. The user is currently feeling stressed, so would like a recipe that is easy to prepare."

[0587] As described above, the system of the present invention allows users to efficiently prepare nutritionally balanced meals and easily find appropriate menus according to their emotional state. Furthermore, by automating the ordering process for ingredients and seasonings that are in short supply, it is possible to significantly reduce the effort required for meal preparation in users' daily lives.

[0588] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0589] Step 1:

[0590] The user takes a video of the refrigerator using the device.

[0591] Input: A user uses a smartphone or tablet to record a video of the inside of a refrigerator.

[0592] Output: Video data of the inside of the refrigerator.

[0593] Specific operation: The user launches the dedicated app and shoots a video that captures all the food in the refrigerator.

[0594] Step 2:

[0595] The video data is sent to the server.

[0596] Input: Video data taken inside a refrigerator.

[0597] Output: Video data sent to the server.

[0598] Specific operation: After the user has finished shooting, they use a dedicated app to send the video they have shot to a server via internet communication.

[0599] Step 3:

[0600] The server analyzes the video and generates a list of ingredients.

[0601] Input: Video data sent to the server.

[0602] Output: A list of recognized ingredients.

[0603] How it works: The server uses video analysis software to apply image recognition technology to identify ingredients in the video. Specifically, it uses OpenCV and TensorFlow to identify ingredients and generate a food list.

[0604] Step 4:

[0605] The server automatically generates the menu.

[0606] Input: A list of recognized ingredients.

[0607] Output: A list of generated menus.

[0608] What it does: The server uses nutrition calculation software to generate multiple nutritionally balanced meal plans based on the recognized ingredients, using the nutrition information and recipes stored in the database for this process.

[0609] Step 5:

[0610] Recognize user emotions with an emotion engine.

[0611] Input: User text, voice, or facial expression data.

[0612] Output: Parsed emotional state.

[0613] How it works: The server uses an emotion recognition engine to analyze the text, voice data, and facial expressions entered by the user on the device. Examples include IBM Watson Tone Analyzer and Microsoft Azure Emotion API.

[0614] Step 6:

[0615] Optimize the menu suggested by the server.

[0616] Input: Generated menu list and parsed emotional state.

[0617] Output: A list of optimized menus that match the user's emotions.

[0618] Specific operation: The server proposes menus that suit the user's emotions based on the generated menu information and emotional information. For example, if the user is feeling stressed, it will prioritize suggestions for easy-to-prepare dishes.

[0619] Step 7:

[0620] Optimized menu information is sent to the device.

[0621] Input: A list of optimized menu items.

[0622] Output: Menu information sent to the device.

[0623] Specific operation: The server sends the optimized menu information to the terminal in real time. This sending process is performed in data formats such as JSON and XML.

[0624] Step 8:

[0625] The user checks and selects the menu.

[0626] Input: Multiple menu items displayed on the terminal.

[0627] Output: The selected menu.

[0628] Specific operation: The user checks the menu information received on the device and intuitively swipes and taps to select the desired menu.

[0629] Step 9:

[0630] The server identifies the ingredients that are in short supply and processes the order.

[0631] Input: Selected menu and list of ingredients in the refrigerator.

[0632] Output: A list of ingredients that are in short supply and have been ordered from the online supermarket.

[0633] Specific operation: Based on the selected menu, the server identifies the ingredients that are missing and automatically processes the order via the online supermarket API.

[0634] Step 10:

[0635] Arrange for delivery of prepared meals as needed.

[0636] Input: Information about the user's cooking time and status.

[0637] Output: Cooked food ordered from partner restaurant.

[0638] Specific operation: If the user does not have time to cook or is too tired to cook, the server will use the online ordering system of a partner restaurant to send the user's address and order details and arrange for delivery.

[0639] By following these steps, the user can prepare a nutritionally balanced meal without much effort.

[0640] (Application example 2)

[0641] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0642] While conventional meal suggestion systems could automatically generate menus that took into account the ingredients and nutritional balance of the user's inventory, they were unable to suggest menus that reflected the user's emotional state. Furthermore, there was no centralized system for purchasing ingredients when the user was short of them or when they could not find the time to cook, which often caused stress for users. Another issue was that the suggested menu options were visually difficult to understand.

[0643] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0644] In this invention, the server includes means for receiving a video and recognizing items in the video, means for generating a plurality of combinations based on the recognized items, means for recognizing the user's emotional state, means for adjusting the priority of the combinations to be proposed based on the emotional state, means for identifying missing items, means for ordering the missing items from a predetermined supply source, and means for displaying the generated menu information in an easy-to-view format to allow the user to select. This allows the server to propose optimal menus based on the user's emotional state and available ingredients, and also allows for centralized management of purchasing missing ingredients and delivery of cooked meals, enabling the user to easily prepare or obtain nutritionally balanced meals without stress.

[0645] "Means for receiving video and recognizing items in the video" refers to a technology that acquires video data sent from the user's device and analyzes and identifies the items (ingredients and seasonings) that appear in the video.

[0646] The "means for generating multiple combinations based on recognized items" is a technology that automatically generates menu combinations that take into account diverse nutritional balances based on analyzed item information.

[0647] "Means for recognizing the user's emotional state" refers to technology that analyzes emotions from the user's input, tone of voice, changes in facial expression, etc., and identifies that state.

[0648] The "means for adjusting the priority of proposed combinations based on the emotional state" is a technology that dynamically changes the display priority of the most suitable combination from among the generated menus, taking into account the user's emotional state.

[0649] The "means for identifying missing items" is a technology that automatically detects items that are necessary for the generated menu but are not present in the user's inventory of ingredients.

[0650] The "means for ordering the missing items from a predetermined supply source" is a technology for automatically ordering the detected missing ingredients from a supply source such as an online store or supermarket.

[0651] "Means for displaying generated menu information in an easy-to-see format and allowing users to make selections" refers to a technology that displays a list of suggested menus in a visually easy-to-understand format on the user's terminal, providing an interface that allows users to easily make selections.

[0652] The present invention is a system that allows users to take videos of ingredients in their refrigerators and storage areas, analyzes the videos, and automatically generates nutritionally balanced menus. Furthermore, by adjusting the menu suggestions according to the user's emotional state, it is possible to provide meal suggestions that take into account the user's mental state and preferences.

[0653] First, a user uses a device such as a smartphone or tablet to record video of the inside of a refrigerator or storage unit. A dedicated application is installed on the device, and the video is sent to a cloud server using this application. The video is sent using a communication line such as the Internet.

[0654] Next, a video analysis program runs on the cloud server. Specifically, it uses video analysis engines such as OpenCV and TensorFlow to identify and list items (ingredients and seasonings) in the video. Based on the results of this analysis, multiple menus are generated that take nutritional balance into consideration. Combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, garnish) are generated, and cooking time and calorie information is also added to each menu.

[0655] After ingredient recognition is complete, an emotion engine is activated to recognize the user's emotional state. This emotion engine uses, for example, the Google Cloud Natural Language API or the Microsoft Azure Emotion API to analyze emotions based on the text the user enters into their device, their tone of voice, or changes in facial expressions. Based on this emotional information, the suggested meal plans are adjusted. For example, if the user is feeling stressed, the system can prioritize easy-to-prepare meals.

[0656] Next, the cloud server optimizes the priority of the menu suggestions based on the generated menu information and emotion information, and sends the results to the user's device. At this time, the generated menu information is displayed visually in an easy-to-understand user interface, allowing the user to easily select a menu. For example, a list of menus may be displayed in card format, with each card showing cooking time, calories, and icons of the ingredients used.

[0657] Furthermore, if any of the ingredients or seasonings required for the user's selected meal are missing from the refrigerator or storage, the cloud server will identify them and automatically list them. With the user's permission, the cloud server will order the missing items from a designated supplier (such as a local online supermarket). When the order is placed, it will be scheduled to be delivered by the user's return home time.

[0658] In addition, if the user cannot find time to cook or is too tired from work to cook, the cloud server can also arrange for cooked food to be delivered from affiliated food providers (such as set meal restaurants).

[0659] As a specific example, a user takes a video of the contents of their refrigerator with their smartphone and sends it to a cloud server via an application. The cloud server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Furthermore, an emotion engine recognizes the user's stress level and prioritizes menu options that are easy to prepare. Any "miso" that is in short supply is listed and ordered from an online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0660] Example prompt sentence:

[0661] "What should I make for dinner today? I'm busy and stressed at work, so I'd like something easy to make."

[0662] In this way, the system of the present invention proposes optimal menus based on the user's emotional state and the ingredients they have on hand, and can also centrally manage the purchase of missing ingredients and the delivery of cooked meals, allowing users to easily prepare or obtain nutritionally balanced meals without feeling stressed.

[0663] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0664] Step 1:

[0665] The user uses a device to record video of the food in the refrigerator or storage area. Once the video is complete, the user uses a dedicated application to send the video data to a cloud server. The input is high-resolution video data recorded by the device, and the output is this video data sent to the cloud server.

[0666] Step 2:

[0667] The cloud server analyzes the received video data and identifies the ingredients in the video. This process is performed using OpenCV and TensorFlow. Specifically, the video is broken down into frames, and the items (ingredients and seasonings) in each frame are recognized based on a model. The input is the video data sent by the user, and the output is a list of identified ingredients.

[0668] Step 3:

[0669] The cloud server generates multiple nutritionally balanced menus based on the identified ingredient list, including information such as the main dish, several side dishes, total calories, and cooking time. The input is the identified ingredient list, and the output is the generated menu list.

[0670] Step 4:

[0671] Next, the server recognizes the user's emotional state. Emotion recognition is achieved using the user's text input, voice input, or facial expression recognition. Technologies used include Google Cloud Natural Language API and Microsoft Azure Emotion API. The input includes text, audio, and video data that indicate the user's emotions, and the output is the analyzed emotional state.

[0672] Step 5:

[0673] The cloud server adjusts the priority of the generated menu items based on the recognized emotional state. For example, for a user who is under stress, adjustments are made such as giving priority to displaying easy-to-prepare menu items. The inputs are the emotional state and the generated menu list, and the output is the menu list with adjusted priority.

[0674] Step 6:

[0675] The cloud server sends the menu list with adjusted priorities to the user's device. A dedicated application displays the menu list in a visually easy-to-understand format. For example, cooking time and calorie information may be presented to the user in card format. The input is the menu list with adjusted priorities, and the output is the device screen that displays it.

[0676] Step 7:

[0677] Based on the menu selected by the user, the cloud server generates a list of ingredients that are missing. This list is generated by comparing it with the ingredients the user owns. The input is the list of ingredients the user owns and the selected menu, and the output is the list of ingredients that are missing.

[0678] Step 8:

[0679] The cloud server automatically places orders with a specified supply source (e.g., an online supermarket) based on the list of ingredients that are in short supply. This order is scheduled to coincide with the user's return time. The inputs are the list of ingredients that are in short supply and the user's return time, and the output is the order data sent to the supply source.

[0680] Step 9:

[0681] If the user does not have time to cook or is tired from work, the cloud server arranges for cooked food from affiliated food service providers. The input is the user's emotional state and cooking availability, and the output is the order and delivery request data for the cooked food.

[0682] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0683] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0684] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0685] [Third embodiment]

[0686] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0687] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0688] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0689] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0690] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0691] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0692] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0693] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0694] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0695] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0696] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0697] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0698] This invention is a system that allows users to take videos of ingredients in their refrigerators or storage areas, analyzes the videos, and automatically creates nutritionally balanced menus. This system operates in cooperation with a server, terminals, and users to provide convenience and health to users.

[0699] First, the user uses a device such as a smartphone or tablet to record video of the inside of the refrigerator or storage unit. A dedicated app is installed on the device, and the video is sent to the server using this app. The video is sent via a communication line.

[0700] The server has a program that analyzes the received video. The server identifies and lists the items (ingredients and seasonings) in the video. Based on this list, the server generates multiple menus that take nutritional balance into consideration. Specifically, it generates combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also added to each menu.

[0701] The server then sends the generated menu information to the terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to select. The user selects the appropriate menu from the multiple menus presented and checks its details. If there are any ingredients or seasonings required for the menu that are missing from the refrigerator or storage, the server identifies them and lists them.

[0702] With the user's permission, the server orders missing ingredients and seasonings from a designated source (such as a local online supermarket). When the order is placed, it schedules delivery to coincide with the user's return home time. When the user returns home, the necessary ingredients have been delivered in a timely manner.

[0703] In addition, if the user does not have time to cook or is too tired from work to cook, the server will arrange for pre-cooked meals to be delivered from affiliated restaurants, allowing the user to provide healthy meals for themselves and their family.

[0704] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via the app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Any "miso" that is missing is listed and ordered from the online supermarket in time for the user to return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0705] As described above, the system of the present invention provides comprehensive support for users to efficiently prepare or obtain nutritionally balanced meals, allowing even busy users to easily achieve healthy meals.

[0706] The processing flow will be explained below.

[0707] Step 1:

[0708] Users take videos of the ingredients in their refrigerators or storage areas using a smartphone or tablet, adjusting the camera angle and brightness to ensure all ingredients are clearly visible.

[0709] Step 2:

[0710] The device sends the captured video to a server via a dedicated app, where it performs correction processing to appropriately compress the video data while maintaining image quality.

[0711] Step 3:

[0712] The server receives the video sent from the device and stores it in a database, where it is processed by an analysis program.

[0713] Step 4:

[0714] The server uses an AI-generated model to identify ingredients in the video, detecting items (ingredients and seasonings) in each frame and listing them.

[0715] Step 5:

[0716] Based on the listed ingredient information, the server retrieves nutritional information for each ingredient and available recipes from the database.

[0717] Step 6:

[0718] The server runs a menu generation algorithm that takes nutritional balance into consideration based on the acquired ingredient information and recipe data, generating five combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish).

[0719] Step 7:

[0720] The server adds cooking time and calorie information to each menu item and also generates a detailed list of the ingredients and seasonings required.

[0721] Step 8:

[0722] The server sends the generated menu information to the user's device using a secure communication method.

[0723] Step 9:

[0724] The device displays the menu information received from the server in a dedicated app. The user can check multiple menu options and view detailed information (cooking time, calories, and a list of necessary ingredients).

[0725] Step 10:

[0726] The user selects an appropriate menu from the presented menu options, checks the details, and then lists any missing ingredients and seasonings. The user then authorizes the ordering of missing ingredients within the app.

[0727] Step 11:

[0728] With the user's permission, the server automatically orders the listed ingredients from a designated supplier (such as a local online supermarket) and schedules delivery to coincide with the user's return home time.

[0729] Step 12:

[0730] The server obtains the estimated delivery time of the ordered ingredients from the supply source and notifies the user.

[0731] Step 13:

[0732] When the user returns home, the user checks that the necessary ingredients have been delivered and begins cooking based on the presented menu.

[0733] Step 14:

[0734] If the user cannot find time to cook, the user notifies the server that he / she would like to have cooked food delivered from a partner restaurant or diner.

[0735] Step 15:

[0736] The server receives the user's notification, obtains prepared food from affiliated restaurants and other establishments in the area, and arranges for delivery at the time specified by the user.

[0737] Through the above specific processing steps, the present invention helps users to efficiently prepare nutritionally balanced meals.

[0738] Example 1

[0739] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0740] In today's busy lifestyles, it is difficult for users to easily decide on nutritionally balanced meals and procure the necessary ingredients in a timely manner. In particular, users who are busy with work or housework cannot afford to take the time to identify shortages of ingredients and replenish them, making it difficult to prepare healthy meals. Furthermore, if they cannot find time to cook, they have to go through the hassle of arranging pre-cooked meals as an alternative.

[0741] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0742] In this invention, the server includes a means for receiving a video and recognizing items in the video, a means for generating multiple combinations based on the recognized items, and a means for identifying missing items. It also includes a means for automatically analyzing each frame of the captured video and identifying items, a means for automatically generating a nutritionally balanced menu based on the analyzed list of items, and a means for displaying the generated menu information and allowing the user to select from it. This allows the user to easily decide on a nutritionally balanced menu and procure the necessary ingredients in a timely manner. Furthermore, if cooking time is unavailable, pre-cooked meals can be quickly arranged.

[0743] "Means for receiving video and recognizing objects in the video" refers to a technology that sends a video taken by a user to a server and automatically recognizes objects that appear in the video through analysis.

[0744] The "means for generating a plurality of combinations based on recognized items" is a technology for automatically generating a plurality of nutritionally balanced menus based on a list of recognized items.

[0745] The "means for identifying missing items" is a technology for automatically finding items required for the generated menu that are not in the refrigerator or storage.

[0746] "Means for ordering from a specified source of supply" refers to technology that automatically places an order for the missing items with a specific supplier (for example, an online supermarket or a retail store).

[0747] "Means for analyzing each frame and automatically identifying items" refers to a technology for breaking down a video into individual frames and analyzing the items shown in each frame.

[0748] The "means for automatically generating menus that take nutritional balance into consideration" is a technology for generating multiple menus that take nutritional balance into consideration based on an analyzed item list.

[0749] "Means for displaying menu information and allowing the user to select" refers to technology that displays automatically generated menu information on the user's terminal and allows the user to select their preferred menu from the information.

[0750] This invention is a system that allows users to take videos of ingredients in their refrigerators or storage areas, analyzes the videos, and automatically creates nutritionally balanced menus. This system operates in cooperation with a server, terminals, and users to provide convenience and health to users.

[0751] First, the user uses a device such as a smartphone or tablet to take a video of the inside of the refrigerator or storage unit. A dedicated app is installed on the device, and the taken video is sent to the server using this app. The video is sent via a communication line. For example, the user takes a video of the inside of the refrigerator and sends it to the server via the dedicated app.

[0752] The server has a program that analyzes the received video. Using a video analysis program (e.g., Azure Video Analyzer or Google Cloud Video Intelligence), it analyzes the captured video frame by frame and automatically identifies items (ingredients and seasonings) in the video. Based on the analyzed list of items, the server generates a menu that takes nutritional balance into consideration. Specifically, it uses a generative AI model (e.g., OpenAI GPT-4) to combine a staple food (rice, bread, noodles) with three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also included with each menu.

[0753] Next, the server sends the generated menu information to the terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to make a selection. The user selects the appropriate menu from the multiple menus presented and checks its details. For example, the terminal may display "Main dish: rice, Main: teriyaki chicken thigh, Soup: miso soup, Side dish: stir-fried cabbage and carrots," with each menu item provided along with calorie information and cooking time.

[0754] If there are any ingredients or seasonings needed for a meal that are missing from the refrigerator or storage, the server will identify them and list them. With the user's permission, the server will order the missing ingredients or seasonings from a designated source (such as a local online supermarket). When ordering, the server will schedule delivery to coincide with the user's return home time. For example, the server will list the missing "miso" and order it from the online supermarket to coincide with the user's return home time.

[0755] In addition, if the user cannot find time to cook or is too tired from work to cook, the server will arrange for pre-cooked meals to be delivered from affiliated restaurants. The user can notify the app that they are having difficulty cooking and select pre-cooked meals as an alternative. For example, if the user inputs into the app, "I'm tired today, so I'd like pre-cooked meals delivered," the server will arrange for the above menu to be delivered from an affiliated restaurant.

[0756] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via a dedicated app. The server analyzes the received video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Any "miso" that is missing is listed and ordered from the online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0757] In this way, the system of the present invention provides comprehensive support for users to efficiently prepare or obtain nutritionally balanced meals, allowing even busy users to easily achieve healthy meals.

[0758] Example prompt sentence:

[0759] "Create a system that allows users to take a video of the ingredients in their refrigerator, analyzes the video, and automatically generates a nutritionally balanced meal plan."

[0760] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0761] Step 1: Record and send video

[0762] Subject: User

[0763] Users can launch a dedicated app on their smartphone or tablet and take video of the inside of their refrigerator or storage unit. Once the video is complete, the user can send the video to the server via the app.

[0764] Input: Video data taken inside a refrigerator or storage facility

[0765] Output: Video data sent to the server

[0766] Specific operation: The user opens the refrigerator door, takes sequential photos of the inside with their smartphone camera, and taps the "Send" button to upload the video data to the server.

[0767] Step 2: Receiving and analyzing the video

[0768] Subject: Server

[0769] The server receives the video sent by the user and uses a video analysis program to analyze the video frame by frame and identify the items (ingredients and seasonings) that appear in the video.

[0770] Input: Video data sent by the user

[0771] Output: List of recognized objects

[0772] How it works: The server runs a video analysis program, breaks down the video into frames, and automatically identifies and lists the ingredients and seasonings. For example, a list of items such as "chicken thighs, cabbage, and carrots" is obtained.

[0773] Step 3: Menu Creation

[0774] Subject: Server

[0775] The server generates a menu that takes nutritional balance into consideration based on the list of items obtained through the analysis. Using a generative AI model, multiple menu plans are created. Each menu includes a main dish and three side dishes (soup, main course, and garnish), and cooking time and calorie information are also included.

[0776] Input: A list of recognized items (e.g., "chicken thigh, cabbage, carrot")

[0777] Output: Multiple menu plans that take nutritional balance into consideration

[0778] Specific operation: The server runs the AI ​​model and generates a menu suggestion such as "rice, teriyaki chicken thighs, miso soup, and stir-fried cabbage and carrots."

[0779] Step 4: Display menu information

[0780] Subject: Terminal

[0781] The terminal receives the menu information sent from the server and displays it on the screen in an easy-to-read format. The user can then select an appropriate menu from multiple suggested menus.

[0782] Input: Menu information sent from the server

[0783] Output: Menu information displayed on the device

[0784] Specific operation: The device displays menu suggestions such as "Main dish: rice, Main course: teriyaki chicken thigh, Soup: miso soup, Side dish: stir-fried cabbage and carrots," and the user can choose from them.

[0785] Step 5: Identify and order missing ingredients

[0786] Subject: Server

[0787] The server automatically lists any missing items based on the selected menu, orders missing ingredients and seasonings from a designated supplier (such as an online supermarket), and schedules delivery to coincide with the user's return time.

[0788] Input: Selected menu plan, user's current ingredients list

[0789] Output: List of ingredients to order, order information

[0790] Specific operation: The server lists ingredients such as miso that are in short supply, and after receiving confirmation from the user, automatically places an order with the online supermarket. The delivery time is set to coincide with the user's return home.

[0791] Step 6: Arrange for prepared meals

[0792] Subject: Server

[0793] If the user cannot find time to cook, the server will arrange for the delivery of cooked food from affiliated restaurants or diner.

[0794] Input: User notification of cooking difficulty

[0795] Output: Arrange for delivery of cooked food

[0796] Specific operation: The user enters "I can't cook today" through the app, and the server arranges for the following meal from a partner restaurant: "Main dish: rice, Main course: teriyaki chicken thigh, Soup: miso soup, Side dish: stir-fried cabbage and carrots" and has it delivered.

[0797] (Application example 1)

[0798] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0799] In today's busy lifestyles, it is difficult for users to prepare healthy, nutritionally balanced meals. Managing the contents of the refrigerator and storage areas and planning menus based on that information is also a time-consuming task. Furthermore, if necessary ingredients are in short supply, it is inefficient to go shopping every time. To solve these problems, a system is needed that allows users to easily manage ingredients, create menus, and order missing ingredients.

[0800] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0801] In this invention, the server includes a means for receiving a video and recognizing items in the video, a means for generating a plurality of combinations based on the recognized items, and a means for identifying missing items. This automatically lists ingredients in the refrigerator or storage area from the video taken by the user, and generates a nutritionally balanced menu based on that list. Furthermore, the automatic identification of missing items allows online ordering from a virtual store, and delivery of ingredients or cooked meals can be arranged to coincide with the user's return home time. This allows users to prepare and obtain healthy meals without hassle.

[0802] The "means for receiving video and recognizing items in the video" is a technology for receiving video shot by a user and analyzing and identifying items that appear in the video.

[0803] The "means for generating multiple combinations based on recognized items" is a technology for creating multiple menus based on information about identified items, taking into consideration nutritional balance, cooking methods, etc.

[0804] The "means for identifying missing items" is a technology for comparing the recognized items with the items required for the generated menu and listing the missing items.

[0805] The "means for ordering the missing items from a predetermined supply source" is a technique for ordering the identified missing items from a related supply source (for example, an online supermarket or an affiliated store).

[0806] The "means for generating nutritionally balanced menus" is a technology for creating multiple balanced menus by taking into consideration the nutritional components of recognized items.

[0807] "Means for sending the generated menu information to the user terminal and displaying it so that the user can select" refers to a technology for sending the generated menu data to the user's device and displaying it so that the user can select the menu they desire.

[0808] "Means for placing online orders at a virtual store" refers to technology that allows users to easily place orders at a virtual store (e.g., an online shopping site) based on a list of items they are missing.

[0809] This system allows users to take videos of ingredients in their refrigerators or storage areas, analyzes the videos, and automatically creates nutritionally balanced meals. The system operates in cooperation with the server, terminals, and users to provide convenience and health to users.

[0810] First, the user uses a device such as a smartphone or tablet to record video of the inside of the refrigerator or storage unit. A dedicated application is installed on the device, and the video is sent to the server using this application. The video is sent via a communication line.

[0811] The server has a program that analyzes the received video. This analysis uses the OpenCV library for image analysis and TensorFlow for ingredient recognition. The server identifies and lists the items (ingredients and seasonings) in the video. Based on this list, the server uses a generative AI model to generate multiple menus that take nutritional balance into consideration. Specifically, it generates combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also included with each menu.

[0812] The server then sends the generated menu information to the terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to select. The user selects the appropriate menu from the multiple menus presented and checks its details. If there are any ingredients or seasonings required for the menu that are missing from the refrigerator or storage, the server identifies them and lists them.

[0813] With the user's permission, the server orders missing ingredients and seasonings from a designated supply source (such as a local online supermarket). When the order is placed, it schedules delivery to coincide with the user's return time. This ensures that the necessary ingredients have arrived in a timely manner when the user returns home. In addition, if the user is unable to find time to cook or is too tired from work to cook, the server will arrange for pre-cooked food to be delivered from an affiliated restaurant.

[0814] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via the app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Any "miso" that is missing is listed and ordered from the online supermarket in time for the user to return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0815] An example of a prompt for a generative AI model is as follows:

[0816] "Analyze the video of the contents of the refrigerator and create a list of recognized ingredients. Furthermore, use that data to generate a nutritionally balanced meal plan, identifying and listing any missing ingredients. Display the generated meal plan in the following format: 'Main dish: Teriyaki chicken thigh, Side dish: Cabbage salad, Another side dish: Kinpira carrots'. If any ingredients are missing, provide a function that allows the user to order them from a virtual store."

[0817] The system allows users to prepare and consume healthy meals hassle-free.

[0818] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0819] Step 1:

[0820] Users use devices such as smartphones or tablets to record video of the inside of refrigerators or storage units. A dedicated application is installed on the device, and the video recorded using this application is sent to a server. The input is video data recording the inside of the refrigerator or storage unit, and the output is the video data sent to the server. The device sends the video data via a communication line.

[0821] Step 2:

[0822] To analyze the received video, the server uses the OpenCV library to break down the video data into frames and perform image analysis on each frame. Based on the recognition results, TensorFlow is used to identify ingredients. The input is the image data broken down into frames, and the output is a list of recognized ingredients. Specifically, the server resizes the images of each frame to 224 x 224 pixels and inputs them into the ingredient recognition model.

[0823] Step 3:

[0824] The server uses a generative AI model to generate multiple nutritionally balanced menus based on the list of recognized ingredients. The input is the list of recognized ingredients, and the output is a list of generated menus. Specifically, it takes into account the combination of staple food, side dish, and soup, and also adds cooking time and calorie information to each menu.

[0825] Step 4:

[0826] The server sends the generated menu information to the user terminal. The input is a list of the generated menus, and the output is the menu information sent to the user terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to select an appropriate menu from it.

[0827] Step 5:

[0828] The user selects the appropriate menu from the multiple menus presented and checks its details. The input is the menu information sent from the server, and the output is the details of the menu selected by the user. The selected menu includes the necessary ingredients and seasonings.

[0829] Step 6:

[0830] The server identifies and lists the items that are missing from the refrigerator or storage based on the selected menu. The input is the selected menu and the current status of the refrigerator or storage, and the output is a list of the missing items.

[0831] Step 7:

[0832] With the user's permission, the server orders missing ingredients and seasonings from a designated source (e.g., a local online supermarket). The input is a list of missing items, and the output is order confirmation information. When placing an order, the server schedules delivery to coincide with the user's return home time.

[0833] Step 8:

[0834] If the user cannot find the time to cook, the server will arrange for the delivery of cooked food from a partner restaurant. The inputs are the user's cooking time and the partner restaurant's menu, and the output is confirmation of the delivery of the cooked food.

[0835] This allows users to prepare and consume healthy meals without any hassle.

[0836] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0837] This invention is a system that allows users to take videos of ingredients in their refrigerators and storage areas, analyzes the videos, and automatically generates nutritionally balanced menus. By combining this invention with an emotion engine that recognizes the user's emotions, it is possible to adjust the menu suggestions according to the user's emotional state. This allows for meal suggestions that are tailored to the user's mental state and preferences.

[0838] First, the user uses a device such as a smartphone or tablet to record video of the inside of the refrigerator or storage unit. A dedicated app is installed on the device, and the video is sent to the server using this app. The video is sent via a communication line.

[0839] The server has a program that analyzes the received video. The server identifies and lists the items (ingredients and seasonings) in the video. Based on this list, the server generates multiple menus that take nutritional balance into consideration. Specifically, it generates combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also added to each menu.

[0840] The server then analyzes the user's current emotional state using an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions based on, for example, the text the user types into the device, the tone of voice, or changes in facial expressions. This emotion information is used to adjust the menu suggestions. For example, if the user is feeling stressed, the server can prioritize easy-to-prepare menus.

[0841] The server optimizes the priority of the menu suggestions based on the generated menu information and emotion information and sends them to the device. The device then displays the received menu information in an easy-to-read format, allowing the user to select. The user selects an appropriate menu from the multiple menus presented and checks its details. If there are any ingredients or seasonings required for the menu that are missing from the refrigerator or storage, the server identifies them and lists them.

[0842] With the user's permission, the server orders missing ingredients and seasonings from a designated source (such as a local online supermarket). When the order is placed, it schedules delivery to coincide with the user's return home time. When the user returns home, the necessary ingredients have been delivered in a timely manner.

[0843] In addition, if the user does not have time to cook or is too tired from work to cook, the server will arrange for pre-cooked meals to be delivered from affiliated restaurants, allowing the user to provide healthy meals for themselves and their family.

[0844] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via the app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Furthermore, an emotion engine recognizes the user's stress level and prioritizes menu options that are easy to prepare. Any "miso" that is in short supply is listed and ordered from an online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0845] As described above, the system of the present invention provides comprehensive support for users to efficiently prepare or obtain nutritionally balanced meals, enabling users to easily and effectively achieve healthy eating habits even in busy daily lives.

[0846] The processing flow will be explained below.

[0847] Step 1:

[0848] Users take videos of the ingredients in their refrigerators or storage areas using a smartphone or tablet, adjusting the camera angle and brightness to ensure all ingredients are clearly visible.

[0849] Step 2:

[0850] The device sends the captured video to the server via a dedicated app. Before sending, the device encodes the video and compresses it appropriately.

[0851] Step 3:

[0852] The server receives the video sent from the device and stores it in a database. The received video is then processed by an analysis program.

[0853] Step 4:

[0854] The server uses an AI-generated model to identify ingredients in the video, detecting them in each frame and listing them.

[0855] Step 5:

[0856] Based on the listed ingredient information, the server retrieves nutritional information and available recipes for each ingredient from the database.

[0857] Step 6:

[0858] The server runs a menu generation algorithm that takes nutritional balance into consideration based on the acquired ingredient information and recipe data, generating five combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish).

[0859] Step 7:

[0860] The server adds cooking time and calorie information to each menu, and also generates a detailed list of ingredients and seasonings required.The server also uses an emotion engine to analyze the user's emotional state.

[0861] Step 8:

[0862] The server adjusts its menu suggestions based on the user's emotional state: for example, if the user is feeling stressed, it will prioritize easy-to-prepare meals.

[0863] Step 9:

[0864] The server then transmits the optimized menu information, taking into account the user's emotional state, to the user's device using secure communication methods.

[0865] Step 10:

[0866] The device displays the menu information received from the server in a dedicated app. The user can check multiple menu options and view detailed information (cooking time, calories, and a list of necessary ingredients).

[0867] Step 11:

[0868] The user selects an appropriate menu from the presented options and checks the details. After making a selection, the system lists any missing ingredients and seasonings.

[0869] Step 12:

[0870] The user can authorize the app to order missing ingredients by tapping the order authorization button.

[0871] Step 13:

[0872] With the user's permission, the server automatically orders the missing ingredients from a designated supply source (such as a local online supermarket). When placing an order, the server sets a delivery schedule to match the user's return time.

[0873] Step 14:

[0874] The server obtains the estimated delivery time of the ordered ingredients from the supply source and notifies the user.

[0875] Step 15:

[0876] When the user returns home, the user checks that the necessary ingredients have been delivered and begins cooking based on the presented menu.

[0877] Step 16:

[0878] If the user cannot find time to cook, the user notifies the server that he / she would like to have cooked food delivered from a partner restaurant or diner.

[0879] Step 17:

[0880] The server receives the user's notification, obtains prepared food from affiliated restaurants and other establishments in the area, and arranges for delivery at the time specified by the user.

[0881] Through the above specific processing steps, the present invention supports users in efficiently preparing nutritionally balanced meals, and further utilizes an emotion engine to make optimal suggestions based on the user's emotional state.

[0882] Example 2

[0883] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0884] In today's busy lifestyles, people often lack the time and effort to prepare balanced meals. Furthermore, there are no systems that suggest meals that take into account stress and fatigue, making it difficult for users to easily find menus that suit their individual circumstances. Furthermore, if necessary ingredients or seasonings are missing, there is no efficient way to supplement them, which is a time-consuming process.

[0885] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a video and recognizing items in the video, means for generating a food list based on the recognized items, means for generating a menu taking nutritional balance into consideration, means for analyzing the user's emotions and adjusting the menu based on the emotion information, means for identifying missing items, and means for ordering the missing items from a predetermined supply source. This allows users to efficiently prepare nutritionally balanced meals and easily find appropriate menus based on their emotional state, even in the midst of a busy lifestyle. Furthermore, the ordering process for missing ingredients and seasonings can be automated, significantly reducing the effort required for meal preparation.

[0886] The "means for receiving video" is a function for transmitting video data taken by the user to a server via a dedicated application and for the server to receive the video data.

[0887] "Means for recognizing items" is a function that uses video analysis technology to identify and list items such as ingredients and seasonings that appear in the received video data.

[0888] The "means for generating a food list" is a function for creating a list of ingredients currently stored in the refrigerator or storage area based on information about items recognized through video analysis.

[0889] The "means for generating menus" is a function for automatically generating multiple menu plans based on the recognized food list, taking into consideration nutritional balance.

[0890] The "means for analyzing emotions" is a function for analyzing emotional information in text, audio, or video input by the user and identifying the user's current emotional state.

[0891] The "means for adjusting the menu" is a function for adapting and optimizing the temporarily generated menu plan to the user's emotional state based on the analyzed emotional information.

[0892] The "means for identifying missing items" is a function for identifying the ingredients and seasonings required for the generated menu that are currently missing from the refrigerator or storage.

[0893] The "means for ordering from a predetermined supply source" is a function for automatically ordering missing items from a supply source such as an online supermarket after identifying the items that are in short supply.

[0894] This system allows users to take videos of ingredients in their refrigerators and storage areas, analyzes the videos, and automatically generates nutritionally balanced menus. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, this system can adjust the menu suggestions according to the user's emotional state.

[0895] System configuration

[0896] This system starts when the user uses a device such as a smartphone or tablet to record video of the inside of a refrigerator or storage unit. A dedicated app is installed on the device, and the video recorded using this app is sent to a server. The video is sent via an internet communication line.

[0897] Video Analysis

[0898] The server has a program for analyzing the received video. In this case, OpenCV or TensorFlow can be used as video analysis software. The server identifies items in the video, specifically ingredients and seasonings. This identification can be performed using image recognition technology or machine learning models.

[0899] Ingredient list generation and menu suggestions

[0900] The server generates a food list based on the identified ingredients. This list is stored in a database, and nutritional calculation software is used to automatically generate multiple balanced meal plans. These plans include a combination of a main dish and three side dishes (soup, main course, and side dish). Cooking time and calorie information are also included for each plan.

[0901] Emotion analysis

[0902] The server then uses an emotion engine to analyze the user's emotional state. This analysis can be performed using, for example, IBM Watson Tone Analyzer or Microsoft Azure Emotion API. The emotion engine analyzes the text and voice data entered by the user into the device, or changes in facial expressions captured using the camera. The optimal menu is then selected from the suggested menus based on the user's emotional state. For example, if the user is feeling stressed, it will suggest dishes that are easy to prepare and require little time to prepare.

[0903] Sending and displaying menu information

[0904] The server sends the optimized menu information to the terminal. The terminal displays the received information in an easy-to-read format, allowing the user to easily select their preferred menu. The user selects the desired menu from the suggested menu and checks its details. If there are any items missing from the refrigerator or storage at this point, the server automatically lists them and places an order with a designated supply source (for example, a local online supermarket).

[0905] Automated ordering and delivery

[0906] When placing an order, the delivery is scheduled to coincide with the user's return time, and when the user returns home, the necessary ingredients are delivered in a timely manner.

[0907] Prepared food delivery

[0908] In addition, if the user does not have time to cook or is too tired from work to cook, the server will arrange for cooked food to be delivered from affiliated restaurants, allowing the user to provide nutritionally balanced meals for themselves and their family.

[0909] Specific examples

[0910] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via a dedicated app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as the side dish. Furthermore, an emotion engine recognizes the user's stress level and prioritizes menu options that are easy to prepare. Any "miso" that is in short supply is listed and ordered from an online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0911] Prompt Sentence Examples

[0912] "Please suggest a nutritionally balanced meal using the ingredients in your refrigerator. Currently, the refrigerator contains chicken thighs, cabbage, and carrots. The user is currently feeling stressed, so would like a recipe that is easy to prepare."

[0913] As described above, the system of the present invention allows users to efficiently prepare nutritionally balanced meals and easily find appropriate menus according to their emotional state. Furthermore, by automating the ordering process for ingredients and seasonings that are in short supply, it is possible to significantly reduce the effort required for meal preparation in users' daily lives.

[0914] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0915] Step 1:

[0916] The user takes a video of the refrigerator using the device.

[0917] Input: A user uses a smartphone or tablet to record a video of the inside of a refrigerator.

[0918] Output: Video data of the inside of the refrigerator.

[0919] Specific operation: The user launches the dedicated app and shoots a video that captures all the food in the refrigerator.

[0920] Step 2:

[0921] The video data is sent to the server.

[0922] Input: Video data taken inside a refrigerator.

[0923] Output: Video data sent to the server.

[0924] Specific operation: After the user has finished shooting, they use a dedicated app to send the video they have shot to a server via internet communication.

[0925] Step 3:

[0926] The server analyzes the video and generates a list of ingredients.

[0927] Input: Video data sent to the server.

[0928] Output: A list of recognized ingredients.

[0929] How it works: The server uses video analysis software to apply image recognition technology to identify ingredients in the video. Specifically, it uses OpenCV and TensorFlow to identify ingredients and generate a food list.

[0930] Step 4:

[0931] The server automatically generates the menu.

[0932] Input: A list of recognized ingredients.

[0933] Output: A list of generated menus.

[0934] What it does: The server uses nutrition calculation software to generate multiple nutritionally balanced meal plans based on the recognized ingredients, using the nutrition information and recipes stored in the database for this process.

[0935] Step 5:

[0936] Recognize user emotions with an emotion engine.

[0937] Input: User text, voice, or facial expression data.

[0938] Output: Parsed emotional state.

[0939] How it works: The server uses an emotion recognition engine to analyze the text, voice data, and facial expressions entered by the user on the device. Examples include IBM Watson Tone Analyzer and Microsoft Azure Emotion API.

[0940] Step 6:

[0941] Optimize the menu suggested by the server.

[0942] Input: Generated menu list and parsed emotional state.

[0943] Output: A list of optimized menus that match the user's emotions.

[0944] Specific operation: The server proposes menus that suit the user's emotions based on the generated menu information and emotional information. For example, if the user is feeling stressed, it will prioritize suggestions for easy-to-prepare dishes.

[0945] Step 7:

[0946] Optimized menu information is sent to the device.

[0947] Input: A list of optimized menu items.

[0948] Output: Menu information sent to the device.

[0949] Specific operation: The server sends the optimized menu information to the terminal in real time. This sending process is performed in data formats such as JSON and XML.

[0950] Step 8:

[0951] The user checks and selects the menu.

[0952] Input: Multiple menu items displayed on the terminal.

[0953] Output: The selected menu.

[0954] Specific operation: The user checks the menu information received on the device and intuitively swipes and taps to select the desired menu.

[0955] Step 9:

[0956] The server identifies the ingredients that are in short supply and processes the order.

[0957] Input: Selected menu and list of ingredients in the refrigerator.

[0958] Output: A list of ingredients that are in short supply and have been ordered from the online supermarket.

[0959] Specific operation: Based on the selected menu, the server identifies the ingredients that are missing and automatically processes the order via the online supermarket API.

[0960] Step 10:

[0961] Arrange for delivery of prepared meals as needed.

[0962] Input: Information about the user's cooking time and status.

[0963] Output: Cooked food ordered from partner restaurant.

[0964] Specific operation: If the user does not have time to cook or is too tired to cook, the server will use the online ordering system of a partner restaurant to send the user's address and order details and arrange for delivery.

[0965] By following these steps, the user can prepare a nutritionally balanced meal without much effort.

[0966] (Application example 2)

[0967] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0968] While conventional meal suggestion systems could automatically generate menus that took into account the ingredients and nutritional balance of the user's inventory, they were unable to suggest menus that reflected the user's emotional state. Furthermore, there was no centralized system for purchasing ingredients when the user was short of them or when they could not find the time to cook, which often caused stress for users. Another issue was that the suggested menu options were visually difficult to understand.

[0969] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0970] In this invention, the server includes means for receiving a video and recognizing items in the video, means for generating a plurality of combinations based on the recognized items, means for recognizing the user's emotional state, means for adjusting the priority of the combinations to be proposed based on the emotional state, means for identifying missing items, means for ordering the missing items from a predetermined supply source, and means for displaying the generated menu information in an easy-to-view format to allow the user to select. This allows the server to propose optimal menus based on the user's emotional state and available ingredients, and also allows for centralized management of purchasing missing ingredients and delivery of cooked meals, enabling the user to easily prepare or obtain nutritionally balanced meals without stress.

[0971] "Means for receiving video and recognizing items in the video" refers to a technology that acquires video data sent from the user's device and analyzes and identifies the items (ingredients and seasonings) that appear in the video.

[0972] The "means for generating multiple combinations based on recognized items" is a technology that automatically generates menu combinations that take into account diverse nutritional balances based on analyzed item information.

[0973] "Means for recognizing the user's emotional state" refers to technology that analyzes emotions from the user's input, tone of voice, changes in facial expression, etc., and identifies that state.

[0974] The "means for adjusting the priority of proposed combinations based on the emotional state" is a technology that dynamically changes the display priority of the most suitable combination from among the generated menus, taking into account the user's emotional state.

[0975] The "means for identifying missing items" is a technology that automatically detects items that are necessary for the generated menu but are not present in the user's inventory of ingredients.

[0976] The "means for ordering the missing items from a predetermined supply source" is a technology for automatically ordering the detected missing ingredients from a supply source such as an online store or supermarket.

[0977] "Means for displaying generated menu information in an easy-to-see format and allowing users to make selections" refers to a technology that displays a list of suggested menus in a visually easy-to-understand format on the user's terminal, providing an interface that allows users to easily make selections.

[0978] The present invention is a system that allows users to take videos of ingredients in their refrigerators and storage areas, analyzes the videos, and automatically generates nutritionally balanced menus. Furthermore, by adjusting the menu suggestions according to the user's emotional state, it is possible to provide meal suggestions that take into account the user's mental state and preferences.

[0979] First, a user uses a device such as a smartphone or tablet to record video of the inside of a refrigerator or storage unit. A dedicated application is installed on the device, and the video is sent to a cloud server using this application. The video is sent using a communication line such as the Internet.

[0980] Next, a video analysis program runs on the cloud server. Specifically, it uses video analysis engines such as OpenCV and TensorFlow to identify and list items (ingredients and seasonings) in the video. Based on the results of this analysis, multiple menus are generated that take nutritional balance into consideration. Combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, garnish) are generated, and cooking time and calorie information is also added to each menu.

[0981] After ingredient recognition is complete, an emotion engine is activated to recognize the user's emotional state. This emotion engine uses, for example, the Google Cloud Natural Language API or the Microsoft Azure Emotion API to analyze emotions based on the text the user enters into their device, their tone of voice, or changes in facial expressions. Based on this emotional information, the suggested meal plans are adjusted. For example, if the user is feeling stressed, the system can prioritize easy-to-prepare meals.

[0982] Next, the cloud server optimizes the priority of the menu suggestions based on the generated menu information and emotion information, and sends the results to the user's device. At this time, the generated menu information is displayed visually in an easy-to-understand user interface, allowing the user to easily select a menu. For example, a list of menus may be displayed in card format, with each card showing cooking time, calories, and icons of the ingredients used.

[0983] Furthermore, if any of the ingredients or seasonings required for the user's selected meal are missing from the refrigerator or storage, the cloud server will identify them and automatically list them. With the user's permission, the cloud server will order the missing items from a designated supplier (such as a local online supermarket). When the order is placed, it will be scheduled to be delivered by the user's return home time.

[0984] In addition, if the user cannot find time to cook or is too tired from work to cook, the cloud server can also arrange for cooked food to be delivered from affiliated food providers (such as set meal restaurants).

[0985] As a specific example, a user takes a video of the contents of their refrigerator with their smartphone and sends it to a cloud server via an application. The cloud server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Furthermore, an emotion engine recognizes the user's stress level and prioritizes menu options that are easy to prepare. Any "miso" that is in short supply is listed and ordered from an online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[0986] Example prompt sentence:

[0987] "What should I make for dinner today? I'm busy and stressed at work, so I'd like something easy to make."

[0988] In this way, the system of the present invention proposes optimal menus based on the user's emotional state and the ingredients they have on hand, and can also centrally manage the purchase of missing ingredients and the delivery of cooked meals, allowing users to easily prepare or obtain nutritionally balanced meals without feeling stressed.

[0989] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0990] Step 1:

[0991] The user uses a device to record video of the food in the refrigerator or storage area. Once the video is complete, the user uses a dedicated application to send the video data to a cloud server. The input is high-resolution video data recorded by the device, and the output is this video data sent to the cloud server.

[0992] Step 2:

[0993] The cloud server analyzes the received video data and identifies the ingredients in the video. This process is performed using OpenCV and TensorFlow. Specifically, the video is broken down into frames, and the items (ingredients and seasonings) in each frame are recognized based on a model. The input is the video data sent by the user, and the output is a list of identified ingredients.

[0994] Step 3:

[0995] The cloud server generates multiple nutritionally balanced menus based on the identified ingredient list, including information such as the main dish, several side dishes, total calories, and cooking time. The input is the identified ingredient list, and the output is the generated menu list.

[0996] Step 4:

[0997] Next, the server recognizes the user's emotional state. Emotion recognition is achieved using the user's text input, voice input, or facial expression recognition. Technologies used include Google Cloud Natural Language API and Microsoft Azure Emotion API. The input includes text, audio, and video data that indicate the user's emotions, and the output is the analyzed emotional state.

[0998] Step 5:

[0999] The cloud server adjusts the priority of the generated menu items based on the recognized emotional state. For example, for a user who is under stress, adjustments are made such as giving priority to displaying easy-to-prepare menu items. The inputs are the emotional state and the generated menu list, and the output is the menu list with adjusted priority.

[1000] Step 6:

[1001] The cloud server sends the menu list with adjusted priorities to the user's device. A dedicated application displays the menu list in a visually easy-to-understand format. For example, cooking time and calorie information may be presented to the user in card format. The input is the menu list with adjusted priorities, and the output is the device screen that displays it.

[1002] Step 7:

[1003] Based on the menu selected by the user, the cloud server generates a list of ingredients that are missing. This list is generated by comparing it with the ingredients the user owns. The input is the list of ingredients the user owns and the selected menu, and the output is the list of ingredients that are missing.

[1004] Step 8:

[1005] The cloud server automatically places orders with a specified supply source (e.g., an online supermarket) based on the list of ingredients that are in short supply. This order is scheduled to coincide with the user's return time. The inputs are the list of ingredients that are in short supply and the user's return time, and the output is the order data sent to the supply source.

[1006] Step 9:

[1007] If the user does not have time to cook or is tired from work, the cloud server arranges for cooked food from affiliated food service providers. The input is the user's emotional state and cooking availability, and the output is the order and delivery request data for the cooked food.

[1008] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1009] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1010] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1011] [Fourth embodiment]

[1012] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1013] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1014] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1015] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1016] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1017] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1018] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1019] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1020] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1021] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1022] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1023] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1024] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1025] This invention is a system that allows users to take videos of ingredients in their refrigerators or storage areas, analyzes the videos, and automatically creates nutritionally balanced menus. This system operates in cooperation with a server, terminals, and users to provide convenience and health to users.

[1026] First, the user uses a device such as a smartphone or tablet to record video of the inside of the refrigerator or storage unit. A dedicated app is installed on the device, and the video is sent to the server using this app. The video is sent via a communication line.

[1027] The server has a program that analyzes the received video. The server identifies and lists the items (ingredients and seasonings) in the video. Based on this list, the server generates multiple menus that take nutritional balance into consideration. Specifically, it generates combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also added to each menu.

[1028] The server then sends the generated menu information to the terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to select. The user selects the appropriate menu from the multiple menus presented and checks its details. If there are any ingredients or seasonings required for the menu that are missing from the refrigerator or storage, the server identifies them and lists them.

[1029] With the user's permission, the server orders missing ingredients and seasonings from a designated source (such as a local online supermarket). When the order is placed, it schedules delivery to coincide with the user's return home time. When the user returns home, the necessary ingredients have been delivered in a timely manner.

[1030] In addition, if the user does not have time to cook or is too tired from work to cook, the server will arrange for pre-cooked meals to be delivered from affiliated restaurants, allowing the user to provide healthy meals for themselves and their family.

[1031] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via the app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Any "miso" that is missing is listed and ordered from the online supermarket in time for the user to return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[1032] As described above, the system of the present invention provides comprehensive support for users to efficiently prepare or obtain nutritionally balanced meals, allowing even busy users to easily achieve healthy meals.

[1033] The processing flow will be explained below.

[1034] Step 1:

[1035] Users take videos of the ingredients in their refrigerators or storage areas using a smartphone or tablet, adjusting the camera angle and brightness to ensure all ingredients are clearly visible.

[1036] Step 2:

[1037] The device sends the captured video to a server via a dedicated app, where it performs correction processing to appropriately compress the video data while maintaining image quality.

[1038] Step 3:

[1039] The server receives the video sent from the device and stores it in a database, where it is processed by an analysis program.

[1040] Step 4:

[1041] The server uses an AI-generated model to identify ingredients in the video, detecting items (ingredients and seasonings) in each frame and listing them.

[1042] Step 5:

[1043] Based on the listed ingredient information, the server retrieves nutritional information for each ingredient and available recipes from the database.

[1044] Step 6:

[1045] The server runs a menu generation algorithm that takes nutritional balance into consideration based on the acquired ingredient information and recipe data, generating five combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish).

[1046] Step 7:

[1047] The server adds cooking time and calorie information to each menu item and also generates a detailed list of the ingredients and seasonings required.

[1048] Step 8:

[1049] The server sends the generated menu information to the user's device using a secure communication method.

[1050] Step 9:

[1051] The device displays the menu information received from the server in a dedicated app. The user can check multiple menu options and view detailed information (cooking time, calories, and a list of necessary ingredients).

[1052] Step 10:

[1053] The user selects an appropriate menu from the presented menu options, checks the details, and then lists any missing ingredients and seasonings. The user then authorizes the ordering of missing ingredients within the app.

[1054] Step 11:

[1055] With the user's permission, the server automatically orders the listed ingredients from a designated supplier (such as a local online supermarket) and schedules delivery to coincide with the user's return home time.

[1056] Step 12:

[1057] The server obtains the estimated delivery time of the ordered ingredients from the supply source and notifies the user.

[1058] Step 13:

[1059] When the user returns home, the user checks that the necessary ingredients have been delivered and begins cooking based on the presented menu.

[1060] Step 14:

[1061] If the user cannot find time to cook, the user notifies the server that he / she would like to have cooked food delivered from a partner restaurant or diner.

[1062] Step 15:

[1063] The server receives the user's notification, obtains prepared food from affiliated restaurants and other establishments in the area, and arranges for delivery at the time specified by the user.

[1064] Through the above specific processing steps, the present invention helps users to efficiently prepare nutritionally balanced meals.

[1065] Example 1

[1066] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1067] In today's busy lifestyles, it is difficult for users to easily decide on nutritionally balanced meals and procure the necessary ingredients in a timely manner. In particular, users who are busy with work or housework cannot afford to take the time to identify shortages of ingredients and replenish them, making it difficult to prepare healthy meals. Furthermore, if they cannot find time to cook, they have to go through the hassle of arranging pre-cooked meals as an alternative.

[1068] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1069] In this invention, the server includes a means for receiving a video and recognizing items in the video, a means for generating multiple combinations based on the recognized items, and a means for identifying missing items. It also includes a means for automatically analyzing each frame of the captured video and identifying items, a means for automatically generating a nutritionally balanced menu based on the analyzed list of items, and a means for displaying the generated menu information and allowing the user to select from it. This allows the user to easily decide on a nutritionally balanced menu and procure the necessary ingredients in a timely manner. Furthermore, if cooking time is unavailable, pre-cooked meals can be quickly arranged.

[1070] "Means for receiving video and recognizing objects in the video" refers to a technology that sends a video taken by a user to a server and automatically recognizes objects that appear in the video through analysis.

[1071] The "means for generating a plurality of combinations based on recognized items" is a technology for automatically generating a plurality of nutritionally balanced menus based on a list of recognized items.

[1072] The "means for identifying missing items" is a technology for automatically finding items required for the generated menu that are not in the refrigerator or storage.

[1073] "Means for ordering from a specified source of supply" refers to technology that automatically places an order for the missing items with a specific supplier (for example, an online supermarket or a retail store).

[1074] "Means for analyzing each frame and automatically identifying items" refers to a technology for breaking down a video into individual frames and analyzing the items shown in each frame.

[1075] The "means for automatically generating menus that take nutritional balance into consideration" is a technology for generating multiple menus that take nutritional balance into consideration based on an analyzed item list.

[1076] "Means for displaying menu information and allowing the user to select" refers to technology that displays automatically generated menu information on the user's terminal and allows the user to select their preferred menu from the information.

[1077] This invention is a system that allows users to take videos of ingredients in their refrigerators or storage areas, analyzes the videos, and automatically creates nutritionally balanced menus. This system operates in cooperation with a server, terminals, and users to provide convenience and health to users.

[1078] First, the user uses a device such as a smartphone or tablet to take a video of the inside of the refrigerator or storage unit. A dedicated app is installed on the device, and the taken video is sent to the server using this app. The video is sent via a communication line. For example, the user takes a video of the inside of the refrigerator and sends it to the server via the dedicated app.

[1079] The server has a program that analyzes the received video. Using a video analysis program (e.g., Azure Video Analyzer or Google Cloud Video Intelligence), it analyzes the captured video frame by frame and automatically identifies items (ingredients and seasonings) in the video. Based on the analyzed list of items, the server generates a menu that takes nutritional balance into consideration. Specifically, it uses a generative AI model (e.g., OpenAI GPT-4) to combine a staple food (rice, bread, noodles) with three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also included with each menu.

[1080] Next, the server sends the generated menu information to the terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to make a selection. The user selects the appropriate menu from the multiple menus presented and checks its details. For example, the terminal may display "Main dish: rice, Main: teriyaki chicken thigh, Soup: miso soup, Side dish: stir-fried cabbage and carrots," with each menu item provided along with calorie information and cooking time.

[1081] If there are any ingredients or seasonings needed for a meal that are missing from the refrigerator or storage, the server will identify them and list them. With the user's permission, the server will order the missing ingredients or seasonings from a designated source (such as a local online supermarket). When ordering, the server will schedule delivery to coincide with the user's return home time. For example, the server will list the missing "miso" and order it from the online supermarket to coincide with the user's return home time.

[1082] In addition, if the user cannot find time to cook or is too tired from work to cook, the server will arrange for pre-cooked meals to be delivered from affiliated restaurants. The user can notify the app that they are having difficulty cooking and select pre-cooked meals as an alternative. For example, if the user inputs into the app, "I'm tired today, so I'd like pre-cooked meals delivered," the server will arrange for the above menu to be delivered from an affiliated restaurant.

[1083] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via a dedicated app. The server analyzes the received video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Any "miso" that is missing is listed and ordered from the online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[1084] In this way, the system of the present invention provides comprehensive support for users to efficiently prepare or obtain nutritionally balanced meals, allowing even busy users to easily achieve healthy meals.

[1085] Example prompt sentence:

[1086] "Create a system that allows users to take a video of the ingredients in their refrigerator, analyzes the video, and automatically generates a nutritionally balanced meal plan."

[1087] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1088] Step 1: Record and send video

[1089] Subject: User

[1090] Users can launch a dedicated app on their smartphone or tablet and take video of the inside of their refrigerator or storage unit. Once the video is complete, the user can send the video to the server via the app.

[1091] Input: Video data taken inside a refrigerator or storage facility

[1092] Output: Video data sent to the server

[1093] Specific operation: The user opens the refrigerator door, takes sequential photos of the inside with their smartphone camera, and taps the "Send" button to upload the video data to the server.

[1094] Step 2: Receiving and analyzing the video

[1095] Subject: Server

[1096] The server receives the video sent by the user and uses a video analysis program to analyze the video frame by frame and identify the items (ingredients and seasonings) that appear in the video.

[1097] Input: Video data sent by the user

[1098] Output: List of recognized objects

[1099] How it works: The server runs a video analysis program, breaks down the video into frames, and automatically identifies and lists the ingredients and seasonings. For example, a list of items such as "chicken thighs, cabbage, and carrots" is obtained.

[1100] Step 3: Menu Creation

[1101] Subject: Server

[1102] The server generates a menu that takes nutritional balance into consideration based on the list of items obtained through the analysis. Using a generative AI model, multiple menu plans are created. Each menu includes a main dish and three side dishes (soup, main course, and garnish), and cooking time and calorie information are also included.

[1103] Input: A list of recognized items (e.g., "chicken thigh, cabbage, carrot")

[1104] Output: Multiple menu plans that take nutritional balance into consideration

[1105] Specific operation: The server runs the AI ​​model and generates a menu suggestion such as "rice, teriyaki chicken thighs, miso soup, and stir-fried cabbage and carrots."

[1106] Step 4: Display menu information

[1107] Subject: Terminal

[1108] The terminal receives the menu information sent from the server and displays it on the screen in an easy-to-read format. The user can then select an appropriate menu from multiple suggested menus.

[1109] Input: Menu information sent from the server

[1110] Output: Menu information displayed on the device

[1111] Specific operation: The device displays menu suggestions such as "Main dish: rice, Main course: teriyaki chicken thigh, Soup: miso soup, Side dish: stir-fried cabbage and carrots," and the user can choose from them.

[1112] Step 5: Identify and order missing ingredients

[1113] Subject: Server

[1114] The server automatically lists any missing items based on the selected menu, orders missing ingredients and seasonings from a designated supplier (such as an online supermarket), and schedules delivery to coincide with the user's return time.

[1115] Input: Selected menu plan, user's current ingredients list

[1116] Output: List of ingredients to order, order information

[1117] Specific operation: The server lists ingredients such as miso that are in short supply, and after receiving confirmation from the user, automatically places an order with the online supermarket. The delivery time is set to coincide with the user's return home.

[1118] Step 6: Arrange for prepared meals

[1119] Subject: Server

[1120] If the user cannot find time to cook, the server will arrange for the delivery of cooked food from affiliated restaurants or diner.

[1121] Input: User notification of cooking difficulty

[1122] Output: Arrange for delivery of cooked food

[1123] Specific operation: The user enters "I can't cook today" through the app, and the server arranges for the following meal from a partner restaurant: "Main dish: rice, Main course: teriyaki chicken thigh, Soup: miso soup, Side dish: stir-fried cabbage and carrots" and has it delivered.

[1124] (Application example 1)

[1125] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1126] In today's busy lifestyles, it is difficult for users to prepare healthy, nutritionally balanced meals. Managing the contents of the refrigerator and storage areas and planning menus based on that information is also a time-consuming task. Furthermore, if necessary ingredients are in short supply, it is inefficient to go shopping every time. To solve these problems, a system is needed that allows users to easily manage ingredients, create menus, and order missing ingredients.

[1127] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1128] In this invention, the server includes a means for receiving a video and recognizing items in the video, a means for generating a plurality of combinations based on the recognized items, and a means for identifying missing items. This automatically lists ingredients in the refrigerator or storage area from the video taken by the user, and generates a nutritionally balanced menu based on that list. Furthermore, the automatic identification of missing items allows online ordering from a virtual store, and delivery of ingredients or cooked meals can be arranged to coincide with the user's return home time. This allows users to prepare and obtain healthy meals without hassle.

[1129] The "means for receiving video and recognizing items in the video" is a technology for receiving video shot by a user and analyzing and identifying items that appear in the video.

[1130] The "means for generating multiple combinations based on recognized items" is a technology for creating multiple menus based on information about identified items, taking into consideration nutritional balance, cooking methods, etc.

[1131] The "means for identifying missing items" is a technology for comparing the recognized items with the items required for the generated menu and listing the missing items.

[1132] The "means for ordering the missing items from a predetermined supply source" is a technique for ordering the identified missing items from a related supply source (for example, an online supermarket or an affiliated store).

[1133] The "means for generating nutritionally balanced menus" is a technology for creating multiple balanced menus by taking into consideration the nutritional components of recognized items.

[1134] "Means for sending the generated menu information to the user terminal and displaying it so that the user can select" refers to a technology for sending the generated menu data to the user's device and displaying it so that the user can select the menu they desire.

[1135] "Means for placing online orders at a virtual store" refers to technology that allows users to easily place orders at a virtual store (e.g., an online shopping site) based on a list of items they are missing.

[1136] This system allows users to take videos of ingredients in their refrigerators or storage areas, analyzes the videos, and automatically creates nutritionally balanced meals. The system operates in cooperation with the server, terminals, and users to provide convenience and health to users.

[1137] First, the user uses a device such as a smartphone or tablet to record video of the inside of the refrigerator or storage unit. A dedicated application is installed on the device, and the video is sent to the server using this application. The video is sent via a communication line.

[1138] The server has a program that analyzes the received video. This analysis uses the OpenCV library for image analysis and TensorFlow for ingredient recognition. The server identifies and lists the items (ingredients and seasonings) in the video. Based on this list, the server uses a generative AI model to generate multiple menus that take nutritional balance into consideration. Specifically, it generates combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also included with each menu.

[1139] The server then sends the generated menu information to the terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to select. The user selects the appropriate menu from the multiple menus presented and checks its details. If there are any ingredients or seasonings required for the menu that are missing from the refrigerator or storage, the server identifies them and lists them.

[1140] With the user's permission, the server orders missing ingredients and seasonings from a designated supply source (such as a local online supermarket). When the order is placed, it schedules delivery to coincide with the user's return time. This ensures that the necessary ingredients have arrived in a timely manner when the user returns home. In addition, if the user is unable to find time to cook or is too tired from work to cook, the server will arrange for pre-cooked food to be delivered from an affiliated restaurant.

[1141] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via the app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Any "miso" that is missing is listed and ordered from the online supermarket in time for the user to return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[1142] An example of a prompt for a generative AI model is as follows:

[1143] "Analyze the video of the contents of the refrigerator and create a list of recognized ingredients. Furthermore, use that data to generate a nutritionally balanced meal plan, identifying and listing any missing ingredients. Display the generated meal plan in the following format: 'Main dish: Teriyaki chicken thigh, Side dish: Cabbage salad, Another side dish: Kinpira carrots'. If any ingredients are missing, provide a function that allows the user to order them from a virtual store."

[1144] The system allows users to prepare and consume healthy meals hassle-free.

[1145] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1146] Step 1:

[1147] Users use devices such as smartphones or tablets to record video of the inside of refrigerators or storage units. A dedicated application is installed on the device, and the video recorded using this application is sent to a server. The input is video data recording the inside of the refrigerator or storage unit, and the output is the video data sent to the server. The device sends the video data via a communication line.

[1148] Step 2:

[1149] To analyze the received video, the server uses the OpenCV library to break down the video data into frames and perform image analysis on each frame. Based on the recognition results, TensorFlow is used to identify ingredients. The input is the image data broken down into frames, and the output is a list of recognized ingredients. Specifically, the server resizes the images of each frame to 224 x 224 pixels and inputs them into the ingredient recognition model.

[1150] Step 3:

[1151] The server uses a generative AI model to generate multiple nutritionally balanced menus based on the list of recognized ingredients. The input is the list of recognized ingredients, and the output is a list of generated menus. Specifically, it takes into account the combination of staple food, side dish, and soup, and also adds cooking time and calorie information to each menu.

[1152] Step 4:

[1153] The server sends the generated menu information to the user terminal. The input is a list of the generated menus, and the output is the menu information sent to the user terminal. The terminal displays the received menu information in an easy-to-read format, allowing the user to select an appropriate menu from it.

[1154] Step 5:

[1155] The user selects the appropriate menu from the multiple menus presented and checks its details. The input is the menu information sent from the server, and the output is the details of the menu selected by the user. The selected menu includes the necessary ingredients and seasonings.

[1156] Step 6:

[1157] The server identifies and lists the items that are missing from the refrigerator or storage based on the selected menu. The input is the selected menu and the current status of the refrigerator or storage, and the output is a list of the missing items.

[1158] Step 7:

[1159] With the user's permission, the server orders missing ingredients and seasonings from a designated source (e.g., a local online supermarket). The input is a list of missing items, and the output is order confirmation information. When placing an order, the server schedules delivery to coincide with the user's return home time.

[1160] Step 8:

[1161] If the user cannot find the time to cook, the server will arrange for the delivery of cooked food from a partner restaurant. The inputs are the user's cooking time and the partner restaurant's menu, and the output is confirmation of the delivery of the cooked food.

[1162] This allows users to prepare and consume healthy meals without any hassle.

[1163] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1164] This invention is a system that allows users to take videos of ingredients in their refrigerators and storage areas, analyzes the videos, and automatically generates nutritionally balanced menus. By combining this invention with an emotion engine that recognizes the user's emotions, it is possible to adjust the menu suggestions according to the user's emotional state. This allows for meal suggestions that are tailored to the user's mental state and preferences.

[1165] First, the user uses a device such as a smartphone or tablet to record video of the inside of the refrigerator or storage unit. A dedicated app is installed on the device, and the video is sent to the server using this app. The video is sent via a communication line.

[1166] The server has a program that analyzes the received video. The server identifies and lists the items (ingredients and seasonings) in the video. Based on this list, the server generates multiple menus that take nutritional balance into consideration. Specifically, it generates combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish). Cooking time and calorie information are also added to each menu.

[1167] The server then analyzes the user's current emotional state using an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions based on, for example, the text the user types into the device, the tone of voice, or changes in facial expressions. This emotion information is used to adjust the menu suggestions. For example, if the user is feeling stressed, the server can prioritize easy-to-prepare menus.

[1168] The server optimizes the priority of the menu suggestions based on the generated menu information and emotion information and sends them to the device. The device then displays the received menu information in an easy-to-read format, allowing the user to select. The user selects an appropriate menu from the multiple menus presented and checks its details. If there are any ingredients or seasonings required for the menu that are missing from the refrigerator or storage, the server identifies them and lists them.

[1169] With the user's permission, the server orders missing ingredients and seasonings from a designated source (such as a local online supermarket). When the order is placed, it schedules delivery to coincide with the user's return home time. When the user returns home, the necessary ingredients have been delivered in a timely manner.

[1170] In addition, if the user does not have time to cook or is too tired from work to cook, the server will arrange for pre-cooked meals to be delivered from affiliated restaurants, allowing the user to provide healthy meals for themselves and their family.

[1171] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via the app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Furthermore, an emotion engine recognizes the user's stress level and prioritizes menu options that are easy to prepare. Any "miso" that is in short supply is listed and ordered from an online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[1172] As described above, the system of the present invention provides comprehensive support for users to efficiently prepare or obtain nutritionally balanced meals, enabling users to easily and effectively achieve healthy eating habits even in busy daily lives.

[1173] The processing flow will be explained below.

[1174] Step 1:

[1175] Users take videos of the ingredients in their refrigerators or storage areas using a smartphone or tablet, adjusting the camera angle and brightness to ensure all ingredients are clearly visible.

[1176] Step 2:

[1177] The device sends the captured video to the server via a dedicated app. Before sending, the device encodes the video and compresses it appropriately.

[1178] Step 3:

[1179] The server receives the video sent from the device and stores it in a database. The received video is then processed by an analysis program.

[1180] Step 4:

[1181] The server uses an AI-generated model to identify ingredients in the video, detecting them in each frame and listing them.

[1182] Step 5:

[1183] Based on the listed ingredient information, the server retrieves nutritional information and available recipes for each ingredient from the database.

[1184] Step 6:

[1185] The server runs a menu generation algorithm that takes nutritional balance into consideration based on the acquired ingredient information and recipe data, generating five combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, and garnish).

[1186] Step 7:

[1187] The server adds cooking time and calorie information to each menu, and also generates a detailed list of ingredients and seasonings required.The server also uses an emotion engine to analyze the user's emotional state.

[1188] Step 8:

[1189] The server adjusts its menu suggestions based on the user's emotional state: for example, if the user is feeling stressed, it will prioritize easy-to-prepare meals.

[1190] Step 9:

[1191] The server then transmits the optimized menu information, taking into account the user's emotional state, to the user's device using secure communication methods.

[1192] Step 10:

[1193] The device displays the menu information received from the server in a dedicated app. The user can check multiple menu options and view detailed information (cooking time, calories, and a list of necessary ingredients).

[1194] Step 11:

[1195] The user selects an appropriate menu from the presented options and checks the details. After making a selection, the system lists any missing ingredients and seasonings.

[1196] Step 12:

[1197] The user can authorize the app to order missing ingredients by tapping the order authorization button.

[1198] Step 13:

[1199] With the user's permission, the server automatically orders the missing ingredients from a designated supply source (such as a local online supermarket). When placing an order, the server sets a delivery schedule to match the user's return time.

[1200] Step 14:

[1201] The server obtains the estimated delivery time of the ordered ingredients from the supply source and notifies the user.

[1202] Step 15:

[1203] When the user returns home, the user checks that the necessary ingredients have been delivered and begins cooking based on the presented menu.

[1204] Step 16:

[1205] If the user cannot find time to cook, the user notifies the server that he / she would like to have cooked food delivered from a partner restaurant or diner.

[1206] Step 17:

[1207] The server receives the user's notification, obtains prepared food from affiliated restaurants and other establishments in the area, and arranges for delivery at the time specified by the user.

[1208] Through the above specific processing steps, the present invention supports users in efficiently preparing nutritionally balanced meals, and further utilizes an emotion engine to make optimal suggestions based on the user's emotional state.

[1209] Example 2

[1210] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1211] In today's busy lifestyles, people often lack the time and effort to prepare balanced meals. Furthermore, there are no systems that suggest meals that take into account stress and fatigue, making it difficult for users to easily find menus that suit their individual circumstances. Furthermore, if necessary ingredients or seasonings are missing, there is no efficient way to supplement them, which is a time-consuming process.

[1212] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for receiving a video and recognizing items in the video, means for generating a food list based on the recognized items, means for generating a menu taking nutritional balance into consideration, means for analyzing the user's emotions and adjusting the menu based on the emotion information, means for identifying missing items, and means for ordering the missing items from a predetermined supply source. This allows users to efficiently prepare nutritionally balanced meals and easily find appropriate menus based on their emotional state, even in the midst of a busy lifestyle. Furthermore, the ordering process for missing ingredients and seasonings can be automated, significantly reducing the effort required for meal preparation.

[1213] The "means for receiving video" is a function for transmitting video data taken by the user to a server via a dedicated application and for the server to receive the video data.

[1214] "Means for recognizing items" is a function that uses video analysis technology to identify and list items such as ingredients and seasonings that appear in the received video data.

[1215] The "means for generating a food list" is a function for creating a list of ingredients currently stored in the refrigerator or storage area based on information about items recognized through video analysis.

[1216] The "means for generating menus" is a function for automatically generating multiple menu plans based on the recognized food list, taking into consideration nutritional balance.

[1217] The "means for analyzing emotions" is a function for analyzing emotional information in text, audio, or video input by the user and identifying the user's current emotional state.

[1218] The "means for adjusting the menu" is a function for adapting and optimizing the temporarily generated menu plan to the user's emotional state based on the analyzed emotional information.

[1219] The "means for identifying missing items" is a function for identifying the ingredients and seasonings required for the generated menu that are currently missing from the refrigerator or storage.

[1220] The "means for ordering from a predetermined supply source" is a function for automatically ordering missing items from a supply source such as an online supermarket after identifying the items that are in short supply.

[1221] This system allows users to take videos of ingredients in their refrigerators and storage areas, analyzes the videos, and automatically generates nutritionally balanced menus. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, this system can adjust the menu suggestions according to the user's emotional state.

[1222] System configuration

[1223] This system starts when the user uses a device such as a smartphone or tablet to record video of the inside of a refrigerator or storage unit. A dedicated app is installed on the device, and the video recorded using this app is sent to a server. The video is sent via an internet communication line.

[1224] Video Analysis

[1225] The server has a program for analyzing the received video. In this case, OpenCV or TensorFlow can be used as video analysis software. The server identifies items in the video, specifically ingredients and seasonings. This identification can be performed using image recognition technology or machine learning models.

[1226] Ingredient list generation and menu suggestions

[1227] The server generates a food list based on the identified ingredients. This list is stored in a database, and nutritional calculation software is used to automatically generate multiple balanced meal plans. These plans include a combination of a main dish and three side dishes (soup, main course, and side dish). Cooking time and calorie information are also included for each plan.

[1228] Emotion analysis

[1229] The server then uses an emotion engine to analyze the user's emotional state. This analysis can be performed using, for example, IBM Watson Tone Analyzer or Microsoft Azure Emotion API. The emotion engine analyzes the text and voice data entered by the user into the device, or changes in facial expressions captured using the camera. The optimal menu is then selected from the suggested menus based on the user's emotional state. For example, if the user is feeling stressed, it will suggest dishes that are easy to prepare and require little time to prepare.

[1230] Sending and displaying menu information

[1231] The server sends the optimized menu information to the terminal. The terminal displays the received information in an easy-to-read format, allowing the user to easily select their preferred menu. The user selects the desired menu from the suggested menu and checks its details. If there are any items missing from the refrigerator or storage at this point, the server automatically lists them and places an order with a designated supply source (for example, a local online supermarket).

[1232] Automated ordering and delivery

[1233] When placing an order, the delivery is scheduled to coincide with the user's return time, and when the user returns home, the necessary ingredients are delivered in a timely manner.

[1234] Prepared food delivery

[1235] In addition, if the user does not have time to cook or is too tired from work to cook, the server will arrange for cooked food to be delivered from affiliated restaurants, allowing the user to provide nutritionally balanced meals for themselves and their family.

[1236] Specific examples

[1237] To give a specific example, a user can take a video of the contents of their refrigerator with their smartphone and send it to a server via a dedicated app. The server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as the side dish. Furthermore, an emotion engine recognizes the user's stress level and prioritizes menu options that are easy to prepare. Any "miso" that is in short supply is listed and ordered from an online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[1238] Prompt Sentence Examples

[1239] "Please suggest a nutritionally balanced meal using the ingredients in your refrigerator. Currently, the refrigerator contains chicken thighs, cabbage, and carrots. The user is currently feeling stressed, so would like a recipe that is easy to prepare."

[1240] As described above, the system of the present invention allows users to efficiently prepare nutritionally balanced meals and easily find appropriate menus according to their emotional state. Furthermore, by automating the ordering process for ingredients and seasonings that are in short supply, it is possible to significantly reduce the effort required for meal preparation in users' daily lives.

[1241] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1242] Step 1:

[1243] The user takes a video of the refrigerator using the device.

[1244] Input: A user uses a smartphone or tablet to record a video of the inside of a refrigerator.

[1245] Output: Video data of the inside of the refrigerator.

[1246] Specific operation: The user launches the dedicated app and shoots a video that captures all the food in the refrigerator.

[1247] Step 2:

[1248] The video data is sent to the server.

[1249] Input: Video data taken inside a refrigerator.

[1250] Output: Video data sent to the server.

[1251] Specific operation: After the user has finished shooting, they use a dedicated app to send the video they have shot to a server via internet communication.

[1252] Step 3:

[1253] The server analyzes the video and generates a list of ingredients.

[1254] Input: Video data sent to the server.

[1255] Output: A list of recognized ingredients.

[1256] How it works: The server uses video analysis software to apply image recognition technology to identify ingredients in the video. Specifically, it uses OpenCV and TensorFlow to identify ingredients and generate a food list.

[1257] Step 4:

[1258] The server automatically generates the menu.

[1259] Input: A list of recognized ingredients.

[1260] Output: A list of generated menus.

[1261] What it does: The server uses nutrition calculation software to generate multiple nutritionally balanced meal plans based on the recognized ingredients, using the nutrition information and recipes stored in the database for this process.

[1262] Step 5:

[1263] Recognize user emotions with an emotion engine.

[1264] Input: User text, voice, or facial expression data.

[1265] Output: Parsed emotional state.

[1266] How it works: The server uses an emotion recognition engine to analyze the text, voice data, and facial expressions entered by the user on the device. Examples include IBM Watson Tone Analyzer and Microsoft Azure Emotion API.

[1267] Step 6:

[1268] Optimize the menu suggested by the server.

[1269] Input: Generated menu list and parsed emotional state.

[1270] Output: A list of optimized menus that match the user's emotions.

[1271] Specific operation: The server proposes menus that suit the user's emotions based on the generated menu information and emotional information. For example, if the user is feeling stressed, it will prioritize suggestions for easy-to-prepare dishes.

[1272] Step 7:

[1273] Optimized menu information is sent to the device.

[1274] Input: A list of optimized menu items.

[1275] Output: Menu information sent to the device.

[1276] Specific operation: The server sends the optimized menu information to the terminal in real time. This sending process is performed in data formats such as JSON and XML.

[1277] Step 8:

[1278] The user checks and selects the menu.

[1279] Input: Multiple menu items displayed on the terminal.

[1280] Output: The selected menu.

[1281] Specific operation: The user checks the menu information received on the device and intuitively swipes and taps to select the desired menu.

[1282] Step 9:

[1283] The server identifies the ingredients that are in short supply and processes the order.

[1284] Input: Selected menu and list of ingredients in the refrigerator.

[1285] Output: A list of ingredients that are in short supply and have been ordered from the online supermarket.

[1286] Specific operation: Based on the selected menu, the server identifies the ingredients that are missing and automatically processes the order via the online supermarket API.

[1287] Step 10:

[1288] Arrange for delivery of prepared meals as needed.

[1289] Input: Information about the user's cooking time and status.

[1290] Output: Cooked food ordered from partner restaurant.

[1291] Specific operation: If the user does not have time to cook or is too tired to cook, the server will use the online ordering system of a partner restaurant to send the user's address and order details and arrange for delivery.

[1292] By following these steps, the user can prepare a nutritionally balanced meal without much effort.

[1293] (Application example 2)

[1294] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1295] While conventional meal suggestion systems could automatically generate menus that took into account the ingredients and nutritional balance of the user's inventory, they were unable to suggest menus that reflected the user's emotional state. Furthermore, there was no centralized system for purchasing ingredients when the user was short of them or when they could not find the time to cook, which often caused stress for users. Another issue was that the suggested menu options were visually difficult to understand.

[1296] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1297] In this invention, the server includes means for receiving a video and recognizing items in the video, means for generating a plurality of combinations based on the recognized items, means for recognizing the user's emotional state, means for adjusting the priority of the combinations to be proposed based on the emotional state, means for identifying missing items, means for ordering the missing items from a predetermined supply source, and means for displaying the generated menu information in an easy-to-view format to allow the user to select. This allows the server to propose optimal menus based on the user's emotional state and available ingredients, and also allows for centralized management of purchasing missing ingredients and delivery of cooked meals, enabling the user to easily prepare or obtain nutritionally balanced meals without stress.

[1298] "Means for receiving video and recognizing items in the video" refers to a technology that acquires video data sent from the user's device and analyzes and identifies the items (ingredients and seasonings) that appear in the video.

[1299] The "means for generating multiple combinations based on recognized items" is a technology that automatically generates menu combinations that take into account diverse nutritional balances based on analyzed item information.

[1300] "Means for recognizing the user's emotional state" refers to technology that analyzes emotions from the user's input, tone of voice, changes in facial expression, etc., and identifies that state.

[1301] The "means for adjusting the priority of proposed combinations based on the emotional state" is a technology that dynamically changes the display priority of the most suitable combination from among the generated menus, taking into account the user's emotional state.

[1302] The "means for identifying missing items" is a technology that automatically detects items that are necessary for the generated menu but are not present in the user's inventory of ingredients.

[1303] The "means for ordering the missing items from a predetermined supply source" is a technology for automatically ordering the detected missing ingredients from a supply source such as an online store or supermarket.

[1304] "Means for displaying generated menu information in an easy-to-see format and allowing users to make selections" refers to a technology that displays a list of suggested menus in a visually easy-to-understand format on the user's terminal, providing an interface that allows users to easily make selections.

[1305] The present invention is a system that allows users to take videos of ingredients in their refrigerators and storage areas, analyzes the videos, and automatically generates nutritionally balanced menus. Furthermore, by adjusting the menu suggestions according to the user's emotional state, it is possible to provide meal suggestions that take into account the user's mental state and preferences.

[1306] First, a user uses a device such as a smartphone or tablet to record video of the inside of a refrigerator or storage unit. A dedicated application is installed on the device, and the video is sent to a cloud server using this application. The video is sent using a communication line such as the Internet.

[1307] Next, a video analysis program runs on the cloud server. Specifically, it uses video analysis engines such as OpenCV and TensorFlow to identify and list items (ingredients and seasonings) in the video. Based on the results of this analysis, multiple menus are generated that take nutritional balance into consideration. Combinations of a staple food (rice, bread, noodles) and three side dishes (soup, main dish, garnish) are generated, and cooking time and calorie information is also added to each menu.

[1308] After ingredient recognition is complete, an emotion engine is activated to recognize the user's emotional state. This emotion engine uses, for example, the Google Cloud Natural Language API or the Microsoft Azure Emotion API to analyze emotions based on the text the user enters into their device, their tone of voice, or changes in facial expressions. Based on this emotional information, the suggested meal plans are adjusted. For example, if the user is feeling stressed, the system can prioritize easy-to-prepare meals.

[1309] Next, the cloud server optimizes the priority of the menu suggestions based on the generated menu information and emotion information, and sends the results to the user's device. At this time, the generated menu information is displayed visually in an easy-to-understand user interface, allowing the user to easily select a menu. For example, a list of menus may be displayed in card format, with each card showing cooking time, calories, and icons of the ingredients used.

[1310] Furthermore, if any of the ingredients or seasonings required for the user's selected meal are missing from the refrigerator or storage, the cloud server will identify them and automatically list them. With the user's permission, the cloud server will order the missing items from a designated supplier (such as a local online supermarket). When the order is placed, it will be scheduled to be delivered by the user's return home time.

[1311] In addition, if the user cannot find time to cook or is too tired from work to cook, the cloud server can also arrange for cooked food to be delivered from affiliated food providers (such as set meal restaurants).

[1312] As a specific example, a user takes a video of the contents of their refrigerator with their smartphone and sends it to a cloud server via an application. The cloud server analyzes the video and generates a list of ingredients, such as "chicken thighs, cabbage, and carrots." Based on this, multiple menu options are presented, such as "rice" as the staple food, "teriyaki chicken thighs" as the main side dish, "miso soup" as the soup, and "stir-fried cabbage and carrots" as a side dish. Furthermore, an emotion engine recognizes the user's stress level and prioritizes menu options that are easy to prepare. Any "miso" that is in short supply is listed and ordered from an online supermarket in time for the user's return home. If there is no time to cook, the above menu items are delivered from a partner restaurant.

[1313] Example prompt sentence:

[1314] "What should I make for dinner today? I'm busy and stressed at work, so I'd like something easy to make."

[1315] In this way, the system of the present invention proposes optimal menus based on the user's emotional state and the ingredients they have on hand, and can also centrally manage the purchase of missing ingredients and the delivery of cooked meals, allowing users to easily prepare or obtain nutritionally balanced meals without feeling stressed.

[1316] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1317] Step 1:

[1318] The user uses a device to record video of the food in the refrigerator or storage area. Once the video is complete, the user uses a dedicated application to send the video data to a cloud server. The input is high-resolution video data recorded by the device, and the output is this video data sent to the cloud server.

[1319] Step 2:

[1320] The cloud server analyzes the received video data and identifies the ingredients in the video. This process is performed using OpenCV and TensorFlow. Specifically, the video is broken down into frames, and the items (ingredients and seasonings) in each frame are recognized based on a model. The input is the video data sent by the user, and the output is a list of identified ingredients.

[1321] Step 3:

[1322] The cloud server generates multiple nutritionally balanced menus based on the identified ingredient list, including information such as the main dish, several side dishes, total calories, and cooking time. The input is the identified ingredient list, and the output is the generated menu list.

[1323] Step 4:

[1324] Next, the server recognizes the user's emotional state. Emotion recognition is achieved using the user's text input, voice input, or facial expression recognition. Technologies used include Google Cloud Natural Language API and Microsoft Azure Emotion API. The input includes text, audio, and video data that indicate the user's emotions, and the output is the analyzed emotional state.

[1325] Step 5:

[1326] The cloud server adjusts the priority of the generated menu items based on the recognized emotional state. For example, for a user who is under stress, adjustments are made such as giving priority to displaying easy-to-prepare menu items. The inputs are the emotional state and the generated menu list, and the output is the menu list with adjusted priority.

[1327] Step 6:

[1328] The cloud server sends the menu list with adjusted priorities to the user's device. A dedicated application displays the menu list in a visually easy-to-understand format. For example, cooking time and calorie information may be presented to the user in card format. The input is the menu list with adjusted priorities, and the output is the device screen that displays it.

[1329] Step 7:

[1330] Based on the menu selected by the user, the cloud server generates a list of ingredients that are missing. This list is generated by comparing it with the ingredients the user owns. The input is the list of ingredients the user owns and the selected menu, and the output is the list of ingredients that are missing.

[1331] Step 8:

[1332] The cloud server automatically places orders with a specified supply source (e.g., an online supermarket) based on the list of ingredients that are in short supply. This order is scheduled to coincide with the user's return time. The inputs are the list of ingredients that are in short supply and the user's return time, and the output is the order data sent to the supply source.

[1333] Step 9:

[1334] If the user does not have time to cook or is tired from work, the cloud server arranges for cooked food from affiliated food service providers. The input is the user's emotional state and cooking availability, and the output is the order and delivery request data for the cooked food.

[1335] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1336] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1337] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1338] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1339] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1340] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1341] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1342] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1343] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1344] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1345] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1346] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1347] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1348] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1349] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1350] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1351] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1352] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1353] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1354] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1355] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1356] The following is further disclosed regarding the above embodiment.

[1357] (Claim 1)

[1358] means for receiving the video and recognizing the object in the video;

[1359] means for generating a plurality of combinations based on the recognized items;

[1360] a means of identifying the missing items;

[1361] means for ordering the missing items from a predetermined source;

[1362] A system including:

[1363] (Claim 2)

[1364] 2. The system according to claim 1, further comprising means for obtaining an estimated delivery time of the item from a predetermined supply source and adjusting it to coincide with the user's return home time.

[1365] (Claim 3)

[1366] 2. The system according to claim 1, further comprising means for obtaining cooked food from a predetermined provider and delivering it to the user when the user is unable to secure time to cook the food.

[1367] "Example 1"

[1368] (Claim 1)

[1369] means for receiving the video and recognizing the object in the video;

[1370] means for generating a plurality of combinations based on the recognized items;

[1371] a means of identifying the missing items;

[1372] means for ordering the missing items from a predetermined source;

[1373] A means for analyzing the captured video frame by frame and automatically identifying the item;

[1374] A means for automatically generating a menu that takes nutritional balance into consideration based on the analyzed item list;

[1375] A means for displaying the generated menu information and allowing the user to select it;

[1376] A system including:

[1377] (Claim 2)

[1378] 2. The system according to claim 1, further comprising means for obtaining an estimated delivery time of the item from a predetermined supply source and adjusting it to coincide with the user's return home time.

[1379] (Claim 3)

[1380] 2. The system according to claim 1, further comprising means for obtaining cooked food from a predetermined provider and delivering it to the user when the user is unable to secure time to cook the food.

[1381] "Application Example 1"

[1382] (Claim 1)

[1383] means for receiving the video and recognizing the object in the video;

[1384] means for generating a plurality of combinations based on the recognized items;

[1385] a means of identifying the missing items;

[1386] means for ordering the missing items from a predetermined source;

[1387] means for generating a nutritionally balanced menu based on the recognized items;

[1388] means for transmitting the generated menu information to a user terminal and displaying it so that the user can select it;

[1389] A system including:

[1390] (Claim 2)

[1391] 2. The system according to claim 1, further comprising means for obtaining an estimated delivery time of the item from a predetermined supply source and adjusting it to coincide with the user's return home time.

[1392] (Claim 3)

[1393] 2. The system according to claim 1, further comprising means for obtaining cooked food from a predetermined provider and delivering it to the user when the user is unable to secure time to cook the food.

[1394] (Claim 4)

[1395] 2. The system according to claim 1, further comprising means for identifying missing items from the recognized items and for allowing an online order to be placed at a virtual store based on the identified items.

[1396] "Example 2: Combining Emotion Engines"

[1397] (Claim 1)

[1398] means for receiving the video and recognizing the object in the video;

[1399] means for generating a food list based on the recognized items;

[1400] A means for generating menus taking into consideration nutritional balance;

[1401] means for analyzing a user's emotions and adjusting a menu based on the emotion information;

[1402] a means of identifying the missing items;

[1403] means for ordering the missing items from a predetermined source;

[1404] A system including:

[1405] (Claim 2)

[1406] 2. The system according to claim 1, further comprising means for obtaining an estimated delivery time of the item from a predetermined supply source and adjusting it to coincide with the user's return home time.

[1407] (Claim 3)

[1408] 2. The system according to claim 1, further comprising means for obtaining cooked food from a predetermined provider and delivering it to the user when the user is unable to secure time to cook the food.

[1409] "Application example 2 when combining emotion engines"

[1410] (Claim 1)

[1411] means for receiving the video and recognizing the object in the video;

[1412] means for generating a plurality of combinations based on the recognized items;

[1413] means for recognizing the emotional state of a user;

[1414] a means for adjusting the priority of the combinations to be suggested based on the emotional state;

[1415] a means of identifying the missing items;

[1416] means for ordering the missing items from a predetermined source;

[1417] A system including:

[1418] (Claim 2)

[1419] 2. The system according to claim 1, further comprising means for obtaining an estimated delivery time of the item from a predetermined supply source and adjusting it to coincide with the user's return home time.

[1420] (Claim 3)

[1421] 2. The system according to claim 1, further comprising means for obtaining cooked food from a predetermined provider and delivering it to the user when the user is unable to secure time to cook the food.

[1422] (Claim 4)

[1423] 2. The system according to claim 1, further comprising means for displaying the generated menu information in an easy-to-view manner and allowing the user to select the menu information. [Explanation of symbols]

[1424] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving the video and recognizing the object in the video; means for generating a plurality of combinations based on the recognized items; a means of identifying the missing items; means for ordering the missing items from a predetermined source; A system including:

2. 2. The system according to claim 1, further comprising means for obtaining an estimated delivery time of the item from a predetermined supply source and adjusting the estimated delivery time to coincide with the user's return home time.

3. The system according to claim 1, further comprising means for obtaining cooked food from a predetermined supplier and delivering it to the user when the user is unable to secure cooking time.

Images