System

Abstract

A system is provided.SOLUTION: The system includes means for acquiring image data in a refrigerator from a user terminal, means for transmitting the acquired image data to a cloud server, means for analyzing the image data in the cloud server and recognizing food items, means for proposing a menu based on the recognized food items, and means for transmitting the proposed menu to the user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Creating daily menus takes time and requires the effort of checking whether food is in the refrigerator. Furthermore, people may not realize they are running low on staple ingredients, leading to unplanned shopping trips. This leads to inefficient food management and menu planning within the home, and increases stress. [Means for solving the problem]

[0005] The present invention provides a system that acquires image data of the contents of a refrigerator from a user terminal, analyzes the image data on a cloud server, and recognizes food items. Based on the recognized food items, it automatically suggests a menu and allows the user to confirm it. Furthermore, it maintains a list of ingredients regularly provided by the user and compares it with the food items recognized from the image data to detect any missing ingredients. It generates a shopping list based on the detected missing ingredients and sends it to the user terminal. The generated shopping list can be shared using an external messaging service. This invention significantly improves the efficiency of food ingredient management and menu planning within the home, reducing the burden on the user.

[0006] A "user terminal" is an electronic device used by a user, such as a smartphone, tablet, or computer.

[0007] A "cloud server" is a server that can be accessed via the Internet and refers to a computer system that performs processes such as storing, analyzing, and transmitting data.

[0008] "Image data" refers to image data taken by a user terminal, and includes, for example, photos of ingredients in a refrigerator.

[0009] "Food items" refer to ingredients and types of food recognized from image data analyzed by the cloud server.

[0010] "Menu" refers to suggested recipes and meal combinations based on food items.

[0011] The "list of regular ingredients" refers to a list of ingredients and seasonings that are used on a daily basis and that are designated in advance by the user.

[0012] "Missing ingredients" refers to ingredients that are included in the regular ingredients list but are not included in the image data.

[0013] "Shopping list" refers to a list of ingredients that need to be purchased, including any ingredients that are missing.

[0014] "External messaging services" refer to services such as LINE that allow messages to be sent and received via the Internet. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system for streamlining food management and menu suggestions. It automatically recognizes ingredients in the refrigerator and suggests menus based on them, so that users don't have to waste time creating daily menus and managing shopping lists. It also provides a function to detect shortages of regular ingredients and create and share shopping lists.

[0037] System Overview

[0038] 1. Photographing and analyzing the inside of the refrigerator

[0039] A user uses a smartphone app to take a photo of the inside of the refrigerator.

[0040] The device sends the photographed photo data to a cloud server.

[0041] The server inputs the photo data into image analysis software to recognize the food items in the refrigerator. Specifically, the server uses image analysis algorithms (e.g., deep learning techniques) to identify each ingredient in the photo and assign it a label.

[0042] The server generates a list of recognized food items and sends it to the user's app, where the user can view the list of food items in the refrigerator.

[0043] 2. Menu suggestions

[0044] Based on the analysis results, the server searches a database for menu recipes using the food items in the refrigerator. The recipe database stores various combinations of food items and corresponding cooking recipes.

[0045] The server sends the list of found recipes to the user's app.

[0046] Users can select their favorite recipe from the suggested recipes, and the details of the selected recipe (ingredients, cooking method, required time, etc.) will be displayed.

[0047] 3. Check your stock of ingredients and create a shopping list

[0048] The cloud server stores a list of ingredients that the user has pre-set (for example, "onions, carrots, soy sauce").

[0049] The server compares the current food item list with the stock ingredient list to identify any missing food items.

[0050] The server generates a shopping list based on the missing food items, which can include any missing staple ingredients as well as any additional ingredients needed for the user's selected meal.

[0051] Users can check their shopping list in the app and share it using external messaging services (e.g., LINE), allowing them to ask other family members to do the shopping.

[0052] Specific examples

[0053] 1. Photographing and analyzing the inside of the refrigerator

[0054] The user launches the smartphone app and presses the "take a photo of the inside of the refrigerator" button.

[0055] The device uses its camera function to take pictures of the inside of the refrigerator and sends the image data to a cloud server.

[0056] The server analyzes the image and recognizes that there are tomatoes, eggs, cheese, and milk in the refrigerator.

[0057] The server generates a list of food items and sends it to the user's app.

[0058] 2. Menu suggestions

[0059] The server searches the database based on the list of ingredients and finds recipes such as "tomato omelet," "cheese omelet," and "tomato salad."

[0060] The server sends these recipes to the user's app.

[0061] When a user selects "Tomato Omelette" from the app, the recipe details are displayed.

[0062] 3. Check your stock of ingredients and create a shopping list

[0063] The server maintains a list of ingredients (e.g., onions, carrots, soy sauce).

[0064] The server compares the latest list of food items in the refrigerator with the list of regularly-available ingredients and detects that there are shortages of "onions" and "carrots."

[0065] The server generates a shopping list for "onions, carrots" and sends it to the user's app.

[0066] The user checks the shopping list on the app and sends it to her husband via LINE. On the way home, the husband buys onions and carrots at the supermarket.

[0067] As described above, the present invention reduces the burden on the user and provides a method for efficiently managing ingredients and creating menus at home.

[0068] The processing flow will be explained below.

[0069] Step 1:

[0070] The user launches the smartphone app and takes a photo of the inside of the refrigerator. The user then uses the camera function to capture an overall image of the inside of the refrigerator, and presses the "Send" button when the photo is complete.

[0071] Step 2:

[0072] The device sends the captured photo data to the cloud server. Specifically, the device sends an HTTP request to upload the photo data to the server.

[0073] Step 3:

[0074] The server receives the HTTP request and stores the transmitted photo data, ensuring that the server has the necessary photo data for future processing.

[0075] Step 4:

[0076] The server inputs the stored photo data into image analysis software to recognize the food items in the refrigerator. Specifically, it uses image recognition algorithms such as deep learning to identify each ingredient in the photo and assign it an individual label.

[0077] Step 5:

[0078] The server generates a list of food items based on the analysis results and sends the list to the user's app, where the user can view the list of food items in the refrigerator.

[0079] Step 6:

[0080] The server searches the menu recipe database based on the food item list. The recipe database stores various combinations of food items and corresponding cooking recipes. The server identifies recipes for dishes using the ingredients in the refrigerator.

[0081] Step 7:

[0082] The server sends a list of found recipes to the user's app, where the user can select the recipes they like.

[0083] Step 8:

[0084] The user uses the app to review the suggested recipes and selects a recipe, such as "Tomato Omelette." Details of the selected recipe (e.g., ingredients, cooking instructions, cooking time, etc.) are displayed in the app.

[0085] Step 9:

[0086] The server retrieves the user's predefined list of ingredients (e.g., "onions, carrots, soy sauce") from the database. The server compares this list with the latest list of food in the refrigerator.

[0087] Step 10:

[0088] The server compares the list of ingredients in stock with the list of food in the refrigerator and identifies any ingredients that are missing. For example, it determines that "onions" and "carrots" are missing.

[0089] Step 11:

[0090] The server generates a shopping list based on the missing ingredients, which can include the missing regular ingredients as well as any additional ingredients needed for the recipe selected by the user.

[0091] Step 12:

[0092] The server sends the generated shopping list to the user's app, where the user can view the shopping list.

[0093] Step 13:

[0094] The user presses the "Share Shopping List" button in the app and selects an external messaging service such as LINE. The device then sends the shopping list as a LINE message to share with other family members.

[0095] The above are the specific processing steps of the system, which allow users to proactively manage food and create menus.

[0096] Example 1

[0097] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0098] Conventional food management and menu suggestion systems have the problem that users have to manually check the ingredients in the refrigerator and search for recipes, which is time-consuming. Furthermore, to determine if a necessary ingredient is missing, the user must check the refrigerator again, making efficient food management difficult. This invention aims to reduce the user's burden and provide efficient food management and menu suggestions by automatically recognizing ingredients in the refrigerator using image analysis technology and suggesting menus based on that information.

[0099] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0100] In this invention, the server includes means for acquiring image data from a user terminal, means for transmitting the acquired image data via a network, means for analyzing the image data and recognizing object items in the server, means for providing information based on the recognized object items, and means for transmitting the provided information to the user terminal. This reduces the user's workload and enables efficient food management and menu suggestions. The server also includes means for storing a reference list specified by the user, comparing the object items recognized from the transmitted image data with the reference list to detect missing items, generating a list including the missing items, and means for transmitting the generated list to the user terminal, allowing the user to easily identify missing ingredients and use it as a shopping list. Furthermore, the server also includes means for transmitting the generated list from the user terminal using an external communication service, allowing the user to share the shopping list with family and other members and shop collaboratively.

[0101] "User terminal" refers to a device operated by a user, including a smartphone, tablet, or PC.

[0102] "Image data" refers to photographs or scanned images of the inside of the refrigerator, including visual information stored in digital form.

[0103] "Network" refers to a communications system that enables data to be communicated between devices, including the Internet and local area networks.

[0104] A "server" refers to a central computer system that processes data and provides information in response to requests from clients (user terminals).

[0105] "Object item" refers to an individual entity (e.g., food or object) recognized from image data.

[0106] "Information" is content provided based on the object item, including recipes, ingredient lists, etc.

[0107] The "standard list" refers to a list of items that should be kept as standard, set by the user, and includes, for example, a list of regular materials.

[0108] "External communication services" are services used to send and receive data, including messaging applications and email services.

[0109] A "list" is a list of specific items, including a list of missing items and a shopping list.

[0110] This invention is a system for streamlining food management and menu suggestions. It combines a user terminal, a server, an image analysis algorithm, and an external communication service to significantly reduce the user's workload.

[0111] System Configuration

[0112] The system mainly consists of the following hardware and software elements:

[0113] User device: Use a smartphone, tablet, or computer.

[0114] Server: A cloud server is used to process data from multiple user devices.

[0115] Image analysis algorithm: Uses deep learning models to perform object recognition.

[0116] External communication services: Using messaging applications (e.g., messaging apps, email).

[0117] Operating principle

[0118] The system's operation can be broadly divided into the following steps:

[0119] Taking photos of the inside of the refrigerator and analyzing them

[0120] A user uses a smartphone app to take a photo of the inside of their refrigerator. The device then sends the captured photo data to a cloud server. The server then uses an image analysis algorithm (deep learning technology) to recognize food items in the photo and assigns labels to each. Specifically, it uses an object recognition algorithm to extract food items such as "tomato," "egg," and "cheese." The server then generates a list of the recognized food items and sends it to the user's app.

[0121] Menu suggestions

[0122] Based on the analysis results, the server searches a database for menu recipes using the food items in the refrigerator. The recipe database stores various combinations of food items and corresponding cooking recipes. For example, a database search can find recipes such as "tomato omelette," "cheese omelette," and "tomato salad." The server sends a list of these recipes to the user's app, and the user can select their preferred recipe from the suggested recipes. Details of the selected recipe (ingredients, cooking method, required time, etc.) are also displayed in the app.

[0123] Check your stock of ingredients and create a shopping list

[0124] The server maintains a standard list (e.g., onions, carrots, soy sauce, etc.) preset by the user. The server compares the latest list of food items in the refrigerator with the standard list to identify any missing food items. Specifically, if the server detects that "onions" and "carrots" are missing, it generates a shopping list that includes these items. The shopping list can include not only the missing basic ingredients but also additional ingredients required for the meal selected by the user. The user can check the shopping list in the app and share it with other family members using an external communication service (e.g., a messaging app).

[0125] Specific examples

[0126] 1. Photographing and analyzing the inside of the refrigerator

[0127] The user launches the app and presses the "Take a photo of the inside of the refrigerator" button.

[0128] The device uses its camera function to take pictures of the inside of the refrigerator and sends the image data to the server.

[0129] The server analyzes the image and recognizes that there are tomatoes, eggs, cheese, and milk in the refrigerator.

[0130] The server generates a list of food items and sends it to the user's app.

[0131] 2. Menu suggestions

[0132] The server searches the database based on the list of ingredients and finds recipes such as "tomato omelet," "cheese omelet," and "tomato salad."

[0133] The server sends these recipes to the user's app.

[0134] When a user selects "Tomato Omelette" from the app, the recipe details are displayed.

[0135] 3. Check your stock of ingredients and create a shopping list

[0136] The server has a pre-stored list of criteria (e.g., onions, carrots, soy sauce).

[0137] The server compares the current food list with the reference list and identifies that "onions" and "carrots" are in short supply.

[0138] The server generates a shopping list that includes "onions, carrots" and sends it to the user's app.

[0139] A user sends a shopping list to their family via a messaging app, and the family members then go to the supermarket to purchase the items.

[0140] Prompt Sentence Examples

[0141] "Describe a system that recognizes the foods in a refrigerator, generates a list, and notifies the user of that list."

[0142] "Please explain the process of a system that suggests recipes based on a list of foods in the refrigerator."

[0143] "Describe a system that detects shortages of staple ingredients and automatically generates a shopping list."

[0144] In this way, it is possible to significantly improve the efficiency of the user's food ingredient management and menu proposals. This system reduces the burden on the user and provides a method for efficiently managing food at home and creating menus.

[0145] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0146] Step 1:

[0147] The user launches the smartphone app and takes a photo of the inside of the refrigerator.

[0148] Specific operation: The user presses the "Take a photo inside the refrigerator" button and takes a photo of the inside of the refrigerator using the smartphone camera.

[0149] Input: Visual information inside the refrigerator

[0150] Output: Image data

[0151] Step 2:

[0152] The device sends the photographed photo data to a cloud server.

[0153] Specific behavior: Uploads an image to the server using the HTTPS protocol.

[0154] Input: Image data

[0155] Output: Image data stored on a cloud server

[0156] Step 3:

[0157] The server uses image analysis algorithms (e.g., deep learning techniques) to recognize food items in the refrigerator.

[0158] Input: Image data stored on a cloud server

[0159] Processing: Object recognition with deep learning models

[0160] Output: A list of recognized food items

[0161] What it does: The server runs an image analysis model to identify food items such as "tomato," "egg," and "cheese."

[0162] Step 4:

[0163] The server sends the recognized food item list to the user terminal.

[0164] Specific operation: The server transfers the food item list generated to the user's device using the REST API.

[0165] Input: A list of recognized food items

[0166] Output: A list of food items in the refrigerator displayed on the user's device

[0167] Step 5:

[0168] The server searches a database based on the list of food items to find available recipes.

[0169] What it does: Uses a database query to extract the corresponding recipes.

[0170] Input: Food item list

[0171] Processing: Database queries and recipe extraction

[0172] Output: List of available recipes

[0173] Step 6:

[0174] The server sends the recipe list it finds to the user's terminal.

[0175] Specific operation: Send the recipe list to the user's terminal using the REST API.

[0176] Input: Available recipe list

[0177] Output: Recipe list displayed on the user's device

[0178] Step 7:

[0179] The user selects one of the suggested recipes and displays the details.

[0180] Specific behavior: Displays details (ingredients, cooking instructions, cooking time, etc.) of a recipe selected by the user from the app.

[0181] Input: Recipe ID selected by the user

[0182] Output: Detailed information about the selected recipe

[0183] Step 8:

[0184] The server maintains a list of criteria set by the user and compares it with the list of food items to detect missing items.

[0185] Specific Behavior: Compare the list of staple ingredients with the list of food items and identify any missing items.

[0186] Input: Criteria list and current food list

[0187] Processing: List comparison and missing item detection

[0188] Output: Missing items list

[0189] Step 9:

[0190] The server generates a shopping list including the missing items and transmits it to the user terminal.

[0191] Specific operation: Generate a shopping list and send it to the user's device using the REST API.

[0192] Input: Missing Item List

[0193] Output: Shopping list displayed on the user's device

[0194] Step 10:

[0195] The user shares the generated shopping list through an external communication service.

[0196] What it does: A user checks a shopping list within the app and sends it to family and other members using an external messaging service.

[0197] Input: Shopping list

[0198] Output: Shared shopping list

[0199] Through these steps, the system enables users to manage the ingredients in their refrigerator and efficiently suggest menus.

[0200] (Application example 1)

[0201] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0202] In modern households, managing food in the refrigerator and creating daily menus are tasks that require a great deal of time and effort. Creating appropriate shopping lists and replenishing necessary ingredients are also time-consuming. Furthermore, with the increasing demand for eating out and food delivery, there is a need for ways to enjoy meals efficiently without wasting ingredients at home. The present invention aims to solve these problems by providing a system that efficiently handles everything from food management to menu suggestions, shopping list creation, and delivery arrangements.

[0203] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0204] In this invention, the server includes means for acquiring image data of the contents of the refrigerator from the user terminal, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and recognizing food items in the cloud server, means for proposing menus based on the recognized food items, means for transmitting the proposed menus to the user terminal, and means for coordinating with an external service that provides the proposed menus and arranging delivery. This allows the user to reduce food waste in the refrigerator, receive optimal menu suggestions without hassle, and smoothly arrange for the delivery of ingredients that are in short supply.

[0205] A "user terminal" is an electronic device operated by a user, and has the function of acquiring image data of the inside of a refrigerator.

[0206] "Image data" is image information showing the state of food in the refrigerator, and is sent from the user terminal to the cloud server.

[0207] A "cloud server" is a remote computer connected to a network that analyzes image data and processes various types of data.

[0208] "Food item" refers to ingredients or types of ingredients recognized from image data.

[0209] A "menu" is a list of cooking suggestions created based on recognized food items.

[0210] "Suggested menu" refers to a menu list generated by the cloud server, from which the user can select.

[0211] "External services" refer to companies or systems that work in conjunction with the cloud server to deliver suggested menus and missing ingredients.

[0212] "Delivery" refers to a service that delivers meals based on a suggested menu and any missing ingredients to the user's home.

[0213] The "stock ingredient list" is a list of ingredients that the user sets in advance and that the user wants to keep in the refrigerator at all times.

[0214] A "missing ingredient" is an ingredient that is not present in the refrigerator when comparing a recognized food item with the list of ingredients.

[0215] A "shopping list" is a list of ingredients that are in short supply and ingredients that the user wishes to purchase in addition.

[0216] "External messaging service" means a communication means used by a user to send a generated shopping list to another person.

[0217] A "generative AI model" is an artificial intelligence program that has learned algorithms to perform specific tasks based on large amounts of data.

[0218] "Prompt sentence" refers to an instruction sentence that a generative AI model uses to start a task.

[0219] This invention is a system that combines user terminals, cloud servers, delivery partners, and external messaging services to provide a comprehensive range of services, from managing food in the refrigerator to suggesting menus and delivering missing ingredients.

[0220] System Overview

[0221] 1. Acquire and send image data from the user's device:

[0222] The user terminal has a camera function for taking pictures of the inside of the refrigerator.

[0223] When a user uses a smartphone app to take a photo of the inside of the refrigerator, the device sends the image data to a cloud server.

[0224] 2. Ingredient recognition and menu suggestions on the cloud server:

[0225] The cloud server analyzes the received image data and recognizes the food items in the refrigerator using a deep learning model.

[0226] The server lists food items and suggests meals based on the recognized ingredients, selected from recipes stored in a database.

[0227] The server sends the proposed menu to the user terminal, allowing the user to select a menu.

[0228] 3. Detecting missing ingredients and generating a shopping list:

[0229] The server maintains a list of ingredients that are pre-defined by the user.

[0230] Compare recognized food items with a list of staple ingredients to identify missing ingredients.

[0231] A shopping list is generated based on the ingredients that are missing, and delivery of the missing ingredients is arranged through a delivery partner if necessary.

[0232] The generated shopping list is sent to the user's terminal and can also be shared with others via external messaging services.

[0233] Hardware and software used

[0234] Hardware: Smartphone (with camera), cloud server

[0235] Software: OpenCV (for image processing), requests (for HTTP requests), deep learning models (e.g., TENSORFLOW (registered trademark), PyTorch)

[0236] Specific examples

[0237] As an example, consider a scenario where a user takes a photo of the inside of a refrigerator and uploads it to a cloud server.

[0238] The server performs image analysis and recognizes that there are tomatoes, eggs, and milk in the refrigerator.

[0239] The server proposes menu items such as "tomato omelet" and "milk pudding" based on these ingredients and sends them to the user's terminal.

[0240] The user selects "tomato omelette," the server detects the missing ingredient (e.g., "onion"), and arranges for the onion to be delivered through a delivery partner.

[0241] Example prompts from a generative AI model

[0242] An example prompt is, "Based on the ingredients in your refrigerator, suggest healthy meals you might like. If necessary, order additional ingredients through a delivery service."

[0243] These processes allow the user to efficiently manage ingredients in the refrigerator, enjoy suggested meals, and smoothly arrange for delivery of ingredients that are in short supply.

[0244] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0245] Step 1:

[0246] The user launches the smartphone app and presses the "Take a photo of the refrigerator" button. The smartphone camera starts up, and the user takes a photo of the food in the refrigerator. The photo is saved as image data on the device.

[0247] Step 2:

[0248] The device sends the captured image data to the cloud server using an HTTP request, and the data sent is in the form of an image file.

[0249] Step 3:

[0250] The server inputs the received image data into an image analysis program, which uses a deep learning model to recognize food ingredients. The input is the image data, and the output is a list of recognized food items.

[0251] Step 4:

[0252] The server runs a menu suggestion algorithm based on the list of food items. The algorithm references recipe information stored in the database and suggests menus based on the input food items. The output is a list of suggested recipes.

[0253] Step 5:

[0254] The server sends a list of suggested menus to the user's terminal. The user checks the suggested menus on the terminal and selects the menu of their choice. The selected menu becomes the input for the next step.

[0255] Step 6:

[0256] The server detects missing ingredients based on the menu selected by the user and the recognized food item list. The input is the food item list and the stock ingredient list, and the output is the missing ingredients.

[0257] Step 7:

[0258] The server generates a shopping list based on the missing ingredients. The input is the list of missing ingredients, and the output is the shopping list.

[0259] Step 8:

[0260] The server sends the generated shopping list to the user's device, which can then share it with others via an external messaging service. To share, a specific prompt is generated using the generative AI model and sent as a message.

[0261] Step 9:

[0262] The user checks the shopping list and orders any missing ingredients through an external delivery partner. The server then requests the linked external service to arrange delivery.

[0263] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0264] This invention is a system that allows users to efficiently manage food in their refrigerators and create menus, and also combines it with an emotion engine that recognizes the user's emotions and optimizes the suggestions, thereby making optimal suggestions according to the user's emotional state.

[0265] System Overview

[0266] 1. Photographing and analyzing the inside of the refrigerator

[0267] A user takes a photo of the inside of the refrigerator using a smartphone app.

[0268] The device sends the captured photo data to the cloud server, uploading the photo data via an HTTP request.

[0269] The server inputs the received photo data into image analysis software to recognize the food items in the refrigerator. Image recognition algorithms are used to label and identify each ingredient in the photo.

[0270] The server generates a list of the food items obtained and sends it to the user's app, where the user can view the list of food items in the refrigerator.

[0271] 2. Menu suggestions

[0272] The server searches a menu recipe database based on the food item list. The recipe database stores various combinations of food items and corresponding cooking recipes.

[0273] The server sends a list of found recipes to the user's app, where the user can review and select the suggested recipes.

[0274] The app displays detailed information about the recipe the user selects (e.g., ingredients, cooking instructions, cooking time, etc.).

[0275] 3. Check your stock of ingredients and create a shopping list

[0276] The server maintains the user's pre-defined list of ingredients and compares it with the current list of food in the refrigerator.

[0277] The server compares the stock of ingredients to identify any missing ingredients.

[0278] The server generates a shopping list based on the missing ingredients and sends the list to the user's app.

[0279] Users can check their shopping list from the app and share it with other family members using external messaging services such as LINE.

[0280] 4. Emotion engine integration and optimization

[0281] When a user uses the app, they input voice and make touch gestures, and this data is collected and analyzed by the emotion engine.

[0282] The device sends voice data and touch operation data to a cloud server.

[0283] The server uses an emotion engine to estimate the user's emotional state from the collected data, for example, determining whether the user is feeling stressed based on the tone of their voice or the speed of their touch gestures.

[0284] The server then adjusts the menu suggestions it provides based on the user's estimated emotional state. For example, if the user is feeling stressed, it will prioritize easy-to-prepare menu suggestions.

[0285] The server also adjusts the contents of the shopping list based on the emotional state, reducing the burden on the user.

[0286] Specific examples

[0287] 1. Photographing and analyzing the inside of the refrigerator

[0288] The user launches the smartphone app and takes a photo of the inside of the refrigerator.

[0289] The device sends the photo data to the cloud server.

[0290] The server analyzes the photo and recognizes "tomato," "egg," "cheese," and "milk."

[0291] The server generates a list of food items and sends it to the user's app.

[0292] 2. Menu suggestions

[0293] Based on the list of ingredients, the server finds recipes such as "tomato omelet," "cheese omelet," and "tomato salad" from the database.

[0294] The server sends these recipes to the user's app.

[0295] If the user selects "Tomato Omelette," detailed information is displayed.

[0296] 3. Check your stock of ingredients and create a shopping list

[0297] The server maintains and analyzes the user's stockpile of ingredients.

[0298] The server compares the list of food items in the refrigerator with the list of regularly available ingredients and detects that "onions" and "carrots" are missing.

[0299] The server generates a shopping list that includes "onions" and "carrots" and sends it to the user's app.

[0300] Users can check their shopping list, share it using LINE, and ask family members to do the shopping.

[0301] 4. Emotion engine integration and optimization

[0302] While using the app, the user utters the following using voice input: "I'm tired today."

[0303] The device sends the voice data to the cloud server.

[0304] The server uses an emotion engine to infer from the voice data that the user is tired.

[0305] The server will prioritize suggestions for dishes that are easy to prepare (e.g., "easy salad").

[0306] The server also adjusts the shopping list to include more ingredients that are easy to prepare.

[0307] This invention reduces the burden on users and supports food management and cooking tasks more efficiently by proposing optimal menus and shopping lists according to the user's emotional state.

[0308] The processing flow will be explained below.

[0309] Step 1:

[0310] The user starts the smartphone app and takes a photo of the inside of the refrigerator. The user then uses the app's camera function to take a photo of the entire refrigerator.

[0311] Step 2:

[0312] The device sends the captured photo data to the cloud server. Specifically, it creates an HTTP request including the photo data and uploads it to the cloud server.

[0313] Step 3:

[0314] The server receives the HTTP request and stores the sent photo data in cloud storage.

[0315] Step 4:

[0316] The server inputs the stored photo data into image analysis software to recognize the food items in the refrigerator. Image recognition algorithms (such as deep learning models) are used to identify the ingredients in the photo.

[0317] Step 5:

[0318] The server generates a list of food items based on the analysis results and sends the list to the user's app, where the user can view the list.

[0319] Step 6:

[0320] The server searches a menu recipe database based on the food item list. The database contains a variety of food item combinations and corresponding cooking recipes.

[0321] Step 7:

[0322] The server sends the list of recipes found to the user's app, where the user can view multiple menu suggestions and select the one they like best.

[0323] Step 8:

[0324] The user selects "Tomato Omelette" from the suggested recipes. The app displays detailed information about the selected recipe (e.g., ingredients, cooking instructions, cooking time, etc.).

[0325] Step 9:

[0326] The server retrieves the user's pre-defined ingredient list from the database, which includes ingredients and seasonings that the user frequently uses.

[0327] Step 10:

[0328] The server compares the latest list of food items in the refrigerator with the list of ingredients to identify any missing ingredients. For example, if "onions" and "carrots" are not on the list, they are recognized as missing ingredients.

[0329] Step 11:

[0330] The server generates a shopping list based on the missing ingredients, which may include the missing standard ingredients as well as additional ingredients needed for the selected meal.

[0331] Step 12:

[0332] The server sends the generated shopping list to the user's app, where the user can view the list.

[0333] Step 13:

[0334] The user presses the "Share Shopping List" button in the app and selects an external messaging service such as LINE. The device then sends the shopping list as a LINE message to share with other family members.

[0335] Step 14:

[0336] The device collects the user's voice input and touch actions and sends this data to a cloud server. For example, if a user says, "I'm tired today," the device collects this voice data.

[0337] Step 15:

[0338] The server uses an emotion engine to estimate the user's emotional state from the collected data, such as the tone of voice and the speed of touch movements, to determine whether the user is feeling stressed.

[0339] Step 16:

[0340] The server then tailors meal suggestions based on the user's estimated emotional state, for example, suggesting easy, hassle-free meals if the user is feeling stressed.

[0341] Step 17:

[0342] The server also adjusts the shopping list based on the user's emotional state: if the user is tired, for example, the list will prioritize ingredients that require less preparation.

[0343] Through the above processing steps, the system proposes optimal menus and shopping lists according to the user's emotional state. This system improves the efficiency of food management and cooking at home, reducing the burden on the user.

[0344] Example 2

[0345] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0346] For many users, efficiently managing food in the refrigerator and creating menus is a time-consuming and labor-intensive task. Furthermore, users often feel stressed and fatigued because the system does not provide optimal suggestions based on their emotional state. The present invention aims to solve these problems and enable users to manage food and create menus more effectively.

[0347] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring image data of the inside of the refrigerator from the user terminal, means for transmitting the acquired image data to the cloud server, means for analyzing the image data and recognizing food items in the cloud server, means for proposing a menu based on the recognized food items, means for transmitting the proposed menu to the user terminal, means for analyzing the emotional state of the user using an emotion analysis engine, and means for adjusting the proposed menu based on the analyzed emotional state. This not only enables the user to effectively manage the food in the refrigerator, but also enables the optimal menu to be proposed according to the user's emotional state.

[0348] 1. "User terminal" means an electronic device operated by a user, including smartphones and tablets.

[0349] 2. "Image Data" means photographs of food items in a refrigerator, stored in a digital format.

[0350] 3. "Cloud server" means a server that stores and processes data via the Internet, including a group of servers that can be accessed remotely.

[0351] 4. "Food items" refer to specific foods or ingredients in the refrigerator, and are individual ingredients recognized through image analysis.

[0352] 5. A "menu" is a combination of suggested dishes, including a list of dishes from which the user can choose.

[0353] 6. "Emotion Analysis Engine" refers to algorithms and software that analyze a user's voice and touch actions to infer their emotional state.

[0354] 7. "Means for adjusting the proposed menu" refers to means including a function for changing and optimizing the content of the menu provided to the user based on the analyzed emotional state.

[0355] 8. "Stock ingredient list" refers to a list of foods and ingredients that are used on a daily basis and that is preset by the user.

[0356] 9. "Missing ingredients" refers to foods or ingredients that are on the list but not present in the refrigerator when comparing the food in the refrigerator with the list of regular ingredients.

[0357] 10. "Shopping List" refers to a list generated based on missing materials and used by the user when shopping.

[0358] 11. "External messaging services" are tools that allow users to communicate with other users, including messaging applications via the Internet such as LINE and email.

[0359] This invention is a system that helps users manage the food in their refrigerator and create menus more efficiently, and also combines an emotion engine that recognizes the user's emotions and optimizes the suggestions. This system can make optimal suggestions according to the user's emotional state.

[0360] The system consists of user devices such as smartphones and tablets, a cloud server, image analysis software, and an emotion analysis engine.

[0361] Acquiring and sending image data inside the refrigerator

[0362] The user device has a camera function for taking photos of the inside of the refrigerator. The user launches the app and takes a photo of the inside of the refrigerator. The device then sends the captured photo data to a cloud server. This communication is performed via an HTTP request, and the data is encrypted for security reasons.

[0363] Image analysis and food item recognition

[0364] The cloud server inputs the captured photo data into image analysis software (e.g., OpenCV, AWS® Rekognition, etc.). The analysis software recognizes each food item in the refrigerator and assigns a label to each. This generates a list of food items.

[0365] Menu suggestions and adjustments

[0366] The server searches a recipe database (e.g., MySQL®, MongoDB, etc.) based on the recognized food item list. The recipe database stores combinations of food items and corresponding cooking recipes. The list of found recipes is sent to the user's app, where the user can review and select the suggested recipes.

[0367] Checking stock of materials and detecting shortages

[0368] The server maintains the user's pre-defined list of ingredients. It compares the latest refrigerator food list with the list of ingredients and identifies any missing ingredients. The server generates a shopping list based on the missing ingredients and sends it to the user's app. The user can then check the shopping list from the app and share it with other family members using messaging services such as LINE.

[0369] Sentiment analysis engine integration and optimization

[0370] When a user uses the app, voice input and touch action data are analyzed by an emotion analysis engine (e.g., IBM Watson (registered trademark), Azure (registered trademark) Cognitive Services, etc.). The device sends the voice data and touch action data to a cloud server, and the server uses the emotion analysis engine to estimate the user's emotional state. For example, it determines whether the user is feeling stressed based on the tone of the voice or the speed of the touch action. Based on the estimated emotional state, the server adjusts the content of the meal suggestions it provides. If the user is tired, it will prioritize suggestions for easy meals to make. The contents of the shopping list are also adjusted based on the user's emotional state to reduce the burden on the user.

[0371] Specific examples

[0372] For example, a user can use a smartphone app to take a photo of the inside of their refrigerator and send it to a cloud server. The server then uses image analysis software to recognize items such as "tomato," "egg," "cheese," and "milk." From the list of recognition results, the server searches for recipes such as "tomato omelette," "cheese omelette," and "tomato salad," and sends them to the user's app.

[0373] When a user voice-inputs "I'm tired today," the device sends that data to a cloud server. The emotion analysis engine recognizes the user's level of fatigue, and the server suggests easy-to-prepare meals such as "easy salads." In this way, the system suggests optimal meals based on the user's emotional state.

[0374] Prompt Sentence Examples

[0375] "Tell me about your refrigerator food management system. It uses an emotion recognition engine to suggest the best meal plan based on the user's emotional state. Specifically, how does it work and what technology is used?"

[0376] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0377] Program processing flow

[0378] Step 1:

[0379] The user launches the smartphone app and takes a photo of the inside of the refrigerator.

[0380] Input: Video data inside the refrigerator

[0381] How it works: A user takes a picture of food in the refrigerator using the camera function on their smartphone.

[0382] Output: Captured photo data

[0383] Step 2:

[0384] The device sends the photographed photo data to a cloud server.

[0385] Input: Photographed photo data

[0386] How it works: Your device uploads photo data to a cloud server via an HTTP request. The data is sent using an encrypted communication protocol.

[0387] Output: Photo data uploaded to the cloud server

[0388] Step 3:

[0389] The server inputs the photographic data into image analysis software to recognize the food items.

[0390] Input: Uploaded photo data

[0391] How it works: The server uses image analysis software (e.g., OpenCV, AWS Rekognition) to label and identify each food item in a photo, also using machine learning models in the process.

[0392] Output: A list of recognized food items

[0393] Step 4:

[0394] The server sends the list of recognized food items to the user's app.

[0395] Input: A list of recognized food items

[0396] How it works: The server sends a list of food items to the user's app using an HTTP response.

[0397] Output: A list of food items displayed in the user's app

[0398] Step 5:

[0399] The server searches a menu recipe database based on the food item list and suggests an appropriate menu.

[0400] Input: Food item list

[0401] How it works: The server queries a recipe database (e.g. MySQL, MongoDB) to find recipes based on combinations of food items.

[0402] Output: A list of suggested recipes

[0403] Step 6:

[0404] The server sends the suggested recipe list to the user's app.

[0405] Input: Suggested recipe list

[0406] How it works: The server sends the recipe list to the user's app using an HTTP response.

[0407] Output: A list of recipes displayed in the user's app

[0408] Step 7:

[0409] The user selects a desired recipe from the suggested recipe list and displays its detailed information.

[0410] Input: Suggested recipe list

[0411] What it does: A user selects a recipe in the app and sees details (ingredients, cooking steps, cooking time, etc.).

[0412] Output: Details of the selected recipe displayed in the user's app

[0413] Step 8:

[0414] The server maintains a user-specified list of ingredients that are always on hand and compares it with a list of recognized food items to detect missing ingredients.

[0415] Input: List of ingredients and food items

[0416] How it works: The server compares both lists and identifies any missing ingredients.

[0417] Output: List of missing materials

[0418] Step 9:

[0419] The server generates a shopping list including missing ingredients and sends it to the user's app.

[0420] Input: Shortage material list

[0421] How it works: The server generates a shopping list and sends it to the user's app in an HTTP response.

[0422] Output: Shopping list displayed in the user's app

[0423] Step 10:

[0424] The user can view the shopping list from the app and share it with other family members using an external messaging service.

[0425] Input: Shopping list

[0426] How it works: A user uses the share button in the app to share their shopping list via LINE or email.

[0427] Output: Shopping list shared by family members

[0428] Step 11:

[0429] The server uses an emotion analysis engine to analyze the user's emotional state.

[0430] Input: Voice data, touch operation data

[0431] How it works: The server uses an emotion analysis engine (e.g., IBM Watson, Azure Cognitive Services) to analyze the tone of voice and the speed of touch movements to estimate the user's emotional state.

[0432] Output: Estimation of the user's emotional state

[0433] Step 12:

[0434] The server adjusts menu suggestions based on the estimated emotional state.

[0435] Input: Emotional state estimation result, suggested recipe list

[0436] How it works: The server changes the priority of meal suggestions based on the user's emotional state, and when stress is high, prioritizes easy-to-prepare meals.

[0437] Output: A menu list tailored based on emotional state

[0438] Through this series of processes, users can efficiently manage the food in their refrigerator and are presented with the optimal menu based on their emotional state.

[0439] (Application example 2)

[0440] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0441] Conventional food management systems only recognize food items in the refrigerator and suggest menu items, but do not provide optimal suggestions based on the user's emotional state. Furthermore, they lacked a mechanism for automatically adding data on food purchased in physical stores to the refrigerator list, placing a heavy burden on the user as they had to manually enter the information. Furthermore, the system sometimes found it inconvenient to provide a shopping list that was optimal for the user's situation or emotions.

[0442] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data of the inside of the refrigerator from the user terminal, means for transmitting the acquired image data to the cloud server, means for analyzing the image data and recognizing food items in the cloud server, means for proposing a menu based on the recognized food items, means for transmitting the proposed menu to the user terminal, means for scanning barcodes of food items from the user terminal in the physical store and adding the data to the refrigerator list, means for adjusting the menu based on the user's emotional state, means for analyzing the user's voice data and operation actions using an emotion engine and estimating the user's emotional state, and means for generating an optimal shopping list based on the emotional state. This enables suggestions based on the user's emotional state and efficient food management in the physical store.

[0443] A "user terminal" is a mobile information terminal such as a smartphone or tablet that is operated by a user.

[0444] "Image data" refers to image or photograph data of the state inside the refrigerator.

[0445] A "cloud server" is a server system for storing and processing data on a remote server on the Internet.

[0446] "Food items" refers to the types of ingredients and foods contained in the refrigerator.

[0447] A "menu" refers to a list or recipe of suggested dishes based on ingredients.

[0448] A "barcode" is an identification code consisting of a series of black lines and spaces used to represent product information.

[0449] "Emotional state" refers to the emotions and mental state that the user is currently experiencing.

[0450] An "emotion engine" is software or algorithms for analyzing and inferring a user's emotional state.

[0451] "Voice data" refers to the voice of a user recorded as digital data.

[0452] "Operation actions" refer to actions such as touching and swiping that a user performs on a device.

[0453] A "shopping list" is a list of food and daily necessities that need to be purchased.

[0454] This invention is a system that allows users to efficiently manage food in their refrigerators and create menus, and also combines it with an emotion engine that recognizes the user's emotions and optimizes the suggestions, thereby making optimal suggestions according to the user's emotional state.

[0455] System Overview

[0456] 1. Photographing and analyzing the inside of the refrigerator

[0457] The user uses a smartphone app to capture image data of the inside of the refrigerator using the smartphone's camera function.

[0458] The device sends the acquired image data to the cloud server using an HTTP request.

[0459] The server analyzes the image data on the cloud and recognizes food items. Image analysis is performed using OpenCV.

[0460] The server generates a list of recognized food items and transmits it to the user terminal.

[0461] 2. Menu suggestions

[0462] The server searches a menu recipe database based on the food item list, which stores various combinations of food items and corresponding cooking recipes.

[0463] The server sends a list of found recipes to the user's device, and the user can review and select suggested recipes on the app.

[0464] 3. Check your stock of ingredients and create a shopping list

[0465] The server maintains a list of ingredients on hand designated by the user on a cloud server and compares the food items recognized from the transmitted image data with the list of ingredients on hand.

[0466] The server identifies the missing ingredients and generates a shopping list that includes them.

[0467] The server sends the generated shopping list to the user terminal, where the user can share it with others using an external messaging service.

[0468] 4. Emotion engine integration and optimization

[0469] When a user expresses their emotional state through voice input while operating the app, the voice data and operation data are sent to the emotion engine, which uses the Hugging Face Transformers library.

[0470] The server uses an emotion engine to infer the user's emotional state from the data, for example, by analyzing the tone of voice or the speed of haptic movements.

[0471] The server adjusts the meal suggestions and shopping list it provides based on the user's emotional state. For example, if the user is feeling stressed, it will prioritize easy-to-prepare meals.

[0472] Physical store operations

[0473] Furthermore, this system is also intended for use in physical stores. For example, when a user purchases food at a supermarket, they can scan the barcode on the product with a smartphone app. The barcode data is sent to the server and immediately added to the refrigerator list. This allows the user to automatically update their shopping list and manage it efficiently.

[0474] Specific examples

[0475] Food Recognition in the Refrigerator:

[0476] Users take a photo of their refrigerator with their smartphone and upload it to the app, where the food recognition system identifies "tomatoes," "eggs," "cheese," and "milk," and creates a list of these foods.

[0477] Menu suggestions based on user sentiment:

[0478] When a user voice-inputs into the app, "I'm busy and tired today," the emotion engine recognizes the "tired" state and suggests an easy-to-make "tomato salad."

[0479] Examples of prompts:

[0480] Take a picture of the inside of your refrigerator and identify the following food items: tomatoes, eggs, cheese, and milk. Based on this information, suggest a simple meal plan. Also include suggestions for when the user is tired.

[0481] In this way, appropriate suggestions based on the user's emotional state and efficient food management in physical stores can be realized.

[0482] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0483] Step 1:

[0484] A user uses a smartphone app to capture image data of the inside of a refrigerator. The user launches the app and takes a photo of the inside of the refrigerator with the smartphone camera. Once the photo is captured, the app prepares the image data.

[0485] Input: Photo data of the inside of a refrigerator taken with a smartphone camera.

[0486] Output: The image data is saved in the app on your smartphone.

[0487] Step 2:

[0488] The device sends the acquired image data to the cloud server, and the smartphone app uses an HTTP request to upload this image data to the cloud server.

[0489] Input: The acquired image data.

[0490] Output: The image data is transferred to the cloud server.

[0491] Step 3:

[0492] The server analyzes the image data on the cloud and recognizes food items. OpenCV is used to automatically identify food items from the image data and assign labels to each ingredient.

[0493] Input: Image data sent to the cloud server.

[0494] Output: A list of recognized food items.

[0495] Step 4:

[0496] The server searches a menu recipe database based on the recognized food items, uses the food list to search the database for related cooking recipes, and generates a list of found recipes.

[0497] Input: A list of recognized food items.

[0498] Output: A list of recipes.

[0499] Step 5:

[0500] The server sends a list of found recipes to the user's device. The cloud server prepares the recipe list and sends it to the smartphone app using an HTTP request.

[0501] Input: A list of recipes.

[0502] Output: The recipe list is displayed on the smartphone app.

[0503] Step 6:

[0504] A user scans the barcode of a food item at a physical store. The user activates the barcode scanner on their smartphone app and scans the barcode of the food item they want to purchase. The barcode data is sent by the app to a cloud server.

[0505] Input: Scanned barcode data.

[0506] Output: The barcode data is sent to the cloud server.

[0507] Step 7:

[0508] The server receives the barcode data and updates the refrigerator list accordingly, so that the newly added food item is reflected in the refrigerator list.

[0509] Input: Barcode data.

[0510] Output: Updated refrigerator list.

[0511] Step 8:

[0512] The user inputs their emotional state using the app. The user expresses their emotional state, such as "I'm tired today," through voice input or text input. The emotional data is sent to a cloud server.

[0513] Input: User's emotional state (audio or text data).

[0514] Output: Emotion data is sent to the cloud server.

[0515] Step 9:

[0516] The server uses an emotion engine to analyze the emotion data and estimate the user's emotional state. It uses the Hugging Face Transformers library to analyze voice tones and operational movements.

[0517] Input: Emotion data.

[0518] Output: Estimated emotional state.

[0519] Step 10:

[0520] The server then adjusts the menu suggestions it provides based on the user's estimated emotional state. For example, if the user is feeling stressed, it will prioritize easy-to-make recipes.

[0521] Input: Emotional state.

[0522] Output: Adjusted menu suggestions.

[0523] Step 11:

[0524] The server compares the list of ingredients with the list of recognized food items to identify missing ingredients, creates a shopping list based on the missing ingredients, and sends it to the user's device.

[0525] Input: Stock ingredient list and recognized food item list.

[0526] Output: A shopping list including missing ingredients is generated and sent to the user's device.

[0527] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0528] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0529] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0530] [Second embodiment]

[0531] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0532] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0533] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0534] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0535] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0536] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0537] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0538] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0539] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0540] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0541] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0542] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0543] This invention is a system for streamlining food management and menu suggestions. It automatically recognizes ingredients in the refrigerator and suggests menus based on them, so that users don't have to waste time creating daily menus and managing shopping lists. It also provides a function to detect shortages of regular ingredients and create and share shopping lists.

[0544] System Overview

[0545] 1. Photographing and analyzing the inside of the refrigerator

[0546] A user uses a smartphone app to take a photo of the inside of the refrigerator.

[0547] The device sends the photographed photo data to a cloud server.

[0548] The server inputs the photo data into image analysis software to recognize the food items in the refrigerator. Specifically, the server uses image analysis algorithms (e.g., deep learning techniques) to identify each ingredient in the photo and assign it a label.

[0549] The server generates a list of recognized food items and sends it to the user's app, where the user can view the list of food items in the refrigerator.

[0550] 2. Menu suggestions

[0551] Based on the analysis results, the server searches a database for menu recipes using the food items in the refrigerator. The recipe database stores various combinations of food items and corresponding cooking recipes.

[0552] The server sends the list of found recipes to the user's app.

[0553] Users can select their favorite recipe from the suggested recipes, and the details of the selected recipe (ingredients, cooking method, required time, etc.) will be displayed.

[0554] 3. Check your stock of ingredients and create a shopping list

[0555] The cloud server stores a list of ingredients that the user has pre-set (for example, "onions, carrots, soy sauce").

[0556] The server compares the current food item list with the stock ingredient list to identify any missing food items.

[0557] The server generates a shopping list based on the missing food items, which can include any missing staple ingredients as well as any additional ingredients needed for the user's selected meal.

[0558] Users can check their shopping list in the app and share it using external messaging services (e.g., LINE), allowing them to ask other family members to do the shopping.

[0559] Specific examples

[0560] 1. Photographing and analyzing the inside of the refrigerator

[0561] The user launches the smartphone app and presses the "take a photo of the inside of the refrigerator" button.

[0562] The device uses its camera function to take pictures of the inside of the refrigerator and sends the image data to a cloud server.

[0563] The server analyzes the image and recognizes that there are tomatoes, eggs, cheese, and milk in the refrigerator.

[0564] The server generates a list of food items and sends it to the user's app.

[0565] 2. Menu suggestions

[0566] The server searches the database based on the list of ingredients and finds recipes such as "tomato omelet," "cheese omelet," and "tomato salad."

[0567] The server sends these recipes to the user's app.

[0568] When a user selects "Tomato Omelette" from the app, the recipe details are displayed.

[0569] 3. Check your stock of ingredients and create a shopping list

[0570] The server maintains a list of ingredients (e.g., onions, carrots, soy sauce).

[0571] The server compares the latest list of food items in the refrigerator with the list of regularly-available ingredients and detects that there are shortages of "onions" and "carrots."

[0572] The server generates a shopping list for "onions, carrots" and sends it to the user's app.

[0573] The user checks the shopping list on the app and sends it to her husband via LINE. On the way home, the husband buys onions and carrots at the supermarket.

[0574] As described above, the present invention reduces the burden on the user and provides a method for efficiently managing ingredients and creating menus at home.

[0575] The processing flow will be explained below.

[0576] Step 1:

[0577] The user launches the smartphone app and takes a photo of the inside of the refrigerator. The user then uses the camera function to capture an overall image of the inside of the refrigerator, and presses the "Send" button when the photo is complete.

[0578] Step 2:

[0579] The device sends the captured photo data to the cloud server. Specifically, the device sends an HTTP request to upload the photo data to the server.

[0580] Step 3:

[0581] The server receives the HTTP request and stores the transmitted photo data, ensuring that the server has the necessary photo data for future processing.

[0582] Step 4:

[0583] The server inputs the stored photo data into image analysis software to recognize the food items in the refrigerator. Specifically, it uses image recognition algorithms such as deep learning to identify each ingredient in the photo and assign it an individual label.

[0584] Step 5:

[0585] The server generates a list of food items based on the analysis results and sends the list to the user's app, where the user can view the list of food items in the refrigerator.

[0586] Step 6:

[0587] The server searches the menu recipe database based on the food item list. The recipe database stores various combinations of food items and corresponding cooking recipes. The server identifies recipes for dishes using the ingredients in the refrigerator.

[0588] Step 7:

[0589] The server sends a list of found recipes to the user's app, where the user can select the recipes they like.

[0590] Step 8:

[0591] The user uses the app to review the suggested recipes and selects a recipe, such as "Tomato Omelette." Details of the selected recipe (e.g., ingredients, cooking instructions, cooking time, etc.) are displayed in the app.

[0592] Step 9:

[0593] The server retrieves the user's predefined list of ingredients (e.g., "onions, carrots, soy sauce") from the database. The server compares this list with the latest list of food in the refrigerator.

[0594] Step 10:

[0595] The server compares the list of ingredients in stock with the list of food in the refrigerator and identifies any ingredients that are missing. For example, it determines that "onions" and "carrots" are missing.

[0596] Step 11:

[0597] The server generates a shopping list based on the missing ingredients, which can include the missing regular ingredients as well as any additional ingredients needed for the recipe selected by the user.

[0598] Step 12:

[0599] The server sends the generated shopping list to the user's app, where the user can view the shopping list.

[0600] Step 13:

[0601] The user presses the "Share Shopping List" button in the app and selects an external messaging service such as LINE. The device then sends the shopping list as a LINE message to share with other family members.

[0602] The above are the specific processing steps of the system, which allow users to proactively manage food and create menus.

[0603] Example 1

[0604] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0605] Conventional food management and menu suggestion systems have the problem that users have to manually check the ingredients in the refrigerator and search for recipes, which is time-consuming. Furthermore, to determine if a necessary ingredient is missing, the user must check the refrigerator again, making efficient food management difficult. This invention aims to reduce the user's burden and provide efficient food management and menu suggestions by automatically recognizing ingredients in the refrigerator using image analysis technology and suggesting menus based on that information.

[0606] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0607] In this invention, the server includes means for acquiring image data from a user terminal, means for transmitting the acquired image data via a network, means for analyzing the image data and recognizing object items in the server, means for providing information based on the recognized object items, and means for transmitting the provided information to the user terminal. This reduces the user's workload and enables efficient food management and menu suggestions. The server also includes means for storing a reference list specified by the user, comparing the object items recognized from the transmitted image data with the reference list to detect missing items, generating a list including the missing items, and means for transmitting the generated list to the user terminal, allowing the user to easily identify missing ingredients and use it as a shopping list. Furthermore, the server also includes means for transmitting the generated list from the user terminal using an external communication service, allowing the user to share the shopping list with family and other members and shop collaboratively.

[0608] "User terminal" refers to a device operated by a user, including a smartphone, tablet, or PC.

[0609] "Image data" refers to photographs or scanned images of the inside of the refrigerator, including visual information stored in digital form.

[0610] "Network" refers to a communications system that enables data to be communicated between devices, including the Internet and local area networks.

[0611] A "server" refers to a central computer system that processes data and provides information in response to requests from clients (user terminals).

[0612] "Object item" refers to an individual entity (e.g., food or object) recognized from image data.

[0613] "Information" is content provided based on the object item, including recipes, ingredient lists, etc.

[0614] The "standard list" refers to a list of items that should be kept as standard, set by the user, and includes, for example, a list of regular materials.

[0615] "External communication services" are services used to send and receive data, including messaging applications and email services.

[0616] A "list" is a list of specific items, including a list of missing items and a shopping list.

[0617] This invention is a system for streamlining food management and menu suggestions. It combines a user terminal, a server, an image analysis algorithm, and an external communication service to significantly reduce the user's workload.

[0618] System Configuration

[0619] The system mainly consists of the following hardware and software elements:

[0620] User device: Use a smartphone, tablet, or computer.

[0621] Server: A cloud server is used to process data from multiple user devices.

[0622] Image analysis algorithm: Uses deep learning models to perform object recognition.

[0623] External communication services: Using messaging applications (e.g., messaging apps, email).

[0624] Operating principle

[0625] The system's operation can be broadly divided into the following steps:

[0626] Taking photos of the inside of the refrigerator and analyzing them

[0627] A user uses a smartphone app to take a photo of the inside of their refrigerator. The device then sends the captured photo data to a cloud server. The server then uses an image analysis algorithm (deep learning technology) to recognize food items in the photo and assigns labels to each. Specifically, it uses an object recognition algorithm to extract food items such as "tomato," "egg," and "cheese." The server then generates a list of the recognized food items and sends it to the user's app.

[0628] Menu suggestions

[0629] Based on the analysis results, the server searches a database for menu recipes using the food items in the refrigerator. The recipe database stores various combinations of food items and corresponding cooking recipes. For example, a database search can find recipes such as "tomato omelette," "cheese omelette," and "tomato salad." The server sends a list of these recipes to the user's app, and the user can select their preferred recipe from the suggested recipes. Details of the selected recipe (ingredients, cooking method, required time, etc.) are also displayed in the app.

[0630] Check your stock of ingredients and create a shopping list

[0631] The server maintains a standard list (e.g., onions, carrots, soy sauce, etc.) preset by the user. The server compares the latest list of food items in the refrigerator with the standard list to identify any missing food items. Specifically, if the server detects that "onions" and "carrots" are missing, it generates a shopping list that includes these items. The shopping list can include not only the missing basic ingredients but also additional ingredients required for the meal selected by the user. The user can check the shopping list in the app and share it with other family members using an external communication service (e.g., a messaging app).

[0632] Specific examples

[0633] 1. Photographing and analyzing the inside of the refrigerator

[0634] The user launches the app and presses the "Take a photo of the inside of the refrigerator" button.

[0635] The device uses its camera function to take pictures of the inside of the refrigerator and sends the image data to the server.

[0636] The server analyzes the image and recognizes that there are tomatoes, eggs, cheese, and milk in the refrigerator.

[0637] The server generates a list of food items and sends it to the user's app.

[0638] 2. Menu suggestions

[0639] The server searches the database based on the list of ingredients and finds recipes such as "tomato omelet," "cheese omelet," and "tomato salad."

[0640] The server sends these recipes to the user's app.

[0641] When a user selects "Tomato Omelette" from the app, the recipe details are displayed.

[0642] 3. Check your stock of ingredients and create a shopping list

[0643] The server has a pre-stored list of criteria (e.g., onions, carrots, soy sauce).

[0644] The server compares the current food list with the reference list and identifies that "onions" and "carrots" are in short supply.

[0645] The server generates a shopping list that includes "onions, carrots" and sends it to the user's app.

[0646] A user sends a shopping list to their family via a messaging app, and the family members then go to the supermarket to purchase the items.

[0647] Prompt Sentence Examples

[0648] "Describe a system that recognizes the foods in a refrigerator, generates a list, and notifies the user of that list."

[0649] "Please explain the process of a system that suggests recipes based on a list of foods in the refrigerator."

[0650] "Describe a system that detects shortages of staple ingredients and automatically generates a shopping list."

[0651] In this way, it is possible to significantly improve the efficiency of the user's food ingredient management and menu proposals. This system reduces the burden on the user and provides a method for efficiently managing food at home and creating menus.

[0652] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0653] Step 1:

[0654] The user launches the smartphone app and takes a photo of the inside of the refrigerator.

[0655] Specific operation: The user presses the "Take a photo inside the refrigerator" button and takes a photo of the inside of the refrigerator using the smartphone camera.

[0656] Input: Visual information inside the refrigerator

[0657] Output: Image data

[0658] Step 2:

[0659] The device sends the photographed photo data to a cloud server.

[0660] Specific behavior: Uploads an image to the server using the HTTPS protocol.

[0661] Input: Image data

[0662] Output: Image data stored on a cloud server

[0663] Step 3:

[0664] The server uses image analysis algorithms (e.g., deep learning techniques) to recognize food items in the refrigerator.

[0665] Input: Image data stored on a cloud server

[0666] Processing: Object recognition with deep learning models

[0667] Output: A list of recognized food items

[0668] What it does: The server runs an image analysis model to identify food items such as "tomato," "egg," and "cheese."

[0669] Step 4:

[0670] The server sends the recognized food item list to the user terminal.

[0671] Specific operation: The server transfers the food item list generated to the user's device using the REST API.

[0672] Input: A list of recognized food items

[0673] Output: A list of food items in the refrigerator displayed on the user's device

[0674] Step 5:

[0675] The server searches a database based on the list of food items to find available recipes.

[0676] What it does: Uses a database query to extract the corresponding recipes.

[0677] Input: Food item list

[0678] Processing: Database queries and recipe extraction

[0679] Output: List of available recipes

[0680] Step 6:

[0681] The server sends the recipe list it finds to the user's terminal.

[0682] Specific operation: Send the recipe list to the user's terminal using the REST API.

[0683] Input: Available recipe list

[0684] Output: Recipe list displayed on the user's device

[0685] Step 7:

[0686] The user selects one of the suggested recipes and displays the details.

[0687] Specific behavior: Displays details (ingredients, cooking instructions, cooking time, etc.) of a recipe selected by the user from the app.

[0688] Input: Recipe ID selected by the user

[0689] Output: Detailed information about the selected recipe

[0690] Step 8:

[0691] The server maintains a list of criteria set by the user and compares it with the list of food items to detect missing items.

[0692] Specific Behavior: Compare the list of staple ingredients with the list of food items and identify any missing items.

[0693] Input: Criteria list and current food list

[0694] Processing: List comparison and missing item detection

[0695] Output: Missing items list

[0696] Step 9:

[0697] The server generates a shopping list including the missing items and transmits it to the user terminal.

[0698] Specific operation: Generate a shopping list and send it to the user's device using the REST API.

[0699] Input: Missing Item List

[0700] Output: Shopping list displayed on the user's device

[0701] Step 10:

[0702] The user shares the generated shopping list through an external communication service.

[0703] What it does: A user checks a shopping list within the app and sends it to family and other members using an external messaging service.

[0704] Input: Shopping list

[0705] Output: Shared shopping list

[0706] Through these steps, the system enables users to manage the ingredients in their refrigerator and efficiently suggest menus.

[0707] (Application example 1)

[0708] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0709] In modern households, managing food in the refrigerator and creating daily menus are tasks that require a great deal of time and effort. Creating appropriate shopping lists and replenishing necessary ingredients are also time-consuming. Furthermore, with the increasing demand for eating out and food delivery, there is a need for ways to enjoy meals efficiently without wasting ingredients at home. The present invention aims to solve these problems by providing a system that efficiently handles everything from food management to menu suggestions, shopping list creation, and delivery arrangements.

[0710] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0711] In this invention, the server includes means for acquiring image data of the contents of the refrigerator from the user terminal, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and recognizing food items in the cloud server, means for proposing menus based on the recognized food items, means for transmitting the proposed menus to the user terminal, and means for coordinating with an external service that provides the proposed menus and arranging delivery. This allows the user to reduce food waste in the refrigerator, receive optimal menu suggestions without hassle, and smoothly arrange for the delivery of ingredients that are in short supply.

[0712] A "user terminal" is an electronic device operated by a user, and has the function of acquiring image data of the inside of a refrigerator.

[0713] "Image data" is image information showing the state of food in the refrigerator, and is sent from the user terminal to the cloud server.

[0714] A "cloud server" is a remote computer connected to a network that analyzes image data and processes various types of data.

[0715] "Food item" refers to ingredients or types of ingredients recognized from image data.

[0716] A "menu" is a list of cooking suggestions created based on recognized food items.

[0717] "Suggested menu" refers to a menu list generated by the cloud server, from which the user can select.

[0718] "External services" refer to companies or systems that work in conjunction with the cloud server to deliver suggested menus and missing ingredients.

[0719] "Delivery" refers to a service that delivers meals based on a suggested menu and any missing ingredients to the user's home.

[0720] The "stock ingredient list" is a list of ingredients that the user sets in advance and that the user wants to keep in the refrigerator at all times.

[0721] A "missing ingredient" is an ingredient that is not present in the refrigerator when comparing a recognized food item with the list of ingredients.

[0722] A "shopping list" is a list of ingredients that are in short supply and ingredients that the user wishes to purchase in addition.

[0723] "External messaging service" means a communication means used by a user to send a generated shopping list to another person.

[0724] A "generative AI model" is an artificial intelligence program that has learned algorithms to perform specific tasks based on large amounts of data.

[0725] "Prompt sentence" refers to an instruction sentence that a generative AI model uses to start a task.

[0726] This invention is a system that combines user terminals, cloud servers, delivery partners, and external messaging services to provide a comprehensive range of services, from managing food in the refrigerator to suggesting menus and delivering missing ingredients.

[0727] System Overview

[0728] 1. Acquire and send image data from the user's device:

[0729] The user terminal has a camera function for taking pictures of the inside of the refrigerator.

[0730] When a user uses a smartphone app to take a photo of the inside of the refrigerator, the device sends the image data to a cloud server.

[0731] 2. Ingredient recognition and menu suggestions on the cloud server:

[0732] The cloud server analyzes the received image data and recognizes the food items in the refrigerator using a deep learning model.

[0733] The server lists food items and suggests meals based on the recognized ingredients, selected from recipes stored in a database.

[0734] The server sends the proposed menu to the user terminal, allowing the user to select a menu.

[0735] 3. Detecting missing ingredients and generating a shopping list:

[0736] The server maintains a list of ingredients that are pre-defined by the user.

[0737] Compare recognized food items with a list of staple ingredients to identify missing ingredients.

[0738] A shopping list is generated based on the ingredients that are missing, and delivery of the missing ingredients is arranged through a delivery partner if necessary.

[0739] The generated shopping list is sent to the user's terminal and can also be shared with others via external messaging services.

[0740] Hardware and software used

[0741] Hardware: Smartphone (with camera), cloud server

[0742] Software: OpenCV (for image processing), requests (for HTTP requests), deep learning models (e.g. TensorFlow, PyTorch)

[0743] Specific examples

[0744] As an example, consider a scenario where a user takes a photo of the inside of a refrigerator and uploads it to a cloud server.

[0745] The server performs image analysis and recognizes that there are tomatoes, eggs, and milk in the refrigerator.

[0746] The server proposes menu items such as "tomato omelet" and "milk pudding" based on these ingredients and sends them to the user's terminal.

[0747] The user selects "tomato omelette," the server detects the missing ingredient (e.g., "onion"), and arranges for the onion to be delivered through a delivery partner.

[0748] Example prompts from a generative AI model

[0749] An example prompt is, "Based on the ingredients in your refrigerator, suggest healthy meals you might like. If necessary, order additional ingredients through a delivery service."

[0750] These processes allow the user to efficiently manage ingredients in the refrigerator, enjoy suggested meals, and smoothly arrange for delivery of ingredients that are in short supply.

[0751] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0752] Step 1:

[0753] The user launches the smartphone app and presses the "Take a photo of the refrigerator" button. The smartphone camera starts up, and the user takes a photo of the food in the refrigerator. The photo is saved as image data on the device.

[0754] Step 2:

[0755] The device sends the captured image data to the cloud server using an HTTP request, and the data sent is in the form of an image file.

[0756] Step 3:

[0757] The server inputs the received image data into an image analysis program, which uses a deep learning model to recognize food ingredients. The input is the image data, and the output is a list of recognized food items.

[0758] Step 4:

[0759] The server runs a menu suggestion algorithm based on the list of food items. The algorithm references recipe information stored in the database and suggests menus based on the input food items. The output is a list of suggested recipes.

[0760] Step 5:

[0761] The server sends a list of suggested menus to the user's terminal. The user checks the suggested menus on the terminal and selects the menu of their choice. The selected menu becomes the input for the next step.

[0762] Step 6:

[0763] The server detects missing ingredients based on the menu selected by the user and the recognized food item list. The input is the food item list and the stock ingredient list, and the output is the missing ingredients.

[0764] Step 7:

[0765] The server generates a shopping list based on the missing ingredients. The input is the list of missing ingredients, and the output is the shopping list.

[0766] Step 8:

[0767] The server sends the generated shopping list to the user's device, which can then share it with others via an external messaging service. To share, a specific prompt is generated using the generative AI model and sent as a message.

[0768] Step 9:

[0769] The user checks the shopping list and orders any missing ingredients through an external delivery partner. The server then requests the linked external service to arrange delivery.

[0770] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0771] This invention is a system that allows users to efficiently manage food in their refrigerators and create menus, and also combines it with an emotion engine that recognizes the user's emotions and optimizes the suggestions, thereby making optimal suggestions according to the user's emotional state.

[0772] System Overview

[0773] 1. Photographing and analyzing the inside of the refrigerator

[0774] A user takes a photo of the inside of the refrigerator using a smartphone app.

[0775] The device sends the captured photo data to the cloud server, uploading the photo data via an HTTP request.

[0776] The server inputs the received photo data into image analysis software to recognize the food items in the refrigerator. Image recognition algorithms are used to label and identify each ingredient in the photo.

[0777] The server generates a list of the food items obtained and sends it to the user's app, where the user can view the list of food items in the refrigerator.

[0778] 2. Menu suggestions

[0779] The server searches a menu recipe database based on the food item list. The recipe database stores various combinations of food items and corresponding cooking recipes.

[0780] The server sends a list of found recipes to the user's app, where the user can review and select the suggested recipes.

[0781] The app displays detailed information about the recipe the user selects (e.g., ingredients, cooking instructions, cooking time, etc.).

[0782] 3. Check your stock of ingredients and create a shopping list

[0783] The server maintains the user's pre-defined list of ingredients and compares it with the current list of food in the refrigerator.

[0784] The server compares the stock of ingredients to identify any missing ingredients.

[0785] The server generates a shopping list based on the missing ingredients and sends the list to the user's app.

[0786] Users can check their shopping list from the app and share it with other family members using external messaging services such as LINE.

[0787] 4. Emotion engine integration and optimization

[0788] When a user uses the app, they input voice and make touch gestures, and this data is collected and analyzed by the emotion engine.

[0789] The device sends voice data and touch operation data to a cloud server.

[0790] The server uses an emotion engine to estimate the user's emotional state from the collected data, for example, determining whether the user is feeling stressed based on the tone of their voice or the speed of their touch gestures.

[0791] The server then adjusts the menu suggestions it provides based on the user's estimated emotional state. For example, if the user is feeling stressed, it will prioritize easy-to-prepare menu suggestions.

[0792] The server also adjusts the contents of the shopping list based on the emotional state, reducing the burden on the user.

[0793] Specific examples

[0794] 1. Photographing and analyzing the inside of the refrigerator

[0795] The user launches the smartphone app and takes a photo of the inside of the refrigerator.

[0796] The device sends the photo data to the cloud server.

[0797] The server analyzes the photo and recognizes "tomato," "egg," "cheese," and "milk."

[0798] The server generates a list of food items and sends it to the user's app.

[0799] 2. Menu suggestions

[0800] Based on the list of ingredients, the server finds recipes such as "tomato omelet," "cheese omelet," and "tomato salad" from the database.

[0801] The server sends these recipes to the user's app.

[0802] If the user selects "Tomato Omelette," detailed information is displayed.

[0803] 3. Check your stock of ingredients and create a shopping list

[0804] The server maintains and analyzes the user's stockpile of ingredients.

[0805] The server compares the list of food items in the refrigerator with the list of regularly available ingredients and detects that "onions" and "carrots" are missing.

[0806] The server generates a shopping list that includes "onions" and "carrots" and sends it to the user's app.

[0807] Users can check their shopping list, share it using LINE, and ask family members to do the shopping.

[0808] 4. Emotion engine integration and optimization

[0809] While using the app, the user utters the following using voice input: "I'm tired today."

[0810] The device sends the voice data to the cloud server.

[0811] The server uses an emotion engine to infer from the voice data that the user is tired.

[0812] The server will prioritize suggestions for dishes that are easy to prepare (e.g., "easy salad").

[0813] The server also adjusts the shopping list to include more ingredients that are easy to prepare.

[0814] This invention reduces the burden on users and supports food management and cooking tasks more efficiently by proposing optimal menus and shopping lists according to the user's emotional state.

[0815] The processing flow will be explained below.

[0816] Step 1:

[0817] The user starts the smartphone app and takes a photo of the inside of the refrigerator. The user then uses the app's camera function to take a photo of the entire refrigerator.

[0818] Step 2:

[0819] The device sends the captured photo data to the cloud server. Specifically, it creates an HTTP request including the photo data and uploads it to the cloud server.

[0820] Step 3:

[0821] The server receives the HTTP request and stores the sent photo data in cloud storage.

[0822] Step 4:

[0823] The server inputs the stored photo data into image analysis software to recognize the food items in the refrigerator. Image recognition algorithms (such as deep learning models) are used to identify the ingredients in the photo.

[0824] Step 5:

[0825] The server generates a list of food items based on the analysis results and sends the list to the user's app, where the user can view the list.

[0826] Step 6:

[0827] The server searches a menu recipe database based on the food item list. The database contains a variety of food item combinations and corresponding cooking recipes.

[0828] Step 7:

[0829] The server sends the list of recipes found to the user's app, where the user can view multiple menu suggestions and select the one they like best.

[0830] Step 8:

[0831] The user selects "Tomato Omelette" from the suggested recipes. The app displays detailed information about the selected recipe (e.g., ingredients, cooking instructions, cooking time, etc.).

[0832] Step 9:

[0833] The server retrieves the user's pre-defined ingredient list from the database, which includes ingredients and seasonings that the user frequently uses.

[0834] Step 10:

[0835] The server compares the latest list of food items in the refrigerator with the list of ingredients to identify any missing ingredients. For example, if "onions" and "carrots" are not on the list, they are recognized as missing ingredients.

[0836] Step 11:

[0837] The server generates a shopping list based on the missing ingredients, which may include the missing standard ingredients as well as additional ingredients needed for the selected meal.

[0838] Step 12:

[0839] The server sends the generated shopping list to the user's app, where the user can view the list.

[0840] Step 13:

[0841] The user presses the "Share Shopping List" button in the app and selects an external messaging service such as LINE. The device then sends the shopping list as a LINE message to share with other family members.

[0842] Step 14:

[0843] The device collects the user's voice input and touch actions and sends this data to a cloud server. For example, if a user says, "I'm tired today," the device collects this voice data.

[0844] Step 15:

[0845] The server uses an emotion engine to estimate the user's emotional state from the collected data, such as the tone of voice and the speed of touch movements, to determine whether the user is feeling stressed.

[0846] Step 16:

[0847] The server then tailors meal suggestions based on the user's estimated emotional state, for example, suggesting easy, hassle-free meals if the user is feeling stressed.

[0848] Step 17:

[0849] The server also adjusts the shopping list based on the user's emotional state: if the user is tired, for example, the list will prioritize ingredients that require less preparation.

[0850] Through the above processing steps, the system proposes optimal menus and shopping lists according to the user's emotional state. This system improves the efficiency of food management and cooking at home, reducing the burden on the user.

[0851] Example 2

[0852] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0853] For many users, efficiently managing food in the refrigerator and creating menus is a time-consuming and labor-intensive task. Furthermore, users often feel stressed and fatigued because the system does not provide optimal suggestions based on their emotional state. The present invention aims to solve these problems and enable users to manage food and create menus more effectively.

[0854] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring image data of the inside of the refrigerator from the user terminal, means for transmitting the acquired image data to the cloud server, means for analyzing the image data and recognizing food items in the cloud server, means for proposing a menu based on the recognized food items, means for transmitting the proposed menu to the user terminal, means for analyzing the emotional state of the user using an emotion analysis engine, and means for adjusting the proposed menu based on the analyzed emotional state. This not only enables the user to effectively manage the food in the refrigerator, but also enables the optimal menu to be proposed according to the user's emotional state.

[0855] 1. "User terminal" means an electronic device operated by a user, including smartphones and tablets.

[0856] 2. "Image Data" means photographs of food items in a refrigerator, stored in a digital format.

[0857] 3. "Cloud server" means a server that stores and processes data via the Internet, including a group of servers that can be accessed remotely.

[0858] 4. "Food items" refer to specific foods or ingredients in the refrigerator, and are individual ingredients recognized through image analysis.

[0859] 5. A "menu" is a combination of suggested dishes, including a list of dishes from which the user can choose.

[0860] 6. "Emotion Analysis Engine" refers to algorithms and software that analyze a user's voice and touch actions to infer their emotional state.

[0861] 7. "Means for adjusting the proposed menu" refers to means including a function for changing and optimizing the content of the menu provided to the user based on the analyzed emotional state.

[0862] 8. "Stock ingredient list" refers to a list of foods and ingredients that are used on a daily basis and that is preset by the user.

[0863] 9. "Missing ingredients" refers to foods or ingredients that are on the list but not present in the refrigerator when comparing the food in the refrigerator with the list of regular ingredients.

[0864] 10. "Shopping List" refers to a list generated based on missing materials and used by the user when shopping.

[0865] 11. "External messaging services" are tools that allow users to communicate with other users, including messaging applications via the Internet such as LINE and email.

[0866] This invention is a system that helps users manage the food in their refrigerator and create menus more efficiently, and also combines an emotion engine that recognizes the user's emotions and optimizes the suggestions. This system can make optimal suggestions according to the user's emotional state.

[0867] The system consists of user devices such as smartphones and tablets, a cloud server, image analysis software, and an emotion analysis engine.

[0868] Acquiring and sending image data inside the refrigerator

[0869] The user device has a camera function for taking photos of the inside of the refrigerator. The user launches the app and takes a photo of the inside of the refrigerator. The device then sends the captured photo data to a cloud server. This communication is performed via an HTTP request, and the data is encrypted for security reasons.

[0870] Image analysis and food item recognition

[0871] The cloud server inputs the captured photo data into image analysis software (e.g., OpenCV, AWS Rekognition, etc.), which recognizes each food item in the refrigerator and assigns a label to each one, generating a list of food items.

[0872] Menu suggestions and adjustments

[0873] The server searches a recipe database (e.g., MySQL, MongoDB, etc.) based on the recognized food item list. The recipe database stores combinations of food items and corresponding cooking recipes. The list of found recipes is sent to the user's app, where the user can review and select the suggested recipes.

[0874] Checking stock of materials and detecting shortages

[0875] The server maintains the user's pre-defined list of ingredients. It compares the latest refrigerator food list with the list of ingredients and identifies any missing ingredients. The server generates a shopping list based on the missing ingredients and sends it to the user's app. The user can then check the shopping list from the app and share it with other family members using messaging services such as LINE.

[0876] Sentiment analysis engine integration and optimization

[0877] When a user uses an app, voice input and touch action data are analyzed by an emotion analysis engine (e.g., IBM Watson, Azure Cognitive Services, etc.). The device sends the voice data and touch action data to a cloud server, which then uses the emotion analysis engine to estimate the user's emotional state. For example, it determines whether the user is feeling stressed based on the tone of their voice or the speed of their touch action. Based on the estimated emotional state, the server adjusts the meal suggestions it provides. If the user is tired, it will prioritize easy-to-make meals. The contents of the shopping list are also adjusted based on the user's emotional state to reduce the burden on the user.

[0878] Specific examples

[0879] For example, a user can use a smartphone app to take a photo of the inside of their refrigerator and send it to a cloud server. The server then uses image analysis software to recognize items such as "tomato," "egg," "cheese," and "milk." From the list of recognition results, the server searches for recipes such as "tomato omelette," "cheese omelette," and "tomato salad," and sends them to the user's app.

[0880] When a user voice-inputs "I'm tired today," the device sends that data to a cloud server. The emotion analysis engine recognizes the user's level of fatigue, and the server suggests easy-to-prepare meals such as "easy salads." In this way, the system suggests optimal meals based on the user's emotional state.

[0881] Prompt Sentence Examples

[0882] "Tell me about your refrigerator food management system. It uses an emotion recognition engine to suggest the best meal plan based on the user's emotional state. Specifically, how does it work and what technology is used?"

[0883] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0884] Program processing flow

[0885] Step 1:

[0886] The user launches the smartphone app and takes a photo of the inside of the refrigerator.

[0887] Input: Video data inside the refrigerator

[0888] How it works: A user takes a picture of food in the refrigerator using the camera function on their smartphone.

[0889] Output: Captured photo data

[0890] Step 2:

[0891] The device sends the photographed photo data to a cloud server.

[0892] Input: Photographed photo data

[0893] How it works: Your device uploads photo data to a cloud server via an HTTP request. The data is sent using an encrypted communication protocol.

[0894] Output: Photo data uploaded to the cloud server

[0895] Step 3:

[0896] The server inputs the photographic data into image analysis software to recognize the food items.

[0897] Input: Uploaded photo data

[0898] How it works: The server uses image analysis software (e.g., OpenCV, AWS Rekognition) to label and identify each food item in a photo, also using machine learning models in the process.

[0899] Output: A list of recognized food items

[0900] Step 4:

[0901] The server sends the list of recognized food items to the user's app.

[0902] Input: A list of recognized food items

[0903] How it works: The server sends a list of food items to the user's app using an HTTP response.

[0904] Output: A list of food items displayed in the user's app

[0905] Step 5:

[0906] The server searches a menu recipe database based on the food item list and suggests an appropriate menu.

[0907] Input: Food item list

[0908] How it works: The server queries a recipe database (e.g. MySQL, MongoDB) to find recipes based on combinations of food items.

[0909] Output: A list of suggested recipes

[0910] Step 6:

[0911] The server sends the suggested recipe list to the user's app.

[0912] Input: Suggested recipe list

[0913] How it works: The server sends the recipe list to the user's app using an HTTP response.

[0914] Output: A list of recipes displayed in the user's app

[0915] Step 7:

[0916] The user selects a desired recipe from the suggested recipe list and displays its detailed information.

[0917] Input: Suggested recipe list

[0918] What it does: A user selects a recipe in the app and sees details (ingredients, cooking steps, cooking time, etc.).

[0919] Output: Details of the selected recipe displayed in the user's app

[0920] Step 8:

[0921] The server maintains a user-specified list of ingredients that are always on hand and compares it with a list of recognized food items to detect missing ingredients.

[0922] Input: List of ingredients and food items

[0923] How it works: The server compares both lists and identifies any missing ingredients.

[0924] Output: List of missing materials

[0925] Step 9:

[0926] The server generates a shopping list including missing ingredients and sends it to the user's app.

[0927] Input: Shortage material list

[0928] How it works: The server generates a shopping list and sends it to the user's app in an HTTP response.

[0929] Output: Shopping list displayed in the user's app

[0930] Step 10:

[0931] The user can view the shopping list from the app and share it with other family members using an external messaging service.

[0932] Input: Shopping list

[0933] How it works: A user uses the share button in the app to share their shopping list via LINE or email.

[0934] Output: Shopping list shared by family members

[0935] Step 11:

[0936] The server uses an emotion analysis engine to analyze the user's emotional state.

[0937] Input: Voice data, touch operation data

[0938] How it works: The server uses an emotion analysis engine (e.g., IBM Watson, Azure Cognitive Services) to analyze the tone of voice and the speed of touch movements to estimate the user's emotional state.

[0939] Output: Estimation of the user's emotional state

[0940] Step 12:

[0941] The server adjusts menu suggestions based on the estimated emotional state.

[0942] Input: Emotional state estimation result, suggested recipe list

[0943] How it works: The server changes the priority of meal suggestions based on the user's emotional state, and when stress is high, prioritizes easy-to-prepare meals.

[0944] Output: A menu list tailored based on emotional state

[0945] Through this series of processes, users can efficiently manage the food in their refrigerator and are presented with the optimal menu based on their emotional state.

[0946] (Application example 2)

[0947] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0948] Conventional food management systems only recognize food items in the refrigerator and suggest menu items, but do not provide optimal suggestions based on the user's emotional state. Furthermore, they lacked a mechanism for automatically adding data on food purchased in physical stores to the refrigerator list, placing a heavy burden on the user as they had to manually enter the information. Furthermore, the system sometimes found it inconvenient to provide a shopping list that was optimal for the user's situation or emotions.

[0949] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data of the inside of the refrigerator from the user terminal, means for transmitting the acquired image data to the cloud server, means for analyzing the image data and recognizing food items in the cloud server, means for proposing a menu based on the recognized food items, means for transmitting the proposed menu to the user terminal, means for scanning barcodes of food items from the user terminal in the physical store and adding the data to the refrigerator list, means for adjusting the menu based on the user's emotional state, means for analyzing the user's voice data and operation actions using an emotion engine and estimating the user's emotional state, and means for generating an optimal shopping list based on the emotional state. This enables suggestions based on the user's emotional state and efficient food management in the physical store.

[0950] A "user terminal" is a mobile information terminal such as a smartphone or tablet that is operated by a user.

[0951] "Image data" refers to image or photograph data of the state inside the refrigerator.

[0952] A "cloud server" is a server system for storing and processing data on a remote server on the Internet.

[0953] "Food items" refers to the types of ingredients and foods contained in the refrigerator.

[0954] A "menu" refers to a list or recipe of suggested dishes based on ingredients.

[0955] A "barcode" is an identification code consisting of a series of black lines and spaces used to represent product information.

[0956] "Emotional state" refers to the emotions and mental state that the user is currently experiencing.

[0957] An "emotion engine" is software or algorithms for analyzing and inferring a user's emotional state.

[0958] "Voice data" refers to the voice of a user recorded as digital data.

[0959] "Operation actions" refer to actions such as touching and swiping that a user performs on a device.

[0960] A "shopping list" is a list of food and daily necessities that need to be purchased.

[0961] This invention is a system that allows users to efficiently manage food in their refrigerators and create menus, and also combines it with an emotion engine that recognizes the user's emotions and optimizes the suggestions, thereby making optimal suggestions according to the user's emotional state.

[0962] System Overview

[0963] 1. Photographing and analyzing the inside of the refrigerator

[0964] The user uses a smartphone app to capture image data of the inside of the refrigerator using the smartphone's camera function.

[0965] The device sends the acquired image data to the cloud server using an HTTP request.

[0966] The server analyzes the image data on the cloud and recognizes food items. Image analysis is performed using OpenCV.

[0967] The server generates a list of recognized food items and transmits it to the user terminal.

[0968] 2. Menu suggestions

[0969] The server searches a menu recipe database based on the food item list, which stores various combinations of food items and corresponding cooking recipes.

[0970] The server sends a list of found recipes to the user's device, and the user can review and select suggested recipes on the app.

[0971] 3. Check your stock of ingredients and create a shopping list

[0972] The server maintains a list of ingredients on hand designated by the user on a cloud server and compares the food items recognized from the transmitted image data with the list of ingredients on hand.

[0973] The server identifies the missing ingredients and generates a shopping list that includes them.

[0974] The server sends the generated shopping list to the user terminal, where the user can share it with others using an external messaging service.

[0975] 4. Emotion engine integration and optimization

[0976] When a user expresses their emotional state through voice input while operating the app, the voice data and operation data are sent to the emotion engine, which uses the Hugging Face Transformers library.

[0977] The server uses an emotion engine to infer the user's emotional state from the data, for example, by analyzing the tone of voice or the speed of haptic movements.

[0978] The server adjusts the meal suggestions and shopping list it provides based on the user's emotional state. For example, if the user is feeling stressed, it will prioritize easy-to-prepare meals.

[0979] Physical store operations

[0980] Furthermore, this system is also intended for use in physical stores. For example, when a user purchases food at a supermarket, they can scan the barcode on the product with a smartphone app. The barcode data is sent to the server and immediately added to the refrigerator list. This allows the user to automatically update their shopping list and manage it efficiently.

[0981] Specific examples

[0982] Food Recognition in the Refrigerator:

[0983] Users take a photo of their refrigerator with their smartphone and upload it to the app, where the food recognition system identifies "tomatoes," "eggs," "cheese," and "milk," and creates a list of these foods.

[0984] Menu suggestions based on user sentiment:

[0985] When a user voice-inputs into the app, "I'm busy and tired today," the emotion engine recognizes the "tired" state and suggests an easy-to-make "tomato salad."

[0986] Examples of prompts:

[0987] Take a picture of the inside of your refrigerator and identify the following food items: tomatoes, eggs, cheese, and milk. Based on this information, suggest a simple meal plan. Also include suggestions for when the user is tired.

[0988] In this way, appropriate suggestions based on the user's emotional state and efficient food management in physical stores can be realized.

[0989] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0990] Step 1:

[0991] A user uses a smartphone app to capture image data of the inside of a refrigerator. The user launches the app and takes a photo of the inside of the refrigerator with the smartphone camera. Once the photo is captured, the app prepares the image data.

[0992] Input: Photo data of the inside of a refrigerator taken with a smartphone camera.

[0993] Output: The image data is saved in the app on your smartphone.

[0994] Step 2:

[0995] The device sends the acquired image data to the cloud server, and the smartphone app uses an HTTP request to upload this image data to the cloud server.

[0996] Input: The acquired image data.

[0997] Output: The image data is transferred to the cloud server.

[0998] Step 3:

[0999] The server analyzes the image data on the cloud and recognizes food items. OpenCV is used to automatically identify food items from the image data and assign labels to each ingredient.

[1000] Input: Image data sent to the cloud server.

[1001] Output: A list of recognized food items.

[1002] Step 4:

[1003] The server searches a menu recipe database based on the recognized food items, uses the food list to search the database for related cooking recipes, and generates a list of found recipes.

[1004] Input: A list of recognized food items.

[1005] Output: A list of recipes.

[1006] Step 5:

[1007] The server sends a list of found recipes to the user's device. The cloud server prepares the recipe list and sends it to the smartphone app using an HTTP request.

[1008] Input: A list of recipes.

[1009] Output: The recipe list is displayed on the smartphone app.

[1010] Step 6:

[1011] A user scans the barcode of a food item at a physical store. The user activates the barcode scanner on their smartphone app and scans the barcode of the food item they want to purchase. The barcode data is sent by the app to a cloud server.

[1012] Input: Scanned barcode data.

[1013] Output: The barcode data is sent to the cloud server.

[1014] Step 7:

[1015] The server receives the barcode data and updates the refrigerator list accordingly, so that the newly added food item is reflected in the refrigerator list.

[1016] Input: Barcode data.

[1017] Output: Updated refrigerator list.

[1018] Step 8:

[1019] The user inputs their emotional state using the app. The user expresses their emotional state, such as "I'm tired today," through voice input or text input. The emotional data is sent to a cloud server.

[1020] Input: User's emotional state (audio or text data).

[1021] Output: Emotion data is sent to the cloud server.

[1022] Step 9:

[1023] The server uses an emotion engine to analyze the emotion data and estimate the user's emotional state. It uses the Hugging Face Transformers library to analyze voice tones and operational movements.

[1024] Input: Emotion data.

[1025] Output: Estimated emotional state.

[1026] Step 10:

[1027] The server then adjusts the menu suggestions it provides based on the user's estimated emotional state. For example, if the user is feeling stressed, it will prioritize easy-to-make recipes.

[1028] Input: Emotional state.

[1029] Output: Adjusted menu suggestions.

[1030] Step 11:

[1031] The server compares the list of ingredients with the list of recognized food items to identify missing ingredients, creates a shopping list based on the missing ingredients, and sends it to the user's device.

[1032] Input: Stock ingredient list and recognized food item list.

[1033] Output: A shopping list including missing ingredients is generated and sent to the user's device.

[1034] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1035] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1036] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1037] [Third embodiment]

[1038] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1039] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1040] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1041] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1042] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1043] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1044] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1045] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1046] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1047] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1048] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1049] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1050] This invention is a system for streamlining food management and menu suggestions. It automatically recognizes ingredients in the refrigerator and suggests menus based on them, so that users don't have to waste time creating daily menus and managing shopping lists. It also provides a function to detect shortages of regular ingredients and create and share shopping lists.

[1051] System Overview

[1052] 1. Photographing and analyzing the inside of the refrigerator

[1053] A user uses a smartphone app to take a photo of the inside of the refrigerator.

[1054] The device sends the photographed photo data to a cloud server.

[1055] The server inputs the photo data into image analysis software to recognize the food items in the refrigerator. Specifically, the server uses image analysis algorithms (e.g., deep learning techniques) to identify each ingredient in the photo and assign it a label.

[1056] The server generates a list of recognized food items and sends it to the user's app, where the user can view the list of food items in the refrigerator.

[1057] 2. Menu suggestions

[1058] Based on the analysis results, the server searches a database for menu recipes using the food items in the refrigerator. The recipe database stores various combinations of food items and corresponding cooking recipes.

[1059] The server sends the list of found recipes to the user's app.

[1060] Users can select their favorite recipe from the suggested recipes, and the details of the selected recipe (ingredients, cooking method, required time, etc.) will be displayed.

[1061] 3. Check your stock of ingredients and create a shopping list

[1062] The cloud server stores a list of ingredients that the user has pre-set (for example, "onions, carrots, soy sauce").

[1063] The server compares the current food item list with the stock ingredient list to identify any missing food items.

[1064] The server generates a shopping list based on the missing food items, which can include any missing staple ingredients as well as any additional ingredients needed for the user's selected meal.

[1065] Users can check their shopping list in the app and share it using external messaging services (e.g., LINE), allowing them to ask other family members to do the shopping.

[1066] Specific examples

[1067] 1. Photographing and analyzing the inside of the refrigerator

[1068] The user launches the smartphone app and presses the "take a photo of the inside of the refrigerator" button.

[1069] The device uses its camera function to take pictures of the inside of the refrigerator and sends the image data to a cloud server.

[1070] The server analyzes the image and recognizes that there are tomatoes, eggs, cheese, and milk in the refrigerator.

[1071] The server generates a list of food items and sends it to the user's app.

[1072] 2. Menu suggestions

[1073] The server searches the database based on the list of ingredients and finds recipes such as "tomato omelet," "cheese omelet," and "tomato salad."

[1074] The server sends these recipes to the user's app.

[1075] When a user selects "Tomato Omelette" from the app, the recipe details are displayed.

[1076] 3. Check your stock of ingredients and create a shopping list

[1077] The server maintains a list of ingredients (e.g., onions, carrots, soy sauce).

[1078] The server compares the latest list of food items in the refrigerator with the list of regularly-available ingredients and detects that there are shortages of "onions" and "carrots."

[1079] The server generates a shopping list for "onions, carrots" and sends it to the user's app.

[1080] The user checks the shopping list on the app and sends it to her husband via LINE. On the way home, the husband buys onions and carrots at the supermarket.

[1081] As described above, the present invention reduces the burden on the user and provides a method for efficiently managing ingredients and creating menus at home.

[1082] The processing flow will be explained below.

[1083] Step 1:

[1084] The user launches the smartphone app and takes a photo of the inside of the refrigerator. The user then uses the camera function to capture an overall image of the inside of the refrigerator, and presses the "Send" button when the photo is complete.

[1085] Step 2:

[1086] The device sends the captured photo data to the cloud server. Specifically, the device sends an HTTP request to upload the photo data to the server.

[1087] Step 3:

[1088] The server receives the HTTP request and stores the transmitted photo data, ensuring that the server has the necessary photo data for future processing.

[1089] Step 4:

[1090] The server inputs the stored photo data into image analysis software to recognize the food items in the refrigerator. Specifically, it uses image recognition algorithms such as deep learning to identify each ingredient in the photo and assign it an individual label.

[1091] Step 5:

[1092] The server generates a list of food items based on the analysis results and sends the list to the user's app, where the user can view the list of food items in the refrigerator.

[1093] Step 6:

[1094] The server searches the menu recipe database based on the food item list. The recipe database stores various combinations of food items and corresponding cooking recipes. The server identifies recipes for dishes using the ingredients in the refrigerator.

[1095] Step 7:

[1096] The server sends a list of found recipes to the user's app, where the user can select the recipes they like.

[1097] Step 8:

[1098] The user uses the app to review the suggested recipes and selects a recipe, such as "Tomato Omelette." Details of the selected recipe (e.g., ingredients, cooking instructions, cooking time, etc.) are displayed in the app.

[1099] Step 9:

[1100] The server retrieves the user's predefined list of ingredients (e.g., "onions, carrots, soy sauce") from the database. The server compares this list with the latest list of food in the refrigerator.

[1101] Step 10:

[1102] The server compares the list of ingredients in stock with the list of food in the refrigerator and identifies any ingredients that are missing. For example, it determines that "onions" and "carrots" are missing.

[1103] Step 11:

[1104] The server generates a shopping list based on the missing ingredients, which can include the missing regular ingredients as well as any additional ingredients needed for the recipe selected by the user.

[1105] Step 12:

[1106] The server sends the generated shopping list to the user's app, where the user can view the shopping list.

[1107] Step 13:

[1108] The user presses the "Share Shopping List" button in the app and selects an external messaging service such as LINE. The device then sends the shopping list as a LINE message to share with other family members.

[1109] The above are the specific processing steps of the system, which allow users to proactively manage food and create menus.

[1110] Example 1

[1111] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1112] Conventional food management and menu suggestion systems have the problem that users have to manually check the ingredients in the refrigerator and search for recipes, which is time-consuming. Furthermore, to determine if a necessary ingredient is missing, the user must check the refrigerator again, making efficient food management difficult. This invention aims to reduce the user's burden and provide efficient food management and menu suggestions by automatically recognizing ingredients in the refrigerator using image analysis technology and suggesting menus based on that information.

[1113] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1114] In this invention, the server includes means for acquiring image data from a user terminal, means for transmitting the acquired image data via a network, means for analyzing the image data and recognizing object items in the server, means for providing information based on the recognized object items, and means for transmitting the provided information to the user terminal. This reduces the user's workload and enables efficient food management and menu suggestions. The server also includes means for storing a reference list specified by the user, comparing the object items recognized from the transmitted image data with the reference list to detect missing items, generating a list including the missing items, and means for transmitting the generated list to the user terminal, allowing the user to easily identify missing ingredients and use it as a shopping list. Furthermore, the server also includes means for transmitting the generated list from the user terminal using an external communication service, allowing the user to share the shopping list with family and other members and shop collaboratively.

[1115] "User terminal" refers to a device operated by a user, including a smartphone, tablet, or PC.

[1116] "Image data" refers to photographs or scanned images of the inside of the refrigerator, including visual information stored in digital form.

[1117] "Network" refers to a communications system that enables data to be communicated between devices, including the Internet and local area networks.

[1118] A "server" refers to a central computer system that processes data and provides information in response to requests from clients (user terminals).

[1119] "Object item" refers to an individual entity (e.g., food or object) recognized from image data.

[1120] "Information" is content provided based on the object item, including recipes, ingredient lists, etc.

[1121] The "standard list" refers to a list of items that should be kept as standard, set by the user, and includes, for example, a list of regular materials.

[1122] "External communication services" are services used to send and receive data, including messaging applications and email services.

[1123] A "list" is a list of specific items, including a list of missing items and a shopping list.

[1124] This invention is a system for streamlining food management and menu suggestions. It combines a user terminal, a server, an image analysis algorithm, and an external communication service to significantly reduce the user's workload.

[1125] System Configuration

[1126] The system mainly consists of the following hardware and software elements:

[1127] User device: Use a smartphone, tablet, or computer.

[1128] Server: A cloud server is used to process data from multiple user devices.

[1129] Image analysis algorithm: Uses deep learning models to perform object recognition.

[1130] External communication services: Using messaging applications (e.g., messaging apps, email).

[1131] Operating principle

[1132] The system's operation can be broadly divided into the following steps:

[1133] Taking photos of the inside of the refrigerator and analyzing them

[1134] A user uses a smartphone app to take a photo of the inside of their refrigerator. The device then sends the captured photo data to a cloud server. The server then uses an image analysis algorithm (deep learning technology) to recognize food items in the photo and assigns labels to each. Specifically, it uses an object recognition algorithm to extract food items such as "tomato," "egg," and "cheese." The server then generates a list of the recognized food items and sends it to the user's app.

[1135] Menu suggestions

[1136] Based on the analysis results, the server searches a database for menu recipes using the food items in the refrigerator. The recipe database stores various combinations of food items and corresponding cooking recipes. For example, a database search can find recipes such as "tomato omelette," "cheese omelette," and "tomato salad." The server sends a list of these recipes to the user's app, and the user can select their preferred recipe from the suggested recipes. Details of the selected recipe (ingredients, cooking method, required time, etc.) are also displayed in the app.

[1137] Check your stock of ingredients and create a shopping list

[1138] The server maintains a standard list (e.g., onions, carrots, soy sauce, etc.) preset by the user. The server compares the latest list of food items in the refrigerator with the standard list to identify any missing food items. Specifically, if the server detects that "onions" and "carrots" are missing, it generates a shopping list that includes these items. The shopping list can include not only the missing basic ingredients but also additional ingredients required for the meal selected by the user. The user can check the shopping list in the app and share it with other family members using an external communication service (e.g., a messaging app).

[1139] Specific examples

[1140] 1. Photographing and analyzing the inside of the refrigerator

[1141] The user launches the app and presses the "Take a photo of the inside of the refrigerator" button.

[1142] The device uses its camera function to take pictures of the inside of the refrigerator and sends the image data to the server.

[1143] The server analyzes the image and recognizes that there are tomatoes, eggs, cheese, and milk in the refrigerator.

[1144] The server generates a list of food items and sends it to the user's app.

[1145] 2. Menu suggestions

[1146] The server searches the database based on the list of ingredients and finds recipes such as "tomato omelet," "cheese omelet," and "tomato salad."

[1147] The server sends these recipes to the user's app.

[1148] When a user selects "Tomato Omelette" from the app, the recipe details are displayed.

[1149] 3. Check your stock of ingredients and create a shopping list

[1150] The server has a pre-stored list of criteria (e.g., onions, carrots, soy sauce).

[1151] The server compares the current food list with the reference list and identifies that "onions" and "carrots" are in short supply.

[1152] The server generates a shopping list that includes "onions, carrots" and sends it to the user's app.

[1153] A user sends a shopping list to their family via a messaging app, and the family members then go to the supermarket to purchase the items.

[1154] Prompt Sentence Examples

[1155] "Describe a system that recognizes the foods in a refrigerator, generates a list, and notifies the user of that list."

[1156] "Please explain the process of a system that suggests recipes based on a list of foods in the refrigerator."

[1157] "Describe a system that detects shortages of staple ingredients and automatically generates a shopping list."

[1158] In this way, it is possible to significantly improve the efficiency of the user's food ingredient management and menu proposals. This system reduces the burden on the user and provides a method for efficiently managing food at home and creating menus.

[1159] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1160] Step 1:

[1161] The user launches the smartphone app and takes a photo of the inside of the refrigerator.

[1162] Specific operation: The user presses the "Take a photo inside the refrigerator" button and takes a photo of the inside of the refrigerator using the smartphone camera.

[1163] Input: Visual information inside the refrigerator

[1164] Output: Image data

[1165] Step 2:

[1166] The device sends the photographed photo data to a cloud server.

[1167] Specific behavior: Uploads an image to the server using the HTTPS protocol.

[1168] Input: Image data

[1169] Output: Image data stored on a cloud server

[1170] Step 3:

[1171] The server uses image analysis algorithms (e.g., deep learning techniques) to recognize food items in the refrigerator.

[1172] Input: Image data stored on a cloud server

[1173] Processing: Object recognition with deep learning models

[1174] Output: A list of recognized food items

[1175] What it does: The server runs an image analysis model to identify food items such as "tomato," "egg," and "cheese."

[1176] Step 4:

[1177] The server sends the recognized food item list to the user terminal.

[1178] Specific operation: The server transfers the food item list generated to the user's device using the REST API.

[1179] Input: A list of recognized food items

[1180] Output: A list of food items in the refrigerator displayed on the user's device

[1181] Step 5:

[1182] The server searches a database based on the list of food items to find available recipes.

[1183] What it does: Uses a database query to extract the corresponding recipes.

[1184] Input: Food item list

[1185] Processing: Database queries and recipe extraction

[1186] Output: List of available recipes

[1187] Step 6:

[1188] The server sends the recipe list it finds to the user's terminal.

[1189] Specific operation: Send the recipe list to the user's terminal using the REST API.

[1190] Input: Available recipe list

[1191] Output: Recipe list displayed on the user's device

[1192] Step 7:

[1193] The user selects one of the suggested recipes and displays the details.

[1194] Specific behavior: Displays details (ingredients, cooking instructions, cooking time, etc.) of a recipe selected by the user from the app.

[1195] Input: Recipe ID selected by the user

[1196] Output: Detailed information about the selected recipe

[1197] Step 8:

[1198] The server maintains a list of criteria set by the user and compares it with the list of food items to detect missing items.

[1199] Specific Behavior: Compare the list of staple ingredients with the list of food items and identify any missing items.

[1200] Input: Criteria list and current food list

[1201] Processing: List comparison and missing item detection

[1202] Output: Missing items list

[1203] Step 9:

[1204] The server generates a shopping list including the missing items and transmits it to the user terminal.

[1205] Specific operation: Generate a shopping list and send it to the user's device using the REST API.

[1206] Input: Missing Item List

[1207] Output: Shopping list displayed on the user's device

[1208] Step 10:

[1209] The user shares the generated shopping list through an external communication service.

[1210] What it does: A user checks a shopping list within the app and sends it to family and other members using an external messaging service.

[1211] Input: Shopping list

[1212] Output: Shared shopping list

[1213] Through these steps, the system enables users to manage the ingredients in their refrigerator and efficiently suggest menus.

[1214] (Application example 1)

[1215] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1216] In modern households, managing food in the refrigerator and creating daily menus are tasks that require a great deal of time and effort. Creating appropriate shopping lists and replenishing necessary ingredients are also time-consuming. Furthermore, with the increasing demand for eating out and food delivery, there is a need for ways to enjoy meals efficiently without wasting ingredients at home. The present invention aims to solve these problems by providing a system that efficiently handles everything from food management to menu suggestions, shopping list creation, and delivery arrangements.

[1217] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1218] In this invention, the server includes means for acquiring image data of the contents of the refrigerator from the user terminal, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and recognizing food items in the cloud server, means for proposing menus based on the recognized food items, means for transmitting the proposed menus to the user terminal, and means for coordinating with an external service that provides the proposed menus and arranging delivery. This allows the user to reduce food waste in the refrigerator, receive optimal menu suggestions without hassle, and smoothly arrange for the delivery of ingredients that are in short supply.

[1219] A "user terminal" is an electronic device operated by a user, and has the function of acquiring image data of the inside of a refrigerator.

[1220] "Image data" is image information showing the state of food in the refrigerator, and is sent from the user terminal to the cloud server.

[1221] A "cloud server" is a remote computer connected to a network that analyzes image data and processes various types of data.

[1222] "Food item" refers to ingredients or types of ingredients recognized from image data.

[1223] A "menu" is a list of cooking suggestions created based on recognized food items.

[1224] "Suggested menu" refers to a menu list generated by the cloud server, from which the user can select.

[1225] "External services" refer to companies or systems that work in conjunction with the cloud server to deliver suggested menus and missing ingredients.

[1226] "Delivery" refers to a service that delivers meals based on a suggested menu and any missing ingredients to the user's home.

[1227] The "stock ingredient list" is a list of ingredients that the user sets in advance and that the user wants to keep in the refrigerator at all times.

[1228] A "missing ingredient" is an ingredient that is not present in the refrigerator when comparing a recognized food item with the list of ingredients.

[1229] A "shopping list" is a list of ingredients that are in short supply and ingredients that the user wishes to purchase in addition.

[1230] "External messaging service" means a communication means used by a user to send a generated shopping list to another person.

[1231] A "generative AI model" is an artificial intelligence program that has learned algorithms to perform specific tasks based on large amounts of data.

[1232] "Prompt sentence" refers to an instruction sentence that a generative AI model uses to start a task.

[1233] This invention is a system that combines user terminals, cloud servers, delivery partners, and external messaging services to provide a comprehensive range of services, from managing food in the refrigerator to suggesting menus and delivering missing ingredients.

[1234] System Overview

[1235] 1. Acquire and send image data from the user's device:

[1236] The user terminal has a camera function for taking pictures of the inside of the refrigerator.

[1237] When a user uses a smartphone app to take a photo of the inside of the refrigerator, the device sends the image data to a cloud server.

[1238] 2. Ingredient recognition and menu suggestions on the cloud server:

[1239] The cloud server analyzes the received image data and recognizes the food items in the refrigerator using a deep learning model.

[1240] The server lists food items and suggests meals based on the recognized ingredients, selected from recipes stored in a database.

[1241] The server sends the proposed menu to the user terminal, allowing the user to select a menu.

[1242] 3. Detecting missing ingredients and generating a shopping list:

[1243] The server maintains a list of ingredients that are pre-defined by the user.

[1244] Compare recognized food items with a list of staple ingredients to identify missing ingredients.

[1245] A shopping list is generated based on the ingredients that are missing, and delivery of the missing ingredients is arranged through a delivery partner if necessary.

[1246] The generated shopping list is sent to the user's terminal and can also be shared with others via external messaging services.

[1247] Hardware and software used

[1248] Hardware: Smartphone (with camera), cloud server

[1249] Software: OpenCV (for image processing), requests (for HTTP requests), deep learning models (e.g. TensorFlow, PyTorch)

[1250] Specific examples

[1251] As an example, consider a scenario where a user takes a photo of the inside of a refrigerator and uploads it to a cloud server.

[1252] The server performs image analysis and recognizes that there are tomatoes, eggs, and milk in the refrigerator.

[1253] The server proposes menu items such as "tomato omelet" and "milk pudding" based on these ingredients and sends them to the user's terminal.

[1254] The user selects "tomato omelette," the server detects the missing ingredient (e.g., "onion"), and arranges for the onion to be delivered through a delivery partner.

[1255] Example prompts from a generative AI model

[1256] An example prompt is, "Based on the ingredients in your refrigerator, suggest healthy meals you might like. If necessary, order additional ingredients through a delivery service."

[1257] These processes allow the user to efficiently manage ingredients in the refrigerator, enjoy suggested meals, and smoothly arrange for delivery of ingredients that are in short supply.

[1258] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1259] Step 1:

[1260] The user launches the smartphone app and presses the "Take a photo of the refrigerator" button. The smartphone camera starts up, and the user takes a photo of the food in the refrigerator. The photo is saved as image data on the device.

[1261] Step 2:

[1262] The device sends the captured image data to the cloud server using an HTTP request, and the data sent is in the form of an image file.

[1263] Step 3:

[1264] The server inputs the received image data into an image analysis program, which uses a deep learning model to recognize food ingredients. The input is the image data, and the output is a list of recognized food items.

[1265] Step 4:

[1266] The server runs a menu suggestion algorithm based on the list of food items. The algorithm references recipe information stored in the database and suggests menus based on the input food items. The output is a list of suggested recipes.

[1267] Step 5:

[1268] The server sends a list of suggested menus to the user's terminal. The user checks the suggested menus on the terminal and selects the menu of their choice. The selected menu becomes the input for the next step.

[1269] Step 6:

[1270] The server detects missing ingredients based on the menu selected by the user and the recognized food item list. The input is the food item list and the stock ingredient list, and the output is the missing ingredients.

[1271] Step 7:

[1272] The server generates a shopping list based on the missing ingredients. The input is the list of missing ingredients, and the output is the shopping list.

[1273] Step 8:

[1274] The server sends the generated shopping list to the user's device, which can then share it with others via an external messaging service. To share, a specific prompt is generated using the generative AI model and sent as a message.

[1275] Step 9:

[1276] The user checks the shopping list and orders any missing ingredients through an external delivery partner. The server then requests the linked external service to arrange delivery.

[1277] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1278] This invention is a system that allows users to efficiently manage food in their refrigerators and create menus, and also combines it with an emotion engine that recognizes the user's emotions and optimizes the suggestions, thereby making optimal suggestions according to the user's emotional state.

[1279] System Overview

[1280] 1. Photographing and analyzing the inside of the refrigerator

[1281] A user takes a photo of the inside of the refrigerator using a smartphone app.

[1282] The device sends the captured photo data to the cloud server, uploading the photo data via an HTTP request.

[1283] The server inputs the received photo data into image analysis software to recognize the food items in the refrigerator. Image recognition algorithms are used to label and identify each ingredient in the photo.

[1284] The server generates a list of the food items obtained and sends it to the user's app, where the user can view the list of food items in the refrigerator.

[1285] 2. Menu suggestions

[1286] The server searches a menu recipe database based on the food item list. The recipe database stores various combinations of food items and corresponding cooking recipes.

[1287] The server sends a list of found recipes to the user's app, where the user can review and select the suggested recipes.

[1288] The app displays detailed information about the recipe the user selects (e.g., ingredients, cooking instructions, cooking time, etc.).

[1289] 3. Check your stock of ingredients and create a shopping list

[1290] The server maintains the user's pre-defined list of ingredients and compares it with the current list of food in the refrigerator.

[1291] The server compares the stock of ingredients to identify any missing ingredients.

[1292] The server generates a shopping list based on the missing ingredients and sends the list to the user's app.

[1293] Users can check their shopping list from the app and share it with other family members using external messaging services such as LINE.

[1294] 4. Emotion engine integration and optimization

[1295] When a user uses the app, they input voice and make touch gestures, and this data is collected and analyzed by the emotion engine.

[1296] The device sends voice data and touch operation data to a cloud server.

[1297] The server uses an emotion engine to estimate the user's emotional state from the collected data, for example, determining whether the user is feeling stressed based on the tone of their voice or the speed of their touch gestures.

[1298] The server then adjusts the menu suggestions it provides based on the user's estimated emotional state. For example, if the user is feeling stressed, it will prioritize easy-to-prepare menu suggestions.

[1299] The server also adjusts the contents of the shopping list based on the emotional state, reducing the burden on the user.

[1300] Specific examples

[1301] 1. Photographing and analyzing the inside of the refrigerator

[1302] The user launches the smartphone app and takes a photo of the inside of the refrigerator.

[1303] The device sends the photo data to the cloud server.

[1304] The server analyzes the photo and recognizes "tomato," "egg," "cheese," and "milk."

[1305] The server generates a list of food items and sends it to the user's app.

[1306] 2. Menu suggestions

[1307] Based on the list of ingredients, the server finds recipes such as "tomato omelet," "cheese omelet," and "tomato salad" from the database.

[1308] The server sends these recipes to the user's app.

[1309] If the user selects "Tomato Omelette," detailed information is displayed.

[1310] 3. Check your stock of ingredients and create a shopping list

[1311] The server maintains and analyzes the user's stockpile of ingredients.

[1312] The server compares the list of food items in the refrigerator with the list of regularly available ingredients and detects that "onions" and "carrots" are missing.

[1313] The server generates a shopping list that includes "onions" and "carrots" and sends it to the user's app.

[1314] Users can check their shopping list, share it using LINE, and ask family members to do the shopping.

[1315] 4. Emotion engine integration and optimization

[1316] While using the app, the user utters the following using voice input: "I'm tired today."

[1317] The device sends the voice data to the cloud server.

[1318] The server uses an emotion engine to infer from the voice data that the user is tired.

[1319] The server will prioritize suggestions for dishes that are easy to prepare (e.g., "easy salad").

[1320] The server also adjusts the shopping list to include more ingredients that are easy to prepare.

[1321] This invention reduces the burden on users and supports food management and cooking tasks more efficiently by proposing optimal menus and shopping lists according to the user's emotional state.

[1322] The processing flow will be explained below.

[1323] Step 1:

[1324] The user starts the smartphone app and takes a photo of the inside of the refrigerator. The user then uses the app's camera function to take a photo of the entire refrigerator.

[1325] Step 2:

[1326] The device sends the captured photo data to the cloud server. Specifically, it creates an HTTP request including the photo data and uploads it to the cloud server.

[1327] Step 3:

[1328] The server receives the HTTP request and stores the sent photo data in cloud storage.

[1329] Step 4:

[1330] The server inputs the stored photo data into image analysis software to recognize the food items in the refrigerator. Image recognition algorithms (such as deep learning models) are used to identify the ingredients in the photo.

[1331] Step 5:

[1332] The server generates a list of food items based on the analysis results and sends the list to the user's app, where the user can view the list.

[1333] Step 6:

[1334] The server searches a menu recipe database based on the food item list. The database contains a variety of food item combinations and corresponding cooking recipes.

[1335] Step 7:

[1336] The server sends the list of recipes found to the user's app, where the user can view multiple menu suggestions and select the one they like best.

[1337] Step 8:

[1338] The user selects "Tomato Omelette" from the suggested recipes. The app displays detailed information about the selected recipe (e.g., ingredients, cooking instructions, cooking time, etc.).

[1339] Step 9:

[1340] The server retrieves the user's pre-defined ingredient list from the database, which includes ingredients and seasonings that the user frequently uses.

[1341] Step 10:

[1342] The server compares the latest list of food items in the refrigerator with the list of ingredients to identify any missing ingredients. For example, if "onions" and "carrots" are not on the list, they are recognized as missing ingredients.

[1343] Step 11:

[1344] The server generates a shopping list based on the missing ingredients, which may include the missing standard ingredients as well as additional ingredients needed for the selected meal.

[1345] Step 12:

[1346] The server sends the generated shopping list to the user's app, where the user can view the list.

[1347] Step 13:

[1348] The user presses the "Share Shopping List" button in the app and selects an external messaging service such as LINE. The device then sends the shopping list as a LINE message to share with other family members.

[1349] Step 14:

[1350] The device collects the user's voice input and touch actions and sends this data to a cloud server. For example, if a user says, "I'm tired today," the device collects this voice data.

[1351] Step 15:

[1352] The server uses an emotion engine to estimate the user's emotional state from the collected data, such as the tone of voice and the speed of touch movements, to determine whether the user is feeling stressed.

[1353] Step 16:

[1354] The server then tailors meal suggestions based on the user's estimated emotional state, for example, suggesting easy, hassle-free meals if the user is feeling stressed.

[1355] Step 17:

[1356] The server also adjusts the shopping list based on the user's emotional state: if the user is tired, for example, the list will prioritize ingredients that require less preparation.

[1357] Through the above processing steps, the system proposes optimal menus and shopping lists according to the user's emotional state. This system improves the efficiency of food management and cooking at home, reducing the burden on the user.

[1358] Example 2

[1359] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1360] For many users, efficiently managing food in the refrigerator and creating menus is a time-consuming and labor-intensive task. Furthermore, users often feel stressed and fatigued because the system does not provide optimal suggestions based on their emotional state. The present invention aims to solve these problems and enable users to manage food and create menus more effectively.

[1361] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring image data of the inside of the refrigerator from the user terminal, means for transmitting the acquired image data to the cloud server, means for analyzing the image data and recognizing food items in the cloud server, means for proposing a menu based on the recognized food items, means for transmitting the proposed menu to the user terminal, means for analyzing the emotional state of the user using an emotion analysis engine, and means for adjusting the proposed menu based on the analyzed emotional state. This not only enables the user to effectively manage the food in the refrigerator, but also enables the optimal menu to be proposed according to the user's emotional state.

[1362] 1. "User terminal" means an electronic device operated by a user, including smartphones and tablets.

[1363] 2. "Image Data" means photographs of food items in a refrigerator, stored in a digital format.

[1364] 3. "Cloud server" means a server that stores and processes data via the Internet, including a group of servers that can be accessed remotely.

[1365] 4. "Food items" refer to specific foods or ingredients in the refrigerator, and are individual ingredients recognized through image analysis.

[1366] 5. A "menu" is a combination of suggested dishes, including a list of dishes from which the user can choose.

[1367] 6. "Emotion Analysis Engine" refers to algorithms and software that analyze a user's voice and touch actions to infer their emotional state.

[1368] 7. "Means for adjusting the proposed menu" refers to means including a function for changing and optimizing the content of the menu provided to the user based on the analyzed emotional state.

[1369] 8. "Stock ingredient list" refers to a list of foods and ingredients that are used on a daily basis and that is preset by the user.

[1370] 9. "Missing ingredients" refers to foods or ingredients that are on the list but not present in the refrigerator when comparing the food in the refrigerator with the list of regular ingredients.

[1371] 10. "Shopping List" refers to a list generated based on missing materials and used by the user when shopping.

[1372] 11. "External messaging services" are tools that allow users to communicate with other users, including messaging applications via the Internet such as LINE and email.

[1373] This invention is a system that helps users manage the food in their refrigerator and create menus more efficiently, and also combines an emotion engine that recognizes the user's emotions and optimizes the suggestions. This system can make optimal suggestions according to the user's emotional state.

[1374] The system consists of user devices such as smartphones and tablets, a cloud server, image analysis software, and an emotion analysis engine.

[1375] Acquiring and sending image data inside the refrigerator

[1376] The user device has a camera function for taking photos of the inside of the refrigerator. The user launches the app and takes a photo of the inside of the refrigerator. The device then sends the captured photo data to a cloud server. This communication is performed via an HTTP request, and the data is encrypted for security reasons.

[1377] Image analysis and food item recognition

[1378] The cloud server inputs the captured photo data into image analysis software (e.g., OpenCV, AWS Rekognition, etc.), which recognizes each food item in the refrigerator and assigns a label to each one, generating a list of food items.

[1379] Menu suggestions and adjustments

[1380] The server searches a recipe database (e.g., MySQL, MongoDB, etc.) based on the recognized food item list. The recipe database stores combinations of food items and corresponding cooking recipes. The list of found recipes is sent to the user's app, where the user can review and select the suggested recipes.

[1381] Checking stock of materials and detecting shortages

[1382] The server maintains the user's pre-defined list of ingredients. It compares the latest refrigerator food list with the list of ingredients and identifies any missing ingredients. The server generates a shopping list based on the missing ingredients and sends it to the user's app. The user can then check the shopping list from the app and share it with other family members using messaging services such as LINE.

[1383] Sentiment analysis engine integration and optimization

[1384] When a user uses an app, voice input and touch action data are analyzed by an emotion analysis engine (e.g., IBM Watson, Azure Cognitive Services, etc.). The device sends the voice data and touch action data to a cloud server, which then uses the emotion analysis engine to estimate the user's emotional state. For example, it determines whether the user is feeling stressed based on the tone of their voice or the speed of their touch action. Based on the estimated emotional state, the server adjusts the meal suggestions it provides. If the user is tired, it will prioritize easy-to-make meals. The contents of the shopping list are also adjusted based on the user's emotional state to reduce the burden on the user.

[1385] Specific examples

[1386] For example, a user can use a smartphone app to take a photo of the inside of their refrigerator and send it to a cloud server. The server then uses image analysis software to recognize items such as "tomato," "egg," "cheese," and "milk." From the list of recognition results, the server searches for recipes such as "tomato omelette," "cheese omelette," and "tomato salad," and sends them to the user's app.

[1387] When a user voice-inputs "I'm tired today," the device sends that data to a cloud server. The emotion analysis engine recognizes the user's level of fatigue, and the server suggests easy-to-prepare meals such as "easy salads." In this way, the system suggests optimal meals based on the user's emotional state.

[1388] Prompt Sentence Examples

[1389] "Tell me about your refrigerator food management system. It uses an emotion recognition engine to suggest the best meal plan based on the user's emotional state. Specifically, how does it work and what technology is used?"

[1390] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1391] Program processing flow

[1392] Step 1:

[1393] The user launches the smartphone app and takes a photo of the inside of the refrigerator.

[1394] Input: Video data inside the refrigerator

[1395] How it works: A user takes a picture of food in the refrigerator using the camera function on their smartphone.

[1396] Output: Captured photo data

[1397] Step 2:

[1398] The device sends the photographed photo data to a cloud server.

[1399] Input: Photographed photo data

[1400] How it works: Your device uploads photo data to a cloud server via an HTTP request. The data is sent using an encrypted communication protocol.

[1401] Output: Photo data uploaded to the cloud server

[1402] Step 3:

[1403] The server inputs the photographic data into image analysis software to recognize the food items.

[1404] Input: Uploaded photo data

[1405] How it works: The server uses image analysis software (e.g., OpenCV, AWS Rekognition) to label and identify each food item in a photo, also using machine learning models in the process.

[1406] Output: A list of recognized food items

[1407] Step 4:

[1408] The server sends the list of recognized food items to the user's app.

[1409] Input: A list of recognized food items

[1410] How it works: The server sends a list of food items to the user's app using an HTTP response.

[1411] Output: A list of food items displayed in the user's app

[1412] Step 5:

[1413] The server searches a menu recipe database based on the food item list and suggests an appropriate menu.

[1414] Input: Food item list

[1415] How it works: The server queries a recipe database (e.g. MySQL, MongoDB) to find recipes based on combinations of food items.

[1416] Output: A list of suggested recipes

[1417] Step 6:

[1418] The server sends the suggested recipe list to the user's app.

[1419] Input: Suggested recipe list

[1420] How it works: The server sends the recipe list to the user's app using an HTTP response.

[1421] Output: A list of recipes displayed in the user's app

[1422] Step 7:

[1423] The user selects a desired recipe from the suggested recipe list and displays its detailed information.

[1424] Input: Suggested recipe list

[1425] What it does: A user selects a recipe in the app and sees details (ingredients, cooking steps, cooking time, etc.).

[1426] Output: Details of the selected recipe displayed in the user's app

[1427] Step 8:

[1428] The server maintains a user-specified list of ingredients that are always on hand and compares it with a list of recognized food items to detect missing ingredients.

[1429] Input: List of ingredients and food items

[1430] How it works: The server compares both lists and identifies any missing ingredients.

[1431] Output: List of missing materials

[1432] Step 9:

[1433] The server generates a shopping list including missing ingredients and sends it to the user's app.

[1434] Input: Shortage material list

[1435] How it works: The server generates a shopping list and sends it to the user's app in an HTTP response.

[1436] Output: Shopping list displayed in the user's app

[1437] Step 10:

[1438] The user can view the shopping list from the app and share it with other family members using an external messaging service.

[1439] Input: Shopping list

[1440] How it works: A user uses the share button in the app to share their shopping list via LINE or email.

[1441] Output: Shopping list shared by family members

[1442] Step 11:

[1443] The server uses an emotion analysis engine to analyze the user's emotional state.

[1444] Input: Voice data, touch operation data

[1445] How it works: The server uses an emotion analysis engine (e.g., IBM Watson, Azure Cognitive Services) to analyze the tone of voice and the speed of touch movements to estimate the user's emotional state.

[1446] Output: Estimation of the user's emotional state

[1447] Step 12:

[1448] The server adjusts menu suggestions based on the estimated emotional state.

[1449] Input: Emotional state estimation result, suggested recipe list

[1450] How it works: The server changes the priority of meal suggestions based on the user's emotional state, and when stress is high, prioritizes easy-to-prepare meals.

[1451] Output: A menu list tailored based on emotional state

[1452] Through this series of processes, users can efficiently manage the food in their refrigerator and are presented with the optimal menu based on their emotional state.

[1453] (Application example 2)

[1454] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1455] Conventional food management systems only recognize food items in the refrigerator and suggest menu items, but do not provide optimal suggestions based on the user's emotional state. Furthermore, they lacked a mechanism for automatically adding data on food purchased in physical stores to the refrigerator list, placing a heavy burden on the user as they had to manually enter the information. Furthermore, the system sometimes found it inconvenient to provide a shopping list that was optimal for the user's situation or emotions.

[1456] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data of the inside of the refrigerator from the user terminal, means for transmitting the acquired image data to the cloud server, means for analyzing the image data and recognizing food items in the cloud server, means for proposing a menu based on the recognized food items, means for transmitting the proposed menu to the user terminal, means for scanning barcodes of food items from the user terminal in the physical store and adding the data to the refrigerator list, means for adjusting the menu based on the user's emotional state, means for analyzing the user's voice data and operation actions using an emotion engine and estimating the user's emotional state, and means for generating an optimal shopping list based on the emotional state. This enables suggestions based on the user's emotional state and efficient food management in the physical store.

[1457] A "user terminal" is a mobile information terminal such as a smartphone or tablet that is operated by a user.

[1458] "Image data" refers to image or photograph data of the state inside the refrigerator.

[1459] A "cloud server" is a server system for storing and processing data on a remote server on the Internet.

[1460] "Food items" refers to the types of ingredients and foods contained in the refrigerator.

[1461] A "menu" refers to a list or recipe of suggested dishes based on ingredients.

[1462] A "barcode" is an identification code consisting of a series of black lines and spaces used to represent product information.

[1463] "Emotional state" refers to the emotions and mental state that the user is currently experiencing.

[1464] An "emotion engine" is software or algorithms for analyzing and inferring a user's emotional state.

[1465] "Voice data" refers to the voice of a user recorded as digital data.

[1466] "Operation actions" refer to actions such as touching and swiping that a user performs on a device.

[1467] A "shopping list" is a list of food and daily necessities that need to be purchased.

[1468] This invention is a system that allows users to efficiently manage food in their refrigerators and create menus, and also combines it with an emotion engine that recognizes the user's emotions and optimizes the suggestions, thereby making optimal suggestions according to the user's emotional state.

[1469] System Overview

[1470] 1. Photographing and analyzing the inside of the refrigerator

[1471] The user uses a smartphone app to capture image data of the inside of the refrigerator using the smartphone's camera function.

[1472] The device sends the acquired image data to the cloud server using an HTTP request.

[1473] The server analyzes the image data on the cloud and recognizes food items. Image analysis is performed using OpenCV.

[1474] The server generates a list of recognized food items and transmits it to the user terminal.

[1475] 2. Menu suggestions

[1476] The server searches a menu recipe database based on the food item list, which stores various combinations of food items and corresponding cooking recipes.

[1477] The server sends a list of found recipes to the user's device, and the user can review and select suggested recipes on the app.

[1478] 3. Check your stock of ingredients and create a shopping list

[1479] The server maintains a list of ingredients on hand designated by the user on a cloud server and compares the food items recognized from the transmitted image data with the list of ingredients on hand.

[1480] The server identifies the missing ingredients and generates a shopping list that includes them.

[1481] The server sends the generated shopping list to the user terminal, where the user can share it with others using an external messaging service.

[1482] 4. Emotion engine integration and optimization

[1483] When a user expresses their emotional state through voice input while operating the app, the voice data and operation data are sent to the emotion engine, which uses the Hugging Face Transformers library.

[1484] The server uses an emotion engine to infer the user's emotional state from the data, for example, by analyzing the tone of voice or the speed of haptic movements.

[1485] The server adjusts the meal suggestions and shopping list it provides based on the user's emotional state. For example, if the user is feeling stressed, it will prioritize easy-to-prepare meals.

[1486] Physical store operations

[1487] Furthermore, this system is also intended for use in physical stores. For example, when a user purchases food at a supermarket, they can scan the barcode on the product with a smartphone app. The barcode data is sent to the server and immediately added to the refrigerator list. This allows the user to automatically update their shopping list and manage it efficiently.

[1488] Specific examples

[1489] Food Recognition in the Refrigerator:

[1490] Users take a photo of their refrigerator with their smartphone and upload it to the app, where the food recognition system identifies "tomatoes," "eggs," "cheese," and "milk," and creates a list of these foods.

[1491] Menu suggestions based on user sentiment:

[1492] When a user voice-inputs into the app, "I'm busy and tired today," the emotion engine recognizes the "tired" state and suggests an easy-to-make "tomato salad."

[1493] Examples of prompts:

[1494] Take a picture of the inside of your refrigerator and identify the following food items: tomatoes, eggs, cheese, and milk. Based on this information, suggest a simple meal plan. Also include suggestions for when the user is tired.

[1495] In this way, appropriate suggestions based on the user's emotional state and efficient food management in physical stores can be realized.

[1496] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1497] Step 1:

[1498] A user uses a smartphone app to capture image data of the inside of a refrigerator. The user launches the app and takes a photo of the inside of the refrigerator with the smartphone camera. Once the photo is captured, the app prepares the image data.

[1499] Input: Photo data of the inside of a refrigerator taken with a smartphone camera.

[1500] Output: The image data is saved in the app on your smartphone.

[1501] Step 2:

[1502] The device sends the acquired image data to the cloud server, and the smartphone app uses an HTTP request to upload this image data to the cloud server.

[1503] Input: The acquired image data.

[1504] Output: The image data is transferred to the cloud server.

[1505] Step 3:

[1506] The server analyzes the image data on the cloud and recognizes food items. OpenCV is used to automatically identify food items from the image data and assign labels to each ingredient.

[1507] Input: Image data sent to the cloud server.

[1508] Output: A list of recognized food items.

[1509] Step 4:

[1510] The server searches a menu recipe database based on the recognized food items, uses the food list to search the database for related cooking recipes, and generates a list of found recipes.

[1511] Input: A list of recognized food items.

[1512] Output: A list of recipes.

[1513] Step 5:

[1514] The server sends a list of found recipes to the user's device. The cloud server prepares the recipe list and sends it to the smartphone app using an HTTP request.

[1515] Input: A list of recipes.

[1516] Output: The recipe list is displayed on the smartphone app.

[1517] Step 6:

[1518] A user scans the barcode of a food item at a physical store. The user activates the barcode scanner on their smartphone app and scans the barcode of the food item they want to purchase. The barcode data is sent by the app to a cloud server.

[1519] Input: Scanned barcode data.

[1520] Output: The barcode data is sent to the cloud server.

[1521] Step 7:

[1522] The server receives the barcode data and updates the refrigerator list accordingly, so that the newly added food item is reflected in the refrigerator list.

[1523] Input: Barcode data.

[1524] Output: Updated refrigerator list.

[1525] Step 8:

[1526] The user inputs their emotional state using the app. The user expresses their emotional state, such as "I'm tired today," through voice input or text input. The emotional data is sent to a cloud server.

[1527] Input: User's emotional state (audio or text data).

[1528] Output: Emotion data is sent to the cloud server.

[1529] Step 9:

[1530] The server uses an emotion engine to analyze the emotion data and estimate the user's emotional state. It uses the Hugging Face Transformers library to analyze voice tones and operational movements.

[1531] Input: Emotion data.

[1532] Output: Estimated emotional state.

[1533] Step 10:

[1534] The server then adjusts the menu suggestions it provides based on the user's estimated emotional state. For example, if the user is feeling stressed, it will prioritize easy-to-make recipes.

[1535] Input: Emotional state.

[1536] Output: Adjusted menu suggestions.

[1537] Step 11:

[1538] The server compares the list of ingredients with the list of recognized food items to identify missing ingredients, creates a shopping list based on the missing ingredients, and sends it to the user's device.

[1539] Input: Stock ingredient list and recognized food item list.

[1540] Output: A shopping list including missing ingredients is generated and sent to the user's device.

[1541] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1542] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1543] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1544] [Fourth embodiment]

[1545] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1546] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1547] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1548] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1549] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1550] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1551] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1552] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1553] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1554] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1555] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1556] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1557] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1558] This invention is a system for streamlining food management and menu suggestions. It automatically recognizes ingredients in the refrigerator and suggests menus based on them, so that users don't have to waste time creating daily menus and managing shopping lists. It also provides a function to detect shortages of regular ingredients and create and share shopping lists.

[1559] System Overview

[1560] 1. Photographing and analyzing the inside of the refrigerator

[1561] A user uses a smartphone app to take a photo of the inside of the refrigerator.

[1562] The device sends the photographed photo data to a cloud server.

[1563] The server inputs the photo data into image analysis software to recognize the food items in the refrigerator. Specifically, the server uses image analysis algorithms (e.g., deep learning techniques) to identify each ingredient in the photo and assign it a label.

[1564] The server generates a list of recognized food items and sends it to the user's app, where the user can view the list of food items in the refrigerator.

[1565] 2. Menu suggestions

[1566] Based on the analysis results, the server searches a database for menu recipes using the food items in the refrigerator. The recipe database stores various combinations of food items and corresponding cooking recipes.

[1567] The server sends the list of found recipes to the user's app.

[1568] Users can select their favorite recipe from the suggested recipes, and the details of the selected recipe (ingredients, cooking method, required time, etc.) will be displayed.

[1569] 3. Check your stock of ingredients and create a shopping list

[1570] The cloud server stores a list of ingredients that the user has pre-set (for example, "onions, carrots, soy sauce").

[1571] The server compares the current food item list with the stock ingredient list to identify any missing food items.

[1572] The server generates a shopping list based on the missing food items, which can include any missing staple ingredients as well as any additional ingredients needed for the user's selected meal.

[1573] Users can check their shopping list in the app and share it using external messaging services (e.g., LINE), allowing them to ask other family members to do the shopping.

[1574] Specific examples

[1575] 1. Photographing and analyzing the inside of the refrigerator

[1576] The user launches the smartphone app and presses the "take a photo of the inside of the refrigerator" button.

[1577] The device uses its camera function to take pictures of the inside of the refrigerator and sends the image data to a cloud server.

[1578] The server analyzes the image and recognizes that there are tomatoes, eggs, cheese, and milk in the refrigerator.

[1579] The server generates a list of food items and sends it to the user's app.

[1580] 2. Menu suggestions

[1581] The server searches the database based on the list of ingredients and finds recipes such as "tomato omelet," "cheese omelet," and "tomato salad."

[1582] The server sends these recipes to the user's app.

[1583] When a user selects "Tomato Omelette" from the app, the recipe details are displayed.

[1584] 3. Check your stock of ingredients and create a shopping list

[1585] The server maintains a list of ingredients (e.g., onions, carrots, soy sauce).

[1586] The server compares the latest list of food items in the refrigerator with the list of regularly-available ingredients and detects that there are shortages of "onions" and "carrots."

[1587] The server generates a shopping list for "onions, carrots" and sends it to the user's app.

[1588] The user checks the shopping list on the app and sends it to her husband via LINE. On the way home, the husband buys onions and carrots at the supermarket.

[1589] As described above, the present invention reduces the burden on the user and provides a method for efficiently managing ingredients and creating menus at home.

[1590] The processing flow will be explained below.

[1591] Step 1:

[1592] The user launches the smartphone app and takes a photo of the inside of the refrigerator. The user then uses the camera function to capture an overall image of the inside of the refrigerator, and presses the "Send" button when the photo is complete.

[1593] Step 2:

[1594] The device sends the captured photo data to the cloud server. Specifically, the device sends an HTTP request to upload the photo data to the server.

[1595] Step 3:

[1596] The server receives the HTTP request and stores the transmitted photo data, ensuring that the server has the necessary photo data for future processing.

[1597] Step 4:

[1598] The server inputs the stored photo data into image analysis software to recognize the food items in the refrigerator. Specifically, it uses image recognition algorithms such as deep learning to identify each ingredient in the photo and assign it an individual label.

[1599] Step 5:

[1600] The server generates a list of food items based on the analysis results and sends the list to the user's app, where the user can view the list of food items in the refrigerator.

[1601] Step 6:

[1602] The server searches the menu recipe database based on the food item list. The recipe database stores various combinations of food items and corresponding cooking recipes. The server identifies recipes for dishes using the ingredients in the refrigerator.

[1603] Step 7:

[1604] The server sends a list of found recipes to the user's app, where the user can select the recipes they like.

[1605] Step 8:

[1606] The user uses the app to review the suggested recipes and selects a recipe, such as "Tomato Omelette." Details of the selected recipe (e.g., ingredients, cooking instructions, cooking time, etc.) are displayed in the app.

[1607] Step 9:

[1608] The server retrieves the user's predefined list of ingredients (e.g., "onions, carrots, soy sauce") from the database. The server compares this list with the latest list of food in the refrigerator.

[1609] Step 10:

[1610] The server compares the list of ingredients in stock with the list of food in the refrigerator and identifies any ingredients that are missing. For example, it determines that "onions" and "carrots" are missing.

[1611] Step 11:

[1612] The server generates a shopping list based on the missing ingredients, which can include the missing regular ingredients as well as any additional ingredients needed for the recipe selected by the user.

[1613] Step 12:

[1614] The server sends the generated shopping list to the user's app, where the user can view the shopping list.

[1615] Step 13:

[1616] The user presses the "Share Shopping List" button in the app and selects an external messaging service such as LINE. The device then sends the shopping list as a LINE message to share with other family members.

[1617] The above are the specific processing steps of the system, which allow users to proactively manage food and create menus.

[1618] Example 1

[1619] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1620] Conventional food management and menu suggestion systems have the problem that users have to manually check the ingredients in the refrigerator and search for recipes, which is time-consuming. Furthermore, to determine if a necessary ingredient is missing, the user must check the refrigerator again, making efficient food management difficult. This invention aims to reduce the user's burden and provide efficient food management and menu suggestions by automatically recognizing ingredients in the refrigerator using image analysis technology and suggesting menus based on that information.

[1621] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1622] In this invention, the server includes means for acquiring image data from a user terminal, means for transmitting the acquired image data via a network, means for analyzing the image data and recognizing object items in the server, means for providing information based on the recognized object items, and means for transmitting the provided information to the user terminal. This reduces the user's workload and enables efficient food management and menu suggestions. The server also includes means for storing a reference list specified by the user, comparing the object items recognized from the transmitted image data with the reference list to detect missing items, generating a list including the missing items, and means for transmitting the generated list to the user terminal, allowing the user to easily identify missing ingredients and use it as a shopping list. Furthermore, the server also includes means for transmitting the generated list from the user terminal using an external communication service, allowing the user to share the shopping list with family and other members and shop collaboratively.

[1623] "User terminal" refers to a device operated by a user, including a smartphone, tablet, or PC.

[1624] "Image data" refers to photographs or scanned images of the inside of the refrigerator, including visual information stored in digital form.

[1625] "Network" refers to a communications system that enables data to be communicated between devices, including the Internet and local area networks.

[1626] A "server" refers to a central computer system that processes data and provides information in response to requests from clients (user terminals).

[1627] "Object item" refers to an individual entity (e.g., food or object) recognized from image data.

[1628] "Information" is content provided based on the object item, including recipes, ingredient lists, etc.

[1629] The "standard list" refers to a list of items that should be kept as standard, set by the user, and includes, for example, a list of regular materials.

[1630] "External communication services" are services used to send and receive data, including messaging applications and email services.

[1631] A "list" is a list of specific items, including a list of missing items and a shopping list.

[1632] This invention is a system for streamlining food management and menu suggestions. It combines a user terminal, a server, an image analysis algorithm, and an external communication service to significantly reduce the user's workload.

[1633] System Configuration

[1634] The system mainly consists of the following hardware and software elements:

[1635] User device: Use a smartphone, tablet, or computer.

[1636] Server: A cloud server is used to process data from multiple user devices.

[1637] Image analysis algorithm: Uses deep learning models to perform object recognition.

[1638] External communication services: Using messaging applications (e.g., messaging apps, email).

[1639] Operating principle

[1640] The system's operation can be broadly divided into the following steps:

[1641] Taking photos of the inside of the refrigerator and analyzing them

[1642] A user uses a smartphone app to take a photo of the inside of their refrigerator. The device then sends the captured photo data to a cloud server. The server then uses an image analysis algorithm (deep learning technology) to recognize food items in the photo and assigns labels to each. Specifically, it uses an object recognition algorithm to extract food items such as "tomato," "egg," and "cheese." The server then generates a list of the recognized food items and sends it to the user's app.

[1643] Menu suggestions

[1644] Based on the analysis results, the server searches a database for menu recipes using the food items in the refrigerator. The recipe database stores various combinations of food items and corresponding cooking recipes. For example, a database search can find recipes such as "tomato omelette," "cheese omelette," and "tomato salad." The server sends a list of these recipes to the user's app, and the user can select their preferred recipe from the suggested recipes. Details of the selected recipe (ingredients, cooking method, required time, etc.) are also displayed in the app.

[1645] Check your stock of ingredients and create a shopping list

[1646] The server maintains a standard list (e.g., onions, carrots, soy sauce, etc.) preset by the user. The server compares the latest list of food items in the refrigerator with the standard list to identify any missing food items. Specifically, if the server detects that "onions" and "carrots" are missing, it generates a shopping list that includes these items. The shopping list can include not only the missing basic ingredients but also additional ingredients required for the meal selected by the user. The user can check the shopping list in the app and share it with other family members using an external communication service (e.g., a messaging app).

[1647] Specific examples

[1648] 1. Photographing and analyzing the inside of the refrigerator

[1649] The user launches the app and presses the "Take a photo of the inside of the refrigerator" button.

[1650] The device uses its camera function to take pictures of the inside of the refrigerator and sends the image data to the server.

[1651] The server analyzes the image and recognizes that there are tomatoes, eggs, cheese, and milk in the refrigerator.

[1652] The server generates a list of food items and sends it to the user's app.

[1653] 2. Menu suggestions

[1654] The server searches the database based on the list of ingredients and finds recipes such as "tomato omelet," "cheese omelet," and "tomato salad."

[1655] The server sends these recipes to the user's app.

[1656] When a user selects "Tomato Omelette" from the app, the recipe details are displayed.

[1657] 3. Check your stock of ingredients and create a shopping list

[1658] The server has a pre-stored list of criteria (e.g., onions, carrots, soy sauce).

[1659] The server compares the current food list with the reference list and identifies that "onions" and "carrots" are in short supply.

[1660] The server generates a shopping list that includes "onions, carrots" and sends it to the user's app.

[1661] A user sends a shopping list to their family via a messaging app, and the family members then go to the supermarket to purchase the items.

[1662] Prompt Sentence Examples

[1663] "Describe a system that recognizes the foods in a refrigerator, generates a list, and notifies the user of that list."

[1664] "Please explain the process of a system that suggests recipes based on a list of foods in the refrigerator."

[1665] "Describe a system that detects shortages of staple ingredients and automatically generates a shopping list."

[1666] In this way, it is possible to significantly improve the efficiency of the user's food ingredient management and menu proposals. This system reduces the burden on the user and provides a method for efficiently managing food at home and creating menus.

[1667] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1668] Step 1:

[1669] The user launches the smartphone app and takes a photo of the inside of the refrigerator.

[1670] Specific operation: The user presses the "Take a photo inside the refrigerator" button and takes a photo of the inside of the refrigerator using the smartphone camera.

[1671] Input: Visual information inside the refrigerator

[1672] Output: Image data

[1673] Step 2:

[1674] The device sends the photographed photo data to a cloud server.

[1675] Specific behavior: Uploads an image to the server using the HTTPS protocol.

[1676] Input: Image data

[1677] Output: Image data stored on a cloud server

[1678] Step 3:

[1679] The server uses image analysis algorithms (e.g., deep learning techniques) to recognize food items in the refrigerator.

[1680] Input: Image data stored on a cloud server

[1681] Processing: Object recognition with deep learning models

[1682] Output: A list of recognized food items

[1683] What it does: The server runs an image analysis model to identify food items such as "tomato," "egg," and "cheese."

[1684] Step 4:

[1685] The server sends the recognized food item list to the user terminal.

[1686] Specific operation: The server transfers the food item list generated to the user's device using the REST API.

[1687] Input: A list of recognized food items

[1688] Output: A list of food items in the refrigerator displayed on the user's device

[1689] Step 5:

[1690] The server searches a database based on the list of food items to find available recipes.

[1691] What it does: Uses a database query to extract the corresponding recipes.

[1692] Input: Food item list

[1693] Processing: Database queries and recipe extraction

[1694] Output: List of available recipes

[1695] Step 6:

[1696] The server sends the recipe list it finds to the user's terminal.

[1697] Specific operation: Send the recipe list to the user's terminal using the REST API.

[1698] Input: Available recipe list

[1699] Output: Recipe list displayed on the user's device

[1700] Step 7:

[1701] The user selects one of the suggested recipes and displays the details.

[1702] Specific behavior: Displays details (ingredients, cooking instructions, cooking time, etc.) of a recipe selected by the user from the app.

[1703] Input: Recipe ID selected by the user

[1704] Output: Detailed information about the selected recipe

[1705] Step 8:

[1706] The server maintains a list of criteria set by the user and compares it with the list of food items to detect missing items.

[1707] Specific Behavior: Compare the list of staple ingredients with the list of food items and identify any missing items.

[1708] Input: Criteria list and current food list

[1709] Processing: List comparison and missing item detection

[1710] Output: Missing items list

[1711] Step 9:

[1712] The server generates a shopping list including the missing items and transmits it to the user terminal.

[1713] Specific operation: Generate a shopping list and send it to the user's device using the REST API.

[1714] Input: Missing Item List

[1715] Output: Shopping list displayed on the user's device

[1716] Step 10:

[1717] The user shares the generated shopping list through an external communication service.

[1718] What it does: A user checks a shopping list within the app and sends it to family and other members using an external messaging service.

[1719] Input: Shopping list

[1720] Output: Shared shopping list

[1721] Through these steps, the system enables users to manage the ingredients in their refrigerator and efficiently suggest menus.

[1722] (Application example 1)

[1723] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1724] In modern households, managing food in the refrigerator and creating daily menus are tasks that require a great deal of time and effort. Creating appropriate shopping lists and replenishing necessary ingredients are also time-consuming. Furthermore, with the increasing demand for eating out and food delivery, there is a need for ways to enjoy meals efficiently without wasting ingredients at home. The present invention aims to solve these problems by providing a system that efficiently handles everything from food management to menu suggestions, shopping list creation, and delivery arrangements.

[1725] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1726] In this invention, the server includes means for acquiring image data of the contents of the refrigerator from the user terminal, means for transmitting the acquired image data to a cloud server, means for analyzing the image data and recognizing food items in the cloud server, means for proposing menus based on the recognized food items, means for transmitting the proposed menus to the user terminal, and means for coordinating with an external service that provides the proposed menus and arranging delivery. This allows the user to reduce food waste in the refrigerator, receive optimal menu suggestions without hassle, and smoothly arrange for the delivery of ingredients that are in short supply.

[1727] A "user terminal" is an electronic device operated by a user, and has the function of acquiring image data of the inside of a refrigerator.

[1728] "Image data" is image information showing the state of food in the refrigerator, and is sent from the user terminal to the cloud server.

[1729] A "cloud server" is a remote computer connected to a network that analyzes image data and processes various types of data.

[1730] "Food item" refers to ingredients or types of ingredients recognized from image data.

[1731] A "menu" is a list of cooking suggestions created based on recognized food items.

[1732] "Suggested menu" refers to a menu list generated by the cloud server, from which the user can select.

[1733] "External services" refer to companies or systems that work in conjunction with the cloud server to deliver suggested menus and missing ingredients.

[1734] "Delivery" refers to a service that delivers meals based on a suggested menu and any missing ingredients to the user's home.

[1735] The "stock ingredient list" is a list of ingredients that the user sets in advance and that the user wants to keep in the refrigerator at all times.

[1736] A "missing ingredient" is an ingredient that is not present in the refrigerator when comparing a recognized food item with the list of ingredients.

[1737] A "shopping list" is a list of ingredients that are in short supply and ingredients that the user wishes to purchase in addition.

[1738] "External messaging service" means a communication means used by a user to send a generated shopping list to another person.

[1739] A "generative AI model" is an artificial intelligence program that has learned algorithms to perform specific tasks based on large amounts of data.

[1740] "Prompt sentence" refers to an instruction sentence that a generative AI model uses to start a task.

[1741] This invention is a system that combines user terminals, cloud servers, delivery partners, and external messaging services to provide a comprehensive range of services, from managing food in the refrigerator to suggesting menus and delivering missing ingredients.

[1742] System Overview

[1743] 1. Acquire and send image data from the user's device:

[1744] The user terminal has a camera function for taking pictures of the inside of the refrigerator.

[1745] When a user uses a smartphone app to take a photo of the inside of the refrigerator, the device sends the image data to a cloud server.

[1746] 2. Ingredient recognition and menu suggestions on the cloud server:

[1747] The cloud server analyzes the received image data and recognizes the food items in the refrigerator using a deep learning model.

[1748] The server lists food items and suggests meals based on the recognized ingredients, selected from recipes stored in a database.

[1749] The server sends the proposed menu to the user terminal, allowing the user to select a menu.

[1750] 3. Detecting missing ingredients and generating a shopping list:

[1751] The server maintains a list of ingredients that are pre-defined by the user.

[1752] Compare recognized food items with a list of staple ingredients to identify missing ingredients.

[1753] A shopping list is generated based on the ingredients that are missing, and delivery of the missing ingredients is arranged through a delivery partner if necessary.

[1754] The generated shopping list is sent to the user's terminal and can also be shared with others via external messaging services.

[1755] Hardware and software used

[1756] Hardware: Smartphone (with camera), cloud server

[1757] Software: OpenCV (for image processing), requests (for HTTP requests), deep learning models (e.g. TensorFlow, PyTorch)

[1758] Specific examples

[1759] As an example, consider a scenario where a user takes a photo of the inside of a refrigerator and uploads it to a cloud server.

[1760] The server performs image analysis and recognizes that there are tomatoes, eggs, and milk in the refrigerator.

[1761] The server proposes menu items such as "tomato omelet" and "milk pudding" based on these ingredients and sends them to the user's terminal.

[1762] The user selects "tomato omelette," the server detects the missing ingredient (e.g., "onion"), and arranges for the onion to be delivered through a delivery partner.

[1763] Example prompts from a generative AI model

[1764] An example prompt is, "Based on the ingredients in your refrigerator, suggest healthy meals you might like. If necessary, order additional ingredients through a delivery service."

[1765] These processes allow the user to efficiently manage ingredients in the refrigerator, enjoy suggested meals, and smoothly arrange for delivery of ingredients that are in short supply.

[1766] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1767] Step 1:

[1768] The user launches the smartphone app and presses the "Take a photo of the refrigerator" button. The smartphone camera starts up, and the user takes a photo of the food in the refrigerator. The photo is saved as image data on the device.

[1769] Step 2:

[1770] The device sends the captured image data to the cloud server using an HTTP request, and the data sent is in the form of an image file.

[1771] Step 3:

[1772] The server inputs the received image data into an image analysis program, which uses a deep learning model to recognize food ingredients. The input is the image data, and the output is a list of recognized food items.

[1773] Step 4:

[1774] The server runs a menu suggestion algorithm based on the list of food items. The algorithm references recipe information stored in the database and suggests menus based on the input food items. The output is a list of suggested recipes.

[1775] Step 5:

[1776] The server sends a list of suggested menus to the user's terminal. The user checks the suggested menus on the terminal and selects the menu of their choice. The selected menu becomes the input for the next step.

[1777] Step 6:

[1778] The server detects missing ingredients based on the menu selected by the user and the recognized food item list. The input is the food item list and the stock ingredient list, and the output is the missing ingredients.

[1779] Step 7:

[1780] The server generates a shopping list based on the missing ingredients. The input is the list of missing ingredients, and the output is the shopping list.

[1781] Step 8:

[1782] The server sends the generated shopping list to the user's device, which can then share it with others via an external messaging service. To share, a specific prompt is generated using the generative AI model and sent as a message.

[1783] Step 9:

[1784] The user checks the shopping list and orders any missing ingredients through an external delivery partner. The server then requests the linked external service to arrange delivery.

[1785] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1786] This invention is a system that allows users to efficiently manage food in their refrigerators and create menus, and also combines it with an emotion engine that recognizes the user's emotions and optimizes the suggestions, thereby making optimal suggestions according to the user's emotional state.

[1787] System Overview

[1788] 1. Photographing and analyzing the inside of the refrigerator

[1789] A user takes a photo of the inside of the refrigerator using a smartphone app.

[1790] The device sends the captured photo data to the cloud server, uploading the photo data via an HTTP request.

[1791] The server inputs the received photo data into image analysis software to recognize the food items in the refrigerator. Image recognition algorithms are used to label and identify each ingredient in the photo.

[1792] The server generates a list of the food items obtained and sends it to the user's app, where the user can view the list of food items in the refrigerator.

[1793] 2. Menu suggestions

[1794] The server searches a menu recipe database based on the food item list. The recipe database stores various combinations of food items and corresponding cooking recipes.

[1795] The server sends a list of found recipes to the user's app, where the user can review and select the suggested recipes.

[1796] The app displays detailed information about the recipe the user selects (e.g., ingredients, cooking instructions, cooking time, etc.).

[1797] 3. Check your stock of ingredients and create a shopping list

[1798] The server maintains the user's pre-defined list of ingredients and compares it with the current list of food in the refrigerator.

[1799] The server compares the stock of ingredients to identify any missing ingredients.

[1800] The server generates a shopping list based on the missing ingredients and sends the list to the user's app.

[1801] Users can check their shopping list from the app and share it with other family members using external messaging services such as LINE.

[1802] 4. Emotion engine integration and optimization

[1803] When a user uses the app, they input voice and make touch gestures, and this data is collected and analyzed by the emotion engine.

[1804] The device sends voice data and touch operation data to a cloud server.

[1805] The server uses an emotion engine to estimate the user's emotional state from the collected data, for example, determining whether the user is feeling stressed based on the tone of their voice or the speed of their touch gestures.

[1806] The server then adjusts the menu suggestions it provides based on the user's estimated emotional state. For example, if the user is feeling stressed, it will prioritize easy-to-prepare menu suggestions.

[1807] The server also adjusts the contents of the shopping list based on the emotional state, reducing the burden on the user.

[1808] Specific examples

[1809] 1. Photographing and analyzing the inside of the refrigerator

[1810] The user launches the smartphone app and takes a photo of the inside of the refrigerator.

[1811] The device sends the photo data to the cloud server.

[1812] The server analyzes the photo and recognizes "tomato," "egg," "cheese," and "milk."

[1813] The server generates a list of food items and sends it to the user's app.

[1814] 2. Menu suggestions

[1815] Based on the list of ingredients, the server finds recipes such as "tomato omelet," "cheese omelet," and "tomato salad" from the database.

[1816] The server sends these recipes to the user's app.

[1817] If the user selects "Tomato Omelette," detailed information is displayed.

[1818] 3. Check your stock of ingredients and create a shopping list

[1819] The server maintains and analyzes the user's stockpile of ingredients.

[1820] The server compares the list of food items in the refrigerator with the list of regularly available ingredients and detects that "onions" and "carrots" are missing.

[1821] The server generates a shopping list that includes "onions" and "carrots" and sends it to the user's app.

[1822] Users can check their shopping list, share it using LINE, and ask family members to do the shopping.

[1823] 4. Emotion engine integration and optimization

[1824] While using the app, the user utters the following using voice input: "I'm tired today."

[1825] The device sends the voice data to the cloud server.

[1826] The server uses an emotion engine to infer from the voice data that the user is tired.

[1827] The server will prioritize suggestions for dishes that are easy to prepare (e.g., "easy salad").

[1828] The server also adjusts the shopping list to include more ingredients that are easy to prepare.

[1829] This invention reduces the burden on users and supports food management and cooking tasks more efficiently by proposing optimal menus and shopping lists according to the user's emotional state.

[1830] The processing flow will be explained below.

[1831] Step 1:

[1832] The user starts the smartphone app and takes a photo of the inside of the refrigerator. The user then uses the app's camera function to take a photo of the entire refrigerator.

[1833] Step 2:

[1834] The device sends the captured photo data to the cloud server. Specifically, it creates an HTTP request including the photo data and uploads it to the cloud server.

[1835] Step 3:

[1836] The server receives the HTTP request and stores the sent photo data in cloud storage.

[1837] Step 4:

[1838] The server inputs the stored photo data into image analysis software to recognize the food items in the refrigerator. Image recognition algorithms (such as deep learning models) are used to identify the ingredients in the photo.

[1839] Step 5:

[1840] The server generates a list of food items based on the analysis results and sends the list to the user's app, where the user can view the list.

[1841] Step 6:

[1842] The server searches a menu recipe database based on the food item list. The database contains a variety of food item combinations and corresponding cooking recipes.

[1843] Step 7:

[1844] The server sends the list of recipes found to the user's app, where the user can view multiple menu suggestions and select the one they like best.

[1845] Step 8:

[1846] The user selects "Tomato Omelette" from the suggested recipes. The app displays detailed information about the selected recipe (e.g., ingredients, cooking instructions, cooking time, etc.).

[1847] Step 9:

[1848] The server retrieves the user's pre-defined ingredient list from the database, which includes ingredients and seasonings that the user frequently uses.

[1849] Step 10:

[1850] The server compares the latest list of food items in the refrigerator with the list of ingredients to identify any missing ingredients. For example, if "onions" and "carrots" are not on the list, they are recognized as missing ingredients.

[1851] Step 11:

[1852] The server generates a shopping list based on the missing ingredients, which may include the missing standard ingredients as well as additional ingredients needed for the selected meal.

[1853] Step 12:

[1854] The server sends the generated shopping list to the user's app, where the user can view the list.

[1855] Step 13:

[1856] The user presses the "Share Shopping List" button in the app and selects an external messaging service such as LINE. The device then sends the shopping list as a LINE message to share with other family members.

[1857] Step 14:

[1858] The device collects the user's voice input and touch actions and sends this data to a cloud server. For example, if a user says, "I'm tired today," the device collects this voice data.

[1859] Step 15:

[1860] The server uses an emotion engine to estimate the user's emotional state from the collected data, such as the tone of voice and the speed of touch movements, to determine whether the user is feeling stressed.

[1861] Step 16:

[1862] The server then tailors meal suggestions based on the user's estimated emotional state, for example, suggesting easy, hassle-free meals if the user is feeling stressed.

[1863] Step 17:

[1864] The server also adjusts the shopping list based on the user's emotional state: if the user is tired, for example, the list will prioritize ingredients that require less preparation.

[1865] Through the above processing steps, the system proposes optimal menus and shopping lists according to the user's emotional state. This system improves the efficiency of food management and cooking at home, reducing the burden on the user.

[1866] Example 2

[1867] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1868] For many users, efficiently managing food in the refrigerator and creating menus is a time-consuming and labor-intensive task. Furthermore, users often feel stressed and fatigued because the system does not provide optimal suggestions based on their emotional state. The present invention aims to solve these problems and enable users to manage food and create menus more effectively.

[1869] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring image data of the inside of the refrigerator from the user terminal, means for transmitting the acquired image data to the cloud server, means for analyzing the image data and recognizing food items in the cloud server, means for proposing a menu based on the recognized food items, means for transmitting the proposed menu to the user terminal, means for analyzing the emotional state of the user using an emotion analysis engine, and means for adjusting the proposed menu based on the analyzed emotional state. This not only enables the user to effectively manage the food in the refrigerator, but also enables the optimal menu to be proposed according to the user's emotional state.

[1870] 1. "User terminal" means an electronic device operated by a user, including smartphones and tablets.

[1871] 2. "Image Data" means photographs of food items in a refrigerator, stored in a digital format.

[1872] 3. "Cloud server" means a server that stores and processes data via the Internet, including a group of servers that can be accessed remotely.

[1873] 4. "Food items" refer to specific foods or ingredients in the refrigerator, and are individual ingredients recognized through image analysis.

[1874] 5. A "menu" is a combination of suggested dishes, including a list of dishes from which the user can choose.

[1875] 6. "Emotion Analysis Engine" refers to algorithms and software that analyze a user's voice and touch actions to infer their emotional state.

[1876] 7. "Means for adjusting the proposed menu" refers to means including a function for changing and optimizing the content of the menu provided to the user based on the analyzed emotional state.

[1877] 8. "Stock ingredient list" refers to a list of foods and ingredients that are used on a daily basis and that is preset by the user.

[1878] 9. "Missing ingredients" refers to foods or ingredients that are on the list but not present in the refrigerator when comparing the food in the refrigerator with the list of regular ingredients.

[1879] 10. "Shopping List" refers to a list generated based on missing materials and used by the user when shopping.

[1880] 11. "External messaging services" are tools that allow users to communicate with other users, including messaging applications via the Internet such as LINE and email.

[1881] This invention is a system that helps users manage the food in their refrigerator and create menus more efficiently, and also combines an emotion engine that recognizes the user's emotions and optimizes the suggestions. This system can make optimal suggestions according to the user's emotional state.

[1882] The system consists of user devices such as smartphones and tablets, a cloud server, image analysis software, and an emotion analysis engine.

[1883] Acquiring and sending image data inside the refrigerator

[1884] The user device has a camera function for taking photos of the inside of the refrigerator. The user launches the app and takes a photo of the inside of the refrigerator. The device then sends the captured photo data to a cloud server. This communication is performed via an HTTP request, and the data is encrypted for security reasons.

[1885] Image analysis and food item recognition

[1886] The cloud server inputs the captured photo data into image analysis software (e.g., OpenCV, AWS Rekognition, etc.), which recognizes each food item in the refrigerator and assigns a label to each one, generating a list of food items.

[1887] Menu suggestions and adjustments

[1888] The server searches a recipe database (e.g., MySQL, MongoDB, etc.) based on the recognized food item list. The recipe database stores combinations of food items and corresponding cooking recipes. The list of found recipes is sent to the user's app, where the user can review and select the suggested recipes.

[1889] Checking stock of materials and detecting shortages

[1890] The server maintains the user's pre-defined list of ingredients. It compares the latest refrigerator food list with the list of ingredients and identifies any missing ingredients. The server generates a shopping list based on the missing ingredients and sends it to the user's app. The user can then check the shopping list from the app and share it with other family members using messaging services such as LINE.

[1891] Sentiment analysis engine integration and optimization

[1892] When a user uses an app, voice input and touch action data are analyzed by an emotion analysis engine (e.g., IBM Watson, Azure Cognitive Services, etc.). The device sends the voice data and touch action data to a cloud server, which then uses the emotion analysis engine to estimate the user's emotional state. For example, it determines whether the user is feeling stressed based on the tone of their voice or the speed of their touch action. Based on the estimated emotional state, the server adjusts the meal suggestions it provides. If the user is tired, it will prioritize easy-to-make meals. The contents of the shopping list are also adjusted based on the user's emotional state to reduce the burden on the user.

[1893] Specific examples

[1894] For example, a user can use a smartphone app to take a photo of the inside of their refrigerator and send it to a cloud server. The server then uses image analysis software to recognize items such as "tomato," "egg," "cheese," and "milk." From the list of recognition results, the server searches for recipes such as "tomato omelette," "cheese omelette," and "tomato salad," and sends them to the user's app.

[1895] When a user voice-inputs "I'm tired today," the device sends that data to a cloud server. The emotion analysis engine recognizes the user's level of fatigue, and the server suggests easy-to-prepare meals such as "easy salads." In this way, the system suggests optimal meals based on the user's emotional state.

[1896] Prompt Sentence Examples

[1897] "Tell me about your refrigerator food management system. It uses an emotion recognition engine to suggest the best meal plan based on the user's emotional state. Specifically, how does it work and what technology is used?"

[1898] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1899] Program processing flow

[1900] Step 1:

[1901] The user launches the smartphone app and takes a photo of the inside of the refrigerator.

[1902] Input: Video data inside the refrigerator

[1903] How it works: A user takes a picture of food in the refrigerator using the camera function on their smartphone.

[1904] Output: Captured photo data

[1905] Step 2:

[1906] The device sends the photographed photo data to a cloud server.

[1907] Input: Photographed photo data

[1908] How it works: Your device uploads photo data to a cloud server via an HTTP request. The data is sent using an encrypted communication protocol.

[1909] Output: Photo data uploaded to the cloud server

[1910] Step 3:

[1911] The server inputs the photographic data into image analysis software to recognize the food items.

[1912] Input: Uploaded photo data

[1913] How it works: The server uses image analysis software (e.g., OpenCV, AWS Rekognition) to label and identify each food item in a photo, also using machine learning models in the process.

[1914] Output: A list of recognized food items

[1915] Step 4:

[1916] The server sends the list of recognized food items to the user's app.

[1917] Input: A list of recognized food items

[1918] How it works: The server sends a list of food items to the user's app using an HTTP response.

[1919] Output: A list of food items displayed in the user's app

[1920] Step 5:

[1921] The server searches a menu recipe database based on the food item list and suggests an appropriate menu.

[1922] Input: Food item list

[1923] How it works: The server queries a recipe database (e.g. MySQL, MongoDB) to find recipes based on combinations of food items.

[1924] Output: A list of suggested recipes

[1925] Step 6:

[1926] The server sends the suggested recipe list to the user's app.

[1927] Input: Suggested recipe list

[1928] How it works: The server sends the recipe list to the user's app using an HTTP response.

[1929] Output: A list of recipes displayed in the user's app

[1930] Step 7:

[1931] The user selects a desired recipe from the suggested recipe list and displays its detailed information.

[1932] Input: Suggested recipe list

[1933] What it does: A user selects a recipe in the app and sees details (ingredients, cooking steps, cooking time, etc.).

[1934] Output: Details of the selected recipe displayed in the user's app

[1935] Step 8:

[1936] The server maintains a user-specified list of ingredients that are always on hand and compares it with a list of recognized food items to detect missing ingredients.

[1937] Input: List of ingredients and food items

[1938] How it works: The server compares both lists and identifies any missing ingredients.

[1939] Output: List of missing materials

[1940] Step 9:

[1941] The server generates a shopping list including missing ingredients and sends it to the user's app.

[1942] Input: Shortage material list

[1943] How it works: The server generates a shopping list and sends it to the user's app in an HTTP response.

[1944] Output: Shopping list displayed in the user's app

[1945] Step 10:

[1946] The user can view the shopping list from the app and share it with other family members using an external messaging service.

[1947] Input: Shopping list

[1948] How it works: A user uses the share button in the app to share their shopping list via LINE or email.

[1949] Output: Shopping list shared by family members

[1950] Step 11:

[1951] The server uses an emotion analysis engine to analyze the user's emotional state.

[1952] Input: Voice data, touch operation data

[1953] How it works: The server uses an emotion analysis engine (e.g., IBM Watson, Azure Cognitive Services) to analyze the tone of voice and the speed of touch movements to estimate the user's emotional state.

[1954] Output: Estimation of the user's emotional state

[1955] Step 12:

[1956] The server adjusts menu suggestions based on the estimated emotional state.

[1957] Input: Emotional state estimation result, suggested recipe list

[1958] How it works: The server changes the priority of meal suggestions based on the user's emotional state, and when stress is high, prioritizes easy-to-prepare meals.

[1959] Output: A menu list tailored based on emotional state

[1960] Through this series of processes, users can efficiently manage the food in their refrigerator and are presented with the optimal menu based on their emotional state.

[1961] (Application example 2)

[1962] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1963] Conventional food management systems only recognize food items in the refrigerator and suggest menu items, but do not provide optimal suggestions based on the user's emotional state. Furthermore, they lacked a mechanism for automatically adding data on food purchased in physical stores to the refrigerator list, placing a heavy burden on the user as they had to manually enter the information. Furthermore, the system sometimes found it inconvenient to provide a shopping list that was optimal for the user's situation or emotions.

[1964] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring image data of the inside of the refrigerator from the user terminal, means for transmitting the acquired image data to the cloud server, means for analyzing the image data and recognizing food items in the cloud server, means for proposing a menu based on the recognized food items, means for transmitting the proposed menu to the user terminal, means for scanning barcodes of food items from the user terminal in the physical store and adding the data to the refrigerator list, means for adjusting the menu based on the user's emotional state, means for analyzing the user's voice data and operation actions using an emotion engine and estimating the user's emotional state, and means for generating an optimal shopping list based on the emotional state. This enables suggestions based on the user's emotional state and efficient food management in the physical store.

[1965] A "user terminal" is a mobile information terminal such as a smartphone or tablet that is operated by a user.

[1966] "Image data" refers to image or photograph data of the state inside the refrigerator.

[1967] A "cloud server" is a server system for storing and processing data on a remote server on the Internet.

[1968] "Food items" refers to the types of ingredients and foods contained in the refrigerator.

[1969] A "menu" refers to a list or recipe of suggested dishes based on ingredients.

[1970] A "barcode" is an identification code consisting of a series of black lines and spaces used to represent product information.

[1971] "Emotional state" refers to the emotions and mental state that the user is currently experiencing.

[1972] An "emotion engine" is software or algorithms for analyzing and inferring a user's emotional state.

[1973] "Voice data" refers to the voice of a user recorded as digital data.

[1974] "Operation actions" refer to actions such as touching and swiping that a user performs on a device.

[1975] A "shopping list" is a list of food and daily necessities that need to be purchased.

[1976] This invention is a system that allows users to efficiently manage food in their refrigerators and create menus, and also combines it with an emotion engine that recognizes the user's emotions and optimizes the suggestions, thereby making optimal suggestions according to the user's emotional state.

[1977] System Overview

[1978] 1. Photographing and analyzing the inside of the refrigerator

[1979] The user uses a smartphone app to capture image data of the inside of the refrigerator using the smartphone's camera function.

[1980] The device sends the acquired image data to the cloud server using an HTTP request.

[1981] The server analyzes the image data on the cloud and recognizes food items. Image analysis is performed using OpenCV.

[1982] The server generates a list of recognized food items and transmits it to the user terminal.

[1983] 2. Menu suggestions

[1984] The server searches a menu recipe database based on the food item list, which stores various combinations of food items and corresponding cooking recipes.

[1985] The server sends a list of found recipes to the user's device, and the user can review and select suggested recipes on the app.

[1986] 3. Check your stock of ingredients and create a shopping list

[1987] The server maintains a list of ingredients on hand designated by the user on a cloud server and compares the food items recognized from the transmitted image data with the list of ingredients on hand.

[1988] The server identifies the missing ingredients and generates a shopping list that includes them.

[1989] The server sends the generated shopping list to the user terminal, where the user can share it with others using an external messaging service.

[1990] 4. Emotion engine integration and optimization

[1991] When a user expresses their emotional state through voice input while operating the app, the voice data and operation data are sent to the emotion engine, which uses the Hugging Face Transformers library.

[1992] The server uses an emotion engine to infer the user's emotional state from the data, for example, by analyzing the tone of voice or the speed of haptic movements.

[1993] The server adjusts the meal suggestions and shopping list it provides based on the user's emotional state. For example, if the user is feeling stressed, it will prioritize easy-to-prepare meals.

[1994] Physical store operations

[1995] Furthermore, this system is also intended for use in physical stores. For example, when a user purchases food at a supermarket, they can scan the barcode on the product with a smartphone app. The barcode data is sent to the server and immediately added to the refrigerator list. This allows the user to automatically update their shopping list and manage it efficiently.

[1996] Specific examples

[1997] Food Recognition in the Refrigerator:

[1998] Users take a photo of their refrigerator with their smartphone and upload it to the app, where the food recognition system identifies "tomatoes," "eggs," "cheese," and "milk," and creates a list of these foods.

[1999] Menu suggestions based on user sentiment:

[2000] When a user voice-inputs into the app, "I'm busy and tired today," the emotion engine recognizes the "tired" state and suggests an easy-to-make "tomato salad."

[2001] Examples of prompts:

[2002] Take a picture of the inside of your refrigerator and identify the following food items: tomatoes, eggs, cheese, and milk. Based on this information, suggest a simple meal plan. Also include suggestions for when the user is tired.

[2003] In this way, appropriate suggestions based on the user's emotional state and efficient food management in physical stores can be realized.

[2004] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2005] Step 1:

[2006] A user uses a smartphone app to capture image data of the inside of a refrigerator. The user launches the app and takes a photo of the inside of the refrigerator with the smartphone camera. Once the photo is captured, the app prepares the image data.

[2007] Input: Photo data of the inside of a refrigerator taken with a smartphone camera.

[2008] Output: The image data is saved in the app on your smartphone.

[2009] Step 2:

[2010] The device sends the acquired image data to the cloud server, and the smartphone app uses an HTTP request to upload this image data to the cloud server.

[2011] Input: The acquired image data.

[2012] Output: The image data is transferred to the cloud server.

[2013] Step 3:

[2014] The server analyzes the image data on the cloud and recognizes food items. OpenCV is used to automatically identify food items from the image data and assign labels to each ingredient.

[2015] Input: Image data sent to the cloud server.

[2016] Output: A list of recognized food items.

[2017] Step 4:

[2018] The server searches a menu recipe database based on the recognized food items, uses the food list to search the database for related cooking recipes, and generates a list of found recipes.

[2019] Input: A list of recognized food items.

[2020] Output: A list of recipes.

[2021] Step 5:

[2022] The server sends a list of found recipes to the user's device. The cloud server prepares the recipe list and sends it to the smartphone app using an HTTP request.

[2023] Input: A list of recipes.

[2024] Output: The recipe list is displayed on the smartphone app.

[2025] Step 6:

[2026] A user scans the barcode of a food item at a physical store. The user activates the barcode scanner on their smartphone app and scans the barcode of the food item they want to purchase. The barcode data is sent by the app to a cloud server.

[2027] Input: Scanned barcode data.

[2028] Output: The barcode data is sent to the cloud server.

[2029] Step 7:

[2030] The server receives the barcode data and updates the refrigerator list accordingly, so that the newly added food item is reflected in the refrigerator list.

[2031] Input: Barcode data.

[2032] Output: Updated refrigerator list.

[2033] Step 8:

[2034] The user inputs their emotional state using the app. The user expresses their emotional state, such as "I'm tired today," through voice input or text input. The emotional data is sent to a cloud server.

[2035] Input: User's emotional state (audio or text data).

[2036] Output: Emotion data is sent to the cloud server.

[2037] Step 9:

[2038] The server uses an emotion engine to analyze the emotion data and estimate the user's emotional state. It uses the Hugging Face Transformers library to analyze voice tones and operational movements.

[2039] Input: Emotion data.

[2040] Output: Estimated emotional state.

[2041] Step 10:

[2042] The server then adjusts the menu suggestions it provides based on the user's estimated emotional state. For example, if the user is feeling stressed, it will prioritize easy-to-make recipes.

[2043] Input: Emotional state.

[2044] Output: Adjusted menu suggestions.

[2045] Step 11:

[2046] The server compares the list of ingredients with the list of recognized food items to identify missing ingredients, creates a shopping list based on the missing ingredients, and sends it to the user's device.

[2047] Input: Stock ingredient list and recognized food item list.

[2048] Output: A shopping list including missing ingredients is generated and sent to the user's device.

[2049] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2050] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2051] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2052] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2053] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2054] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2055] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2056] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2057] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2058] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2059] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2060] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2061] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2062] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2063] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2064] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2065] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2066] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2067] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2068] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2069] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2070] The following is further disclosed regarding the above embodiment.

[2071] (Claim 1)

[2072] means for acquiring image data of the inside of a refrigerator from a user terminal;

[2073] means for transmitting the acquired image data to a cloud server;

[2074] means for analyzing the image data and recognizing food items on a cloud server;

[2075] means for suggesting menu items based on the recognized food items;

[2076] means for transmitting the proposed menu to a user terminal;

[2077] A system including:

[2078] (Claim 2)

[2079] a means for storing a list of ingredients in stock designated by the user in a cloud server, comparing the food items recognized from the transmitted image data with the list of ingredients in stock to detect missing ingredients, and generating a shopping list including the missing ingredients;

[2080] means for transmitting the generated shopping list to a user terminal;

[2081] The system of claim 1 further comprising:

[2082] (Claim 3)

[2083] means for transmitting the generated shopping list from the user terminal using an external messaging service;

[2084] The system of claim 1 further comprising:

[2085] "Example 1"

[2086] (Claim 1)

[2087] means for acquiring image data from a user terminal;

[2088] means for transmitting the acquired image data over a network;

[2089] means for analyzing the image data and recognizing object items in the server;

[2090] means for providing information based on the recognized object items;

[2091] means for transmitting the provided information to a user terminal;

[2092] A system including:

[2093] (Claim 2)

[2094] a means for storing a reference list designated by a user in a server, comparing object items recognized from transmitted image data with the reference list to detect missing items, and generating a list including the missing items;

[2095] means for transmitting the generated list to a user terminal;

[2096] The system of claim 1 further comprising:

[2097] (Claim 3)

[2098] means for transmitting the generated list from the user terminal using an external communication service;

[2099] The system of claim 1 further comprising:

[2100] "Application Example 1"

[2101] (Claim 1)

[2102] means for acquiring image data of the inside of a refrigerator from a user terminal;

[2103] means for transmitting the acquired image data to a cloud server;

[2104] means for analyzing the image data and recognizing food items on a cloud server;

[2105] means for suggesting menu items based on the recognized food items;

[2106] means for transmitting the proposed menu to a user terminal;

[2107] A means to link with external services that provide suggested menus and arrange delivery,

[2108] A system including:

[2109] (Claim 2)

[2110] a means for storing a list of ingredients in stock designated by the user in a cloud server, comparing the food items recognized from the transmitted image data with the list of ingredients in stock to detect missing ingredients, and generating a shopping list including the missing ingredients;

[2111] a means for transmitting the generated shopping list to a delivery partner and arranging a delivery to replenish the missing materials;

[2112] means for transmitting the generated shopping list to a user terminal;

[2113] The system of claim 1 further comprising:

[2114] (Claim 3)

[2115] means for transmitting the generated shopping list from the user terminal using an external messaging service;

[2116] means for generating associated prompt sentences using a generative AI model;

[2117] The system of claim 1 further comprising:

[2118] "Example 2: Combining Emotion Engines"

[2119] Understood. I have rewritten the claims as follows:

[2120] (Claim 1)

[2121] means for acquiring image data of the inside of a refrigerator from a user terminal;

[2122] means for transmitting the acquired image data to a cloud server;

[2123] means for analyzing the image data and recognizing food items on a cloud server;

[2124] means for suggesting menu items based on the recognized food items;

[2125] means for transmitting the proposed menu to a user terminal;

[2126] means for analyzing a user's emotional state using an emotion analysis engine;

[2127] means for adjusting the menu suggestions based on the analyzed emotional state;

[2128] A system including:

[2129] (Claim 2)

[2130] a means for storing a list of ingredients in stock designated by the user in a cloud server, comparing the food items recognized from the transmitted image data with the list of ingredients in stock to detect missing ingredients, and generating a shopping list including the missing ingredients;

[2131] means for transmitting the generated shopping list to a user terminal;

[2132] means for analyzing an emotional state of the user using an emotion analysis engine and adjusting the shopping list based on the analyzed emotional state;

[2133] The system of claim 1 further comprising:

[2134] (Claim 3)

[2135] means for transmitting the generated shopping list from the user terminal using an external messaging service;

[2136] The system of claim 1 further comprising:

[2137] "Application example 2 when combining emotion engines"

[2138] (Claim 1)

[2139] means for acquiring image data of the inside of a refrigerator from a user terminal;

[2140] means for transmitting the acquired image data to a cloud server;

[2141] means for analyzing the image data and recognizing food items on a cloud server;

[2142] means for suggesting menu items based on the recognized food items;

[2143] means for transmitting the proposed menu to a user terminal;

[2144] A means for scanning the barcode of food items from a user terminal in a physical store and adding the data to a refrigerator list;

[2145] means for adjusting a menu based on the emotional state of a user;

[2146] a means for analyzing the user's voice data and operation actions using an emotion engine to estimate the user's emotional state;

[2147] A means for generating an optimal shopping list according to an emotional state;

[2148] A system including:

[2149] (Claim 2)

[2150] a means for storing a list of ingredients in stock designated by the user in a cloud server, comparing the food items recognized from the transmitted image data with the list of ingredients in stock to detect missing ingredients, and generating a shopping list including the missing ingredients;

[2151] means for transmitting the generated shopping list to a user terminal;

[2152] The system of claim 1 further comprising:

[2153] (Claim 3)

[2154] means for transmitting the generated shopping list from the user terminal using an external messaging service;

[2155] The system of claim 1 further comprising: [Explanation of symbols]

[2156] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for acquiring image data of the inside of a refrigerator from a user terminal; means for transmitting the acquired image data to a cloud server; means for analyzing the image data and recognizing food items on a cloud server; means for suggesting menu items based on the recognized food items; means for transmitting the proposed menu to a user terminal; A system including:

2. a means for storing a list of ingredients in stock designated by the user in a cloud server, comparing the food items recognized from the transmitted image data with the list of ingredients in stock to detect missing ingredients, and generating a shopping list including the missing ingredients; means for transmitting the generated shopping list to a user terminal; The system of claim 1 further comprising:

3. means for transmitting the generated shopping list from the user terminal using an external messaging service; The system of claim 1 further comprising:

Images