System

Abstract

A system is provided.SOLUTION: An image capturing unit configured to allow a user to capture an image of a food material, a transmission unit configured to transmit captured image data to a cloud server, an analysis unit configured to analyze the image data in the cloud server and identify a type and a volume of the food material, a storage unit configured to store food material information identified as a result of the analysis, an update unit configured to input an amount of the food material used by the user and update stock information based on the stored food material information, and a generation unit configured to generate a shopping list based on current stock information; Display means for displaying the generated shopping list to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, managing food inventory at home has become complicated for many people, and managing vegetables in particular has been difficult. This lack of management has led to frequent forgetting to use ingredients or duplicate purchases, resulting in increased food waste. It is also difficult to efficiently keep track of ingredients in the refrigerator and plan meals, placing a strain on household finances and impacting the environment. The present invention aims to solve these problems by providing a system that easily keeps track of ingredient types and volumes, efficiently manages purchasing and inventory, and suggests meals. [Means for solving the problem]

[0005] The present invention solves this problem by providing a system that includes the following means: a photographing means for a user to photograph ingredients; a transmission means for transmitting the photographed image data to a cloud server; an analysis means for analyzing the image data on the cloud server to identify the type and quantity of ingredients; a storage means for saving the ingredient information identified as a result of the analysis; an update means for updating inventory information by inputting the amount of ingredients used by the user based on the saved ingredient information; a generation means for generating a shopping list based on current inventory information; and a display means for displaying the generated shopping list to the user. The system also includes a suggestion means for generating a list of suggested dishes based on current inventory information, and a missing ingredient generation means for comparing ingredients needed for a dish selected by the user with current inventory information and generating a list of ingredients that are missing. This invention enables efficient ingredient management and planned recipe suggestions, thereby reducing food waste.

[0006] "Ingredients" is a general term for fresh foods and processed foods used in cooking and preparation.

[0007] "Photographing means" refers to a device or function that allows a user to capture an image of an ingredient, such as the camera function of a smartphone.

[0008] "Transmission means" refers to the functions and protocols for transmitting acquired image data to a cloud server.

[0009] "Cloud server" is a general term for remote servers that provide data storage and processing power over the Internet.

[0010] "Analysis means" refers to the system processes and modules that use AI technology and algorithms to identify the type and volume of ingredients based on the transmitted image data.

[0011] "Storage means" refers to the function or method for storing the ingredient information identified as a result of the analysis in a database or other storage.

[0012] "Update means" refers to the functions and processes for keeping inventory data up to date based on usage information entered by the user.

[0013] "Generation means" refers to a system function or module for creating a shopping list or cooking list based on current inventory information.

[0014] "Display means" refers to a device or interface for visually presenting the generated shopping list or cooking list to the user.

[0015] "Suggestion means" refers to a system function or module that suggests a list of dishes to the user based on current inventory information.

[0016] The "means for generating missing ingredients" refers to the function or process for comparing the ingredients required for the dish selected by the user with current inventory information and generating a list of missing ingredients. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] overview

[0039] This invention is a system that streamlines food inventory management within the home and prevents forgetting to use ingredients or purchasing duplicates. Users take photos of purchased ingredients with their smartphones and send the images to a cloud server, where AI identifies and records the type and volume of ingredients. The data is stored in the cloud and used for inventory management, creating shopping lists, and suggesting recipes.

[0040] Program processing overview

[0041] Food Recognition and Preservation

[0042] User: Takes a photo of the ingredients he / she has purchased using his / her smartphone camera.

[0043] Terminal: Sends captured image data to a cloud server.

[0044] Server: Analyzes the received image data using AI and recognizes the type and volume of ingredients.

[0045] Server: Saves the analysis results in a database.

[0046] Example: A user purchases one cabbage, three carrots, and 500g of chicken, and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database.

[0047] Inventory management

[0048] User: Enters the amount of ingredients used for dinner into the app.

[0049] Terminal: Sends the entered usage information to the cloud server.

[0050] Server: Updates the inventory information in the database.

[0051] Example: A user makes a cabbage salad and uses 1 / 2 a cabbage. The user enters "1 / 2 a cabbage used" into the app, and the server updates the cabbage inventory in the database to "1 / 2 a cabbage."

[0052] Shopping suggestions

[0053] User: Launches the app and creates a shopping list.

[0054] Terminal: Retrieves current inventory information from the cloud.

[0055] Server: Generates a list of dishes that can be suggested based on inventory information and sends it to the device.

[0056] Terminal: Shows the user a list of suggested dishes.

[0057] User: Select the dish they want to make from the suggested dishes.

[0058] Server: Compares the ingredients required for the selected dish with the current inventory information and generates a list of ingredients that are missing.

[0059] Terminal: Displays a list of missing ingredients and allows the user to review and edit the shopping list.

[0060] Example: A user launches an app, and the app retrieves inventory information for "1 / 2 cabbage," "3 carrots," and "500g chicken" from the server. Based on this inventory, the app suggests "stir-fried cabbage and chicken," which the user selects. The server compares the required ingredients (1 cabbage, 300g chicken) with the inventory, and adds the missing 1 / 2 cabbage to the list. The app displays this list to the user, who can then confirm and edit the shopping list.

[0061] Implementation flow

[0062] To implement this system, users use a smartphone app to take a photo of the ingredients they have purchased and send the image to a cloud server via the app. The cloud server then analyzes the image, identifies the type and quantity of ingredients, and stores the information in a database. The app updates inventory information in real time and also reflects the amount of ingredients entered by the user. When shopping, the app also displays suggested dishes based on current inventory information and automatically generates a list of ingredients that are in short supply.

[0063] This allows users to efficiently manage ingredients and receive planned cooking suggestions, thereby reducing food waste and enabling efficient household budget management.

[0064] The processing flow will be explained below.

[0065] Food Recognition and Preservation

[0066] Step 1:

[0067] The user takes a photo of the purchased ingredients using the smartphone camera.

[0068] Step 2:

[0069] The device saves the captured image data and calls an API to send it to the cloud server.

[0070] Step 3:

[0071] The server temporarily stores the received image data and passes the data to an AI model for image analysis.

[0072] Step 4:

[0073] The server uses an AI model to analyze the image data and identify the type and quantity of ingredients, for example, "1 cabbage," "3 carrots," and "500g of chicken."

[0074] Step 5:

[0075] The server saves the analysis results in a database and records them as "1 cabbage," "3 carrots," and "500g of chicken."

[0076] Inventory management

[0077] Step 6:

[0078] The user enters the amount of ingredients used for dinner into the app.

[0079] Step 7:

[0080] The device calls an API to send the entered usage information to the cloud server.

[0081] Step 8:

[0082] The server temporarily stores the received usage data and updates the inventory information in the database.

[0083] Step 9:

[0084] The server reduces the inventory of specific ingredients in the database and updates it to "1 / 2 cabbage," "3 carrots," and "500g chicken."

[0085] Shopping suggestions

[0086] Step 10:

[0087] A user launches the app and checks current inventory information to create a shopping list.

[0088] Step 11:

[0089] The device requests current inventory information from the cloud server.

[0090] Step 12:

[0091] The server retrieves current inventory information from the database and sends it to the terminal.

[0092] Step 13:

[0093] Based on the inventory information received by the terminal, a list of dishes that can be suggested to the user is generated and displayed.

[0094] Step 14:

[0095] The user selects the dish they want to make from the suggested dishes.

[0096] Step 15:

[0097] The server compares all ingredients required for the selected dish with current inventory information and generates a list of ingredients that are in short supply.

[0098] Step 16:

[0099] The server transmits a list of ingredients that are in short supply to the terminal, which then displays it to the user.

[0100] Step 17:

[0101] The user checks the displayed list of ingredients that are in short supply and creates and edits a shopping list.

[0102] Specific examples of implementation

[0103] Step 18:

[0104] After shopping, the user purchases one cabbage, three carrots, and 500g of chicken and takes a photo of it with their smartphone.

[0105] Step 19:

[0106] The device sends the image to the server, which analyzes it and records it in a database ("1 cabbage," "3 carrots," "500g chicken").

[0107] Step 20:

[0108] A user eats 1 / 2 a cabbage for dinner and enters that information into the app.

[0109] Step 21:

[0110] The terminal sends the input information to the server, and the server updates the inventory to "1 / 2 cabbage."

[0111] Step 22:

[0112] Before the next shopping trip, the user launches the app, which retrieves the current inventory information. ("1 / 2 cabbage")

[0113] Step 23:

[0114] The server sends inventory information for "1 / 2 cabbage," "3 carrots," and "500g of chicken" to the terminal, and the terminal then suggests dishes based on that information.

[0115] Step 24:

[0116] The user selects "Stir-fried cabbage and chicken" and adds "1 / 2 cabbage" to the list, for which the server is short.

[0117] Step 25:

[0118] The device displays a list of ingredients that are in short supply, and the user can check and edit the shopping list.

[0119] These are the specific processing steps, from recognizing and storing ingredients to inventory management and suggestions when shopping.

[0120] Example 1

[0121] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0122] In most cases, food inventory management in the home is done manually, resulting in frequent forgetting to use ingredients or duplicate purchases. Furthermore, determining what to buy next requires knowing the current inventory status and recipes for the necessary dishes, which is time-consuming. Furthermore, there are often times when people are out of ideas for cooking, so an efficient way to solve these problems is needed.

[0123] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0124] In this invention, the server includes: [a photographing means for the user to photograph ingredients;] [a transmitting means for transmitting the photographed image data to a network server;] [an analyzing means for analyzing the image data in the network server and identifying the type and amount of ingredients;] [a storing means for saving the ingredient information identified as a result of the analysis;] [an updating means for inputting the amount of ingredients used by the user based on the stored ingredient information and updating inventory information;] [a generating means for generating a shopping list based on current inventory information;] [a display means for displaying the generated shopping list to the user;] [an automatic output means for generating a list of suggested dishes; and] [a list generating means for generating a list of missing ingredients based on the suggested dishes. This enables ingredient inventory management, prevention of forgotten ingredients and duplicate purchases, efficient shopping list creation, and dish suggestions.

[0125] The term "photography means for a user to photograph ingredients" refers to a device or application that a user uses to record the ingredients they have purchased as digital images.

[0126] The "transmission means for transmitting the photographed image data to the network server" refers to a device or application for transmitting the photographed image data of ingredients to a server on the cloud via the Internet.

[0127] "Analysis means for analyzing image data on a network server and identifying the type and volume of ingredients" refers to software or algorithms for analyzing received image data and automatically identifying the type and volume of ingredients.

[0128] "Storage means for storing information on ingredients identified as a result of the analysis" refers to a device or application for recording data on the type and volume of ingredients obtained through the analysis in storage such as a database.

[0129] "Means for updating inventory information by inputting the amount of ingredients used by the user based on the stored ingredient information" refers to a device or application for correcting and updating inventory data in real time based on the amount of ingredients used input by the user through the application.

[0130] "Means for generating a shopping list based on current inventory information" refers to a device or application that references recorded inventory information and automatically creates a list of new ingredients to be purchased.

[0131] The "display means for displaying the generated shopping list to the user" refers to a device or application for visually displaying the automatically generated shopping list on the display of the user's device, for example.

[0132] "Automatic output means for generating a list of suggested dishes" refers to a device or application that automatically generates and presents a list of dishes that can be made based on stored inventory information and data such as user preferences.

[0133] "List generation means for generating a list of missing ingredients based on a proposed dish" refers to a device or application that compares ingredients required for the proposed dish with inventory data and lists missing ingredients.

[0134] MODE FOR CARRYING OUT THE INVENTION

[0135] This invention is a system designed to improve the efficiency of food ingredient inventory management in the home, prevent forgetting to use ingredients or purchasing the same ingredients twice, and provide recipe suggestions based on a well-planned recipe. Specific embodiments of this system are described below.

[0136] A user uses a smartphone to take a photo of the ingredients they have purchased. The smartphone's camera (photography means) is used to capture high-resolution images of the ingredients. For example, imagine a case where a user takes a photo of one cabbage, three carrots, and 500g of chicken upon returning home from the supermarket. The captured image data is sent to a network server on the cloud via an application on the smartphone (transmission means).

[0137] The network server analyzes the received image data. Specifically, it uses image recognition services such as Google® Cloud Vision API and Amazon Rekognition to identify the type and quantity of ingredients in the image (analysis means). The analysis results are stored in a cloud database (e.g., GOOGLE FI® rebase or Amazon DynamoDB) (storage means).

[0138] When a user uses an ingredient, they input the amount used into the smartphone application. For example, if they input "1 / 2 cabbage used," the application sends this information to the cloud server, and the inventory information in the cloud database is updated (update means).

[0139] Furthermore, when a user launches the app to go shopping, the application retrieves current inventory information from the cloud. A shopping list is automatically generated based on the current inventory information (generation means). The generated shopping list is displayed to the user, allowing the user to use the list as a reference when shopping (display means).

[0140] The system also has the ability to suggest dishes using a generative AI model. For example, it uses a language model such as OpenAI's GPT-3 (registered trademark) to generate a list of suggested dishes based on inventory information and display it to the user (automatic output means). When the user selects one of the suggested dishes, the system compares the ingredients needed for that dish with current inventory information and automatically generates a list of ingredients that are in short supply (list generation means).

[0141] The above process allows users to efficiently manage ingredients and plan their shopping and cooking, thereby reducing food waste and enabling efficient management of household finances.

[0142] For example, suppose the user enters the prompt text as follows:

[0143] "I have half a cabbage, three carrots, and 500g of chicken left in my fridge. What kind of dish can I make with these ingredients? Can you also tell me how to make it?"

[0144] Based on these prompts, the generative AI model will suggest optimal dishes and cooking methods to the user, allowing them to expand the range of cooking they can do at home.

[0145] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0146] Program processing flow

[0147] Step 1:

[0148] The user takes a photo of the ingredients

[0149] Specific operation: The user uses the smartphone camera to take a picture of the ingredients they purchased. For example, they take a picture of one cabbage, three carrots, and 500g of chicken lined up.

[0150] Input: An image of the food captured by the camera.

[0151] Output: Image data of ingredients stored on the smartphone.

[0152] Step 2:

[0153] The device sends the image to the cloud server.

[0154] Specific operation: A smartphone app sends captured image data to a cloud server via the Internet. For example, the app uploads the images to AWS (registered trademark) S3.

[0155] Input: Image data of ingredients stored on your smartphone.

[0156] Output: Image data sent to the cloud server.

[0157] Step 3:

[0158] The server analyzes the image and recognizes the type and volume of ingredients.

[0159] Specific operation: The cloud server analyzes the received image data using Google Cloud Vision API and Amazon Rekognition to identify the type and quantity of ingredients. For example, it might recognize it as one cabbage, three carrots, and 500g of chicken.

[0160] Input: Image data stored on a cloud server.

[0161] Output: Analysis results including ingredient type and volume.

[0162] Step 4:

[0163] The server stores the analysis results in a database

[0164] Specific operation: The food ingredient information from the analysis results is saved in a cloud database (e.g., Google Firebase or Amazon DynamoDB).

[0165] Input: Analysis results including ingredient type and volume.

[0166] Output: Ingredient information recorded in a cloud database.

[0167] Step 5:

[0168] The user enters the amount of ingredients used into the app.

[0169] Specific operation: The user inputs the amount of ingredients used for dinner into the smartphone app. For example, the user inputs that half a cabbage was used.

[0170] Input: Amount of ingredients used.

[0171] Output: Usage information entered into the smartphone app.

[0172] Step 6:

[0173] The device sends usage information to the cloud server.

[0174] Specific operation: The smartphone app sends the entered usage information to a cloud server. For example, the app sends usage data to Firebase Firestore.

[0175] Input: Usage information stored in the smartphone app.

[0176] Output: Usage information sent to the cloud server.

[0177] Step 7:

[0178] The server updates the inventory information in the database

[0179] Specific operation: The cloud server updates the inventory information in the cloud database in real time based on the received usage information. For example, the inventory of cabbage is updated to 1 / 2.

[0180] Input: Usage information sent to the cloud server.

[0181] Output: Updated inventory information from the cloud database.

[0182] Step 8:

[0183] The user launches the app and creates a shopping list

[0184] Specific action: The user launches the smartphone app and accesses the screen for creating a shopping list.

[0185] Input: The app launch operation.

[0186] Output: Display of the app's shopping list creation screen.

[0187] Step 9:

[0188] The device retrieves current inventory information from the cloud.

[0189] Specific operation: The smartphone app accesses a cloud database to get the latest inventory information, for example, from Firebase Firestore.

[0190] Input: A retrieval request to the cloud database.

[0191] Output: The latest inventory information obtained.

[0192] Step 10:

[0193] The server generates recipe suggestions based on inventory information and sends them to the device.

[0194] Specific operation: The cloud server uses the acquired inventory information to generate a list of dishes that are best suited to the user using a language model such as GPT-3, and sends this list to the smartphone app.

[0195] Input: Latest inventory information.

[0196] Output: A list of suggested dishes.

[0197] Step 11:

[0198] The device displays suggested dishes

[0199] Specific operation: The smartphone app displays the list of suggested dishes received from the server to the user. For example, it displays a list including "stir-fried cabbage and chicken."

[0200] Input: A list of dish suggestions received from the server.

[0201] Output: A list of cooking suggestions displayed on a smartphone app.

[0202] Step 12:

[0203] The user selects the dish they want to make.

[0204] Specific operation: The user selects the dish they want to make from the list of suggested dishes. For example, they select "Stir-fried cabbage and chicken."

[0205] Input: User's food selection.

[0206] Output: Selected dish information.

[0207] Step 13:

[0208] The server compares the required ingredients with the available stock and generates a list of ingredients that are in short supply.

[0209] Specific operation: The cloud server compares the list of ingredients needed for the selected dish with the current inventory information and lists any ingredients that are missing. For example, it recognizes that 1 / 2 a cabbage is missing.

[0210] Input: Selected dish information and current inventory information.

[0211] Output: A list of missing ingredients.

[0212] Step 14:

[0213] The device displays a list of ingredients that are missing, and the user can check and edit the shopping list.

[0214] Specific operation: The smartphone app displays the list of ingredients that are in short supply received from the server to the user, who then checks and edits the list.

[0215] Input: Missing ingredient list.

[0216] Output: A list of ingredients that are in short supply, displayed on the smartphone app. Users can confirm and edit the list.

[0217] (Application example 1)

[0218] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0219] Inventory management of ingredients and products at home or in a brick-and-mortar store is complicated, and manual management can easily lead to forgetting to use ingredients or purchasing duplicates. Furthermore, if product inventory is not managed properly, there is a high risk of unsold items or shortages. This increases food waste and opportunity losses, hindering efficient operations.

[0220] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0221] In this invention, the server includes [a photographing means for the user to photograph ingredients], [a transmission means for transmitting the photographed image data to the cloud server], [an analysis means for analyzing the image data in the cloud server and identifying the type and amount of ingredients], [a storage means for saving the ingredient information identified as a result of the analysis], [an update means for inputting the amount of ingredients used by the user based on the saved ingredient information and updating inventory information], [a generation means for generating a shopping list based on current inventory information], [a display means for displaying the generated shopping list to the user], [a suggestion means for suggesting products to the user in conjunction with inventory], and [a detection means for detecting missing ingredients or products]. This makes it possible to automatically manage ingredient and product inventory and make appropriate suggestions to the user.

[0222] "Photographing means" refers to a device or method that allows a user to photograph ingredients or products.

[0223] The "transmission means" is a device or method for transmitting captured image data to a cloud server.

[0224] The "analysis means" is a device or method for analyzing image data on a cloud server and identifying the type and volume of ingredients or products.

[0225] The "storage means" refers to a device or method for storing the ingredients and product information identified as analysis results in a database.

[0226] The "update means" is a device or method for inputting the amount of ingredients or products used by the user based on the stored ingredient and product information and updating the inventory information.

[0227] The "generation means" is a device or method for generating a shopping list or a recommended product list based on current inventory information.

[0228] The "display means" refers to a device or method for displaying the generated shopping list or recommended product list to the user.

[0229] The "suggestion means" is a device or method for suggesting products to users in conjunction with inventory.

[0230] "Detection means" refers to a device or method for detecting missing ingredients or products.

[0231] Overall system configuration

[0232] The system of this invention is broadly composed of a photographing means, a transmitting means, a analyzing means, a saving means, a updating means, a generating means, a displaying means, a suggesting means, and a detecting means. These means work together to realize inventory management of ingredients and products and shopping support based on that inventory management.

[0233] Program Overview

[0234] Filming method

[0235] The user uses a smartphone to take a photo of the ingredients or products they have purchased, and the device captures the image using the smartphone's camera.

[0236] Transmission method

[0237] The captured image data is sent to a cloud server. This device or software uses the smartphone's communication functions (Wi-Fi, mobile data communication, etc.).

[0238] Analysis means

[0239] The cloud server analyzes the captured image data using an AI model (for example, a deep learning model such as TENSORFLOW®) to identify the type and volume of specific ingredients or products. This analysis uses image recognition technology.

[0240] Preservation means

[0241] The analyzed ingredient and product information is stored in a cloud database (such as AWS RDS or Google Cloud Firestore). The type, size, and photo date and time of each ingredient or product are recorded in the database.

[0242] Update method

[0243] Users input the quantities of ingredients and products used into the application, and the information is sent to a cloud server, which updates the inventory information in a database based on the input.

[0244] generation means

[0245] Based on the current inventory information, the cloud server generates a shopping list and a list of recommended products, allowing users to efficiently create lists of ingredients and products they need.

[0246] Display means

[0247] The generated shopping list and recommended product list are displayed to the user through a smartphone application. The user interface can be built using a cross-platform framework such as React Native.

[0248] Proposal means

[0249] The server connects with current inventory information and suggests suitable products and recipes to users based on the latest inventory information and the user's past purchase history.

[0250] Detection Method

[0251] Every time inventory data is updated, the cloud server compares it with the existing inventory data to detect any missing ingredients or products, which are then automatically added to the user's shopping list.

[0252] Specific examples

[0253] For example, when a store employee places new tomatoes on a shelf, they take a picture of the tomato using the app, which then uploads the image to the cloud. An AI model on the cloud server analyzes the image, identifies the type and quantity of tomatoes, and stores the information in a database. When a customer opens the app, fresh tomatoes are recommended based on the latest inventory information.

[0254] Prompt Sentence Examples

[0255] "Please upload images of newly arrived tomatoes to the cloud server. The AI ​​on the server side will identify the quantity and quality of the tomatoes, update the inventory information, and reflect it as recommendations on the customer app."

[0256] As a result, the system of the present invention can improve the efficiency of inventory management of food ingredients and products at home or in a physical store, and can support customers in making appropriate product suggestions and generating shopping lists.

[0257] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0258] Step 1:

[0259] The user takes a photo of the food or product using a smartphone.

[0260] Input: Image of ingredients or products

[0261] Output: Image data stored on a smartphone

[0262] Specific actions: The user launches the smartphone camera app, frames the food or product they are purchasing, and presses the shutter button. When doing so, they take as clear an image as possible, taking care to ensure that the food or product is clearly identifiable.

[0263] Step 2:

[0264] The device sends the captured image data to a cloud server.

[0265] Input: Captured image data

[0266] Output: Image data transferred to the cloud server

[0267] What happens: The smartphone app compresses the captured image file and sends an HTTP POST request to a cloud server over the internet, including user authentication information.

[0268] Step 3:

[0269] The server receives the image data at the cloud server and analyzes it using the analysis means.

[0270] Input: Image data transferred to the cloud server

[0271] Output: Analyzed ingredients and product types and volume data

[0272] How it works: The server stores the received image data on disk and performs image recognition using an AI model (e.g., TensorFlow). This AI model identifies the type and volume of ingredients or products from the image and extracts this information as metadata.

[0273] Step 4:

[0274] The server stores the analysis results in a database.

[0275] Input: Analyzed ingredient and product type and volume data

[0276] Output: Analysis result data stored in a database

[0277] Specific operation: The server converts the analysis results into structured data and executes an INSERT query to a cloud database (e.g., AWS RDS, Google Cloud Firestore). The stored data includes the type of ingredients or products, volume, analysis date and time, etc.

[0278] Step 5:

[0279] The user enters the amounts of ingredients and products used into the application.

[0280] Input: Usage information

[0281] Output: Usage data transferred to the cloud server

[0282] Specific operation: The user opens a form on the smartphone application to input the amount of ingredients and products used, enters the necessary information, for example, "1 / 2 cabbage used," and presses the send button. This information is sent to the cloud server.

[0283] Step 6:

[0284] The server updates the inventory information in the database based on the input usage data.

[0285] Input: Usage data transferred to the cloud server

[0286] Output: Updated inventory information

[0287] Specific operation: The server analyzes the received usage data and reduces the corresponding inventory data. It updates the inventory information in the database using an UPDATE query. The updated inventory information reflects the remaining amount of the ingredient or product.

[0288] Step 7:

[0289] The server generates shopping lists and recommended product lists based on current inventory information.

[0290] Input: Updated inventory information

[0291] Output: Generated shopping list and recommended products list

[0292] Specific operation: The server identifies ingredients and products that are in short supply based on the latest inventory data, creates a list of items to purchase, and adds recommended products and recipes to the list, taking into account the user's past purchase history and trend data.

[0293] Step 8:

[0294] The terminal displays the generated shopping list and recommended product list to the user.

[0295] Input: Generated shopping list or recommended product list

[0296] Output: List displayed on smartphone

[0297] What it does: The smartphone application analyzes the list data received from the server and displays it in an easy-to-read format on the user interface, allowing the user to plan their shopping needs.

[0298] Step 9:

[0299] The server works in conjunction with inventory information to suggest products to users.

[0300] Input: Current inventory information, user's past purchase history

[0301] Output: Suggested products and recipe information

[0302] What it does: The server cross-references the inventory database with the user's purchasing history to generate product and recipe recommendations that are best suited to the user. These recommendations are listed in order of most relevant and notified to the user.

[0303] Step 10:

[0304] The server detects ingredients or products that are in short supply and adds them to the shopping list.

[0305] Input: Current inventory information

[0306] Output: Updated shopping list

[0307] What it does: The server periodically scans inventory data to identify ingredients or products that are running low, automatically adding them to the shopping list and notifying the user.

[0308] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0309] overview

[0310] This invention is a system that combines an ingredient management system with an emotion engine that recognizes the user's emotions, making recipe suggestions and optimizing the shopping list according to the user's emotions. In addition to purchasing and inventory management and recipe suggestions, this system is characterized by recognizing the user's emotions with the emotion engine and making suggestions based on the user's emotions.

[0311] Program processing overview

[0312] Food Recognition and Preservation

[0313] User: Takes a photo of the purchased ingredients using the smartphone camera.

[0314] Terminal: Sends captured image data to a cloud server.

[0315] Server: Analyzes the received image data using AI and recognizes the type and volume of ingredients.

[0316] Server: Saves the analysis results in a database.

[0317] Example: A user purchases one cabbage, three carrots, and 500g of chicken, and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database.

[0318] Inventory management

[0319] User: Enters the amount of ingredients used for dinner into the app.

[0320] Terminal: Sends the entered usage information to the cloud server.

[0321] Server: Updates the inventory information in the database.

[0322] Example: A user makes a cabbage salad and uses 1 / 2 a cabbage. The user enters "1 / 2 a cabbage used" into the app, and the server updates the cabbage inventory in the database to "1 / 2 a cabbage."

[0323] Emotion Recognition and Suggestions

[0324] User: Accesses the emotion engine through the app and inputs their emotional state for the day.

[0325] Terminal: Sends the user's emotional data to the cloud server.

[0326] Server: The emotion engine analyzes the user's emotions and suggests dishes and ingredients based on those emotions.

[0327] For example, if the user is tired, easy meals will be suggested, and if the user is energetic, more challenging meals will be suggested.

[0328] Shopping suggestions

[0329] User: Launches the app and checks current inventory information to create a shopping list.

[0330] Terminal: Requests current inventory information from the cloud server.

[0331] Server: Generates a list of dishes that can be suggested based on inventory information and sends it to the device.

[0332] Terminal: Shows the user a list of suggested dishes.

[0333] User: Select the dish they want to make from the suggested dishes.

[0334] Server: Compares all ingredients required for the selected dish with current inventory and generates a list of ingredients that are missing.

[0335] Terminal: Displays a list of missing ingredients and allows the user to review and edit the shopping list.

[0336] Example: A user launches the app and retrieves current inventory information (1 / 2 cabbage, 3 carrots, 500g chicken). The emotion engine recognizes that the user is tired and suggests a simple "stir-fried cabbage and chicken." The server adds the missing 1 / 2 cabbage to the list, and the device displays the shopping list.

[0337] Specific examples of implementation

[0338] After shopping, a user purchases 1 cabbage, 3 carrots, and 500g of chicken and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database (1 cabbage, 3 carrots, 500g of chicken). The user eats 1 / 2 a cabbage for dinner and enters that information into the app. The device sends the input information to the server, which updates the inventory to "1 / 2 a cabbage."

[0339] Next, the user launches the app and checks the current inventory information (1 / 2 cabbage, 3 carrots, 500g of chicken) to create a shopping list. The emotion engine recognizes the user's emotion (e.g., tiredness) and suggests an easy-to-make dish, "Stir-fried cabbage and chicken." The server compares the required ingredients (1 cabbage, 300g of chicken) with the inventory and adds the missing 1 / 2 cabbage to the list. The device displays the list of missing ingredients, and the user can check and edit the shopping list.

[0340] In this way, by combining emotion engines, it becomes possible to make optimal suggestions based on the user's emotions, enabling more personalized ingredient management and cooking suggestions.

[0341] The processing flow will be explained below.

[0342] Processing flow of a system that combines emotion engines

[0343] Emotion Recognition and Suggestions

[0344] Step 1:

[0345] The user launches the app and enters their emotional state for the day.

[0346] Step 2:

[0347] The device transmits the user's emotional data to a cloud server.

[0348] Step 3:

[0349] The server passes the received emotion data to the emotion engine, which analyzes the user's emotional state.

[0350] Step 4:

[0351] The server generates a list of dishes according to the emotion based on the analysis results of the emotion engine.

[0352] Food Recognition and Preservation

[0353] Step 5:

[0354] The user takes a photo of the purchased ingredients using the smartphone camera.

[0355] Step 6:

[0356] The device saves the captured image data and calls an API to send it to the cloud server.

[0357] Step 7:

[0358] The server temporarily stores the received image data and passes the data to an AI model for image analysis.

[0359] Step 8:

[0360] The server uses an AI model to analyze the image data and identify the type and quantity of ingredients, for example, "1 cabbage," "3 carrots," and "500g of chicken."

[0361] Step 9:

[0362] The server saves the analysis results in a database and records them as "1 cabbage," "3 carrots," and "500g of chicken."

[0363] Inventory management

[0364] Step 10:

[0365] The user enters the amount of ingredients used for dinner into the app.

[0366] Step 11:

[0367] The device calls an API to send the entered usage information to the cloud server.

[0368] Step 12:

[0369] The server temporarily stores the received usage data and updates the inventory information in the database.

[0370] Step 13:

[0371] The server reduces the inventory of specific ingredients in the database and updates it to "1 / 2 cabbage," "3 carrots," and "500g chicken."

[0372] Shopping suggestions

[0373] Step 14:

[0374] A user launches the app and checks current inventory information to create a shopping list.

[0375] Step 15:

[0376] The device requests current inventory information from the cloud server.

[0377] Step 16:

[0378] The server retrieves current inventory information from the database and sends it to the terminal.

[0379] Step 17:

[0380] Based on the inventory information received by the terminal, a list of dishes that can be suggested to the user is generated and displayed.

[0381] Step 18:

[0382] The user selects the dish they want to make from the suggested dishes.

[0383] Step 19:

[0384] The server compares all ingredients required for the selected dish with current inventory information and generates a list of ingredients that are in short supply.

[0385] Step 20:

[0386] The server transmits a list of ingredients that are in short supply to the terminal, which then displays it to the user.

[0387] Step 21:

[0388] The user checks the displayed list of ingredients that are in short supply and creates and edits a shopping list.

[0389] Specific examples of implementation

[0390] Step 22:

[0391] After shopping, the user purchases one cabbage, three carrots, and 500g of chicken and takes a photo of it with their smartphone.

[0392] Step 23:

[0393] The device sends the image to the server, which analyzes it and records it in a database ("1 cabbage," "3 carrots," "500g chicken").

[0394] Step 24:

[0395] A user eats 1 / 2 a cabbage for dinner and enters that information into the app.

[0396] Step 25:

[0397] The terminal sends the input information to the server, and the server updates the inventory to "1 / 2 cabbage."

[0398] Step 26:

[0399] The user launches the app before their next shopping trip, and the app retrieves the current inventory information ("1 / 2 cabbage," "3 carrots," and "500g chicken").

[0400] Step 27:

[0401] The device inputs the user's emotional state into a cloud server, which then analyzes the data.

[0402] Step 28:

[0403] The emotion engine recognizes that the user is tired and suggests an easy-to-make dish: stir-fried cabbage and chicken.

[0404] Step 29:

[0405] The server compares the ingredients needed ("1 cabbage" and "300g chicken") with the inventory and adds the missing 1 / 2 cabbage to the list.

[0406] Step 30:

[0407] The device displays a list of ingredients that are in short supply, and the user can check and edit the shopping list.

[0408] The above is the specific processing flow of a system that combines an emotion engine. By taking the user's emotions into consideration, more personalized suggestions become possible, which is expected to improve user satisfaction.

[0409] Example 2

[0410] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0411] In today's busy lifestyles, it is difficult for users to efficiently manage ingredient inventory, select dishes, and receive recipe suggestions tailored to their mood of the day. Conventional systems require users to manually update inventory information and manage ingredient usage, which is tedious and often inaccurate. Furthermore, they lack the ability to suggest dishes based on the user's emotions, resulting in a non-personalized user experience. Therefore, there is a need for a system that allows users to efficiently manage ingredients, inventory, recognize emotions, and receive optimal recipe suggestions.

[0412] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0413] In this invention, the server includes: [a photographing means for the user to photograph ingredients;] [a transmission means for sending the photographed image data to the cloud server;] [an analysis means for analyzing the image data in the cloud server and identifying the type and amount of ingredients;] [a storage means for saving ingredient information identified as a result of the analysis;] [an update means for inputting the amount of ingredients used by the user based on the saved ingredient information and updating inventory information;] [an emotion recognition and suggestion means for recognizing the user's emotions and suggesting dishes and ingredients based on the emotions;] [a notification means for notifying the user of the suggestions generated based on the emotions;] [a generation means for generating a shopping list based on current inventory information;] [a display means for displaying the generated shopping list to the user; and] [an emotion input means for inputting the user's emotional state and sending it to the cloud server. This enables users to efficiently manage ingredients and enables personalized dish suggestions based on the user's emotions.

[0414] "Photographing means" refers to the equipment or method used by the user to photograph the ingredients they have purchased.

[0415] "Transmission means" refers to the equipment or method used to transmit captured image data to the cloud server.

[0416] The "analysis means" refers to the equipment or method used to analyze the image data sent to the cloud server and identify the type and volume of ingredients.

[0417] "Storage means" refers to the equipment or method used to store the ingredient information identified as the analysis result in a database or the like.

[0418] The "update means" refers to a device or method used to input the amount of ingredients used by the user based on the stored ingredient information and update the inventory information.

[0419] "Emotion recognition and suggestion means" refers to devices and methods used to recognize a user's emotions and suggest dishes and ingredients based on those emotions.

[0420] "Notification means" refers to the device or method used to notify the user of suggestions generated based on emotions.

[0421] A "generator" is a device or method used to generate a shopping list based on current inventory information.

[0422] "Display means" refers to the device or method used to display the generated shopping list to the user.

[0423] "Emotion input means" refers to the device or method used to input the user's emotional state and transmit it to the cloud server.

[0424] MODE FOR CARRYING OUT THE INVENTION

[0425] This invention is a system that combines an ingredient management system with an emotion engine that recognizes the user's emotions, making recipe suggestions and optimizing the shopping list according to the user's emotions. In addition to purchasing and inventory management and recipe suggestions, this system is characterized by recognizing the user's emotions with the emotion engine and making suggestions based on the user's emotions.

[0426] Food Recognition and Preservation

[0427] User: Take a photo of the ingredients they purchased using their smartphone camera. The user launches the smartphone app, opens the camera function, and takes a photo of the ingredients. For example, they take a photo of one cabbage, three carrots, and 500g of chicken.

[0428] Device: Sends the captured image data to the cloud server. When the user taps the "Send" button, the image data is uploaded to the cloud server via the Internet.

[0429] Server: The received image data is analyzed using AI to identify the type and quantity of ingredients. The server uses the Google Cloud Vision API to analyze the image and extract information such as "1 cabbage, 3 carrots, 500g of chicken."

[0430] Server: Saves the analysis results in a database. The server saves the acquired ingredient information in a MySQL (registered trademark) database and reflects it in each user's inventory list.

[0431] Inventory management

[0432] User: Enter the amount of ingredients used for dinner into the app. The user opens the smartphone app and enters the amount of ingredients used (for example, "1 / 2 head of cabbage used").

[0433] Device: Sends the entered usage information to the cloud server. When the user taps the "Send" button, the usage information is sent to the cloud server.

[0434] Server: Updates the inventory information in the database. The server updates the database and changes the amount of cabbage in the inventory list to "1 / 2 piece."

[0435] Emotion Recognition and Suggestions

[0436] User: Accesses the emotion engine through the app and inputs the emotional state of the day. The user opens the smartphone app, accesses the emotion engine, and inputs the emotion of the day (e.g., "tired").

[0437] Device: Sends the user's emotional data to the cloud server. When the user taps the "Send" button, the emotional data is sent to the cloud server.

[0438] Server: The emotion engine analyzes the user's emotions and suggests dishes and ingredients based on those emotions. The server uses the Microsoft® Azure® Emotion API to analyze emotions and determines that "a simple dish should be suggested," suggesting, for example, "stir-fried cabbage and chicken."

[0439] Shopping suggestions

[0440] User: Launches the app and checks current inventory information to create a shopping list. The user opens the smartphone app and taps the "Check inventory" button.

[0441] Device: Requests current inventory information from the cloud server. The device automatically requests current inventory information from the server.

[0442] Server: Generates a list of dishes that can be suggested based on inventory information and sends it to the device.The server generates a list of dishes that can be suggested based on inventory information (e.g., 1 / 2 cabbage, 3 carrots, 500g of chicken) and sends it to the device.

[0443] Device: Shows the user a list of suggested dishes. The device shows the user a suggestion for "Stir-fried cabbage and chicken."

[0444] User: Selects the dish they want to make from the suggested dishes. The user selects "Stir-fried cabbage and chicken" from the list of suggested dishes.

[0445] Server: Compares all ingredients needed for the selected dish with the current inventory and generates a list of ingredients that are missing. The server compares the required ingredients (1 cabbage, 300g chicken) with the inventory and adds the missing ingredients (1 / 2 cabbage) to the list.

[0446] Device: Displays a list of ingredients that are missing, and the user can check and edit the shopping list. The device displays a list of ingredients that are missing (1 / 2 a cabbage) to the user, and the user can check and edit the shopping list.

[0447] For example, after a user purchases a cabbage, three carrots, and 500g of chicken, the user takes a photo of the item with their smartphone, and the device sends the image to a server, which analyzes it and records it in a database. Next, the user uses half a cabbage for dinner and enters that information into the app. The device sends the input information to the server, and the server updates the inventory to "half a cabbage."

[0448] Next, the user launches the app and checks their current inventory (1 / 2 cabbage, 3 carrots, 500g of chicken) to create a shopping list. The emotion engine recognizes the user's emotion (e.g., tiredness) and suggests an easy-to-make dish, "Stir-fried cabbage and chicken." The server compares the required ingredients (1 cabbage, 300g of chicken) with the inventory and adds the missing 1 / 2 cabbage to the list. The device displays the list of missing ingredients to the user, who can then check and edit the shopping list.

[0449] In this way, by combining emotion engines, it becomes possible to make optimal suggestions based on the user's emotions, enabling more personalized ingredient management and cooking suggestions.

[0450] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0451] Step 1:

[0452] The user takes a photo of the food.

[0453] Input: Purchased ingredients

[0454] Specific operation: The user launches the smartphone app and uses the camera function to take a picture of ingredients (e.g., 1 cabbage, 3 carrots, and 500g of chicken).

[0455] Output: Captured image data

[0456] Step 2:

[0457] The device sends the image data to the cloud server.

[0458] Input: Photographed image data

[0459] Specific operation: When the user taps the "Send" button, the image data is uploaded to a cloud server via the Internet.

[0460] Output: Image data sent to the cloud server

[0461] Step 3:

[0462] The server analyzes the image data.

[0463] Input: Image data sent to the cloud server

[0464] Specific operation: The server analyzes the image data using the Google Cloud Vision API and identifies the type and quantity of ingredients (e.g., "1 cabbage, 3 carrots, 500g chicken").

[0465] Output: Parsed ingredient information

[0466] Step 4:

[0467] The server stores the analysis results in a database.

[0468] Input: Parsed ingredient information

[0469] Specific operation: The server saves ingredient information in a MySQL database and updates each user's inventory list.

[0470] Output: Updated database

[0471] Step 5:

[0472] The user enters the amount of ingredients used into the app.

[0473] Input: Amount of ingredients used for dinner

[0474] Specific operation: The user opens the smartphone app and enters the ingredients used (e.g., "1 / 2 cabbage used").

[0475] Output: Input usage information

[0476] Step 6:

[0477] The device sends usage information to the cloud server.

[0478] Input: Entered usage information

[0479] Specific operation: When the user taps the "Send" button, the usage information is sent to the cloud server.

[0480] Output: Usage information sent to the cloud server

[0481] Step 7:

[0482] The server updates the inventory information in the database.

[0483] Input: Usage information sent to the cloud server

[0484] Specific operation: The server updates the database and corrects the quantity of the relevant ingredient in the inventory list (e.g., changing the quantity of cabbage to "1 / 2 piece").

[0485] Output: Updated inventory information

[0486] Step 8:

[0487] A user accesses the emotion recognition engine and inputs an emotional state.

[0488] Input: Emotional state of the day

[0489] Specific operation: The user opens the smartphone app, accesses the emotion engine, and inputs their emotion for the day (e.g., "tired").

[0490] Output: Input emotion data

[0491] Step 9:

[0492] The device transmits the emotion data to the cloud server.

[0493] Input: Input emotion data

[0494] Specific operation: When the user taps the "Send" button, the emotion data is sent to the cloud server.

[0495] Output: Emotion data sent to the cloud server

[0496] Step 10:

[0497] The server analyzes emotions using an emotion engine and generates suggestions.

[0498] Input: Emotion data sent to the cloud server

[0499] What it does: The server uses the Microsoft Azure Emotion API to analyze emotions and generate dish and ingredient suggestions (e.g., "Stir-fried cabbage and chicken") based on those emotions.

[0500] Output: Emotion-based food and ingredient suggestions

[0501] Step 11:

[0502] The terminal notifies the user of the generated proposal.

[0503] Input: Emotion-based dish and ingredient suggestions

[0504] Specific behavior: The device notifies the user and suggests "Stir-fried cabbage and chicken."

[0505] Output: User notification

[0506] Step 12:

[0507] The user launches the app and checks inventory information.

[0508] Input: Current inventory information

[0509] Specific operation: When a user launches the smartphone app and taps the "Check stock" button, the app displays current stock information.

[0510] Output: User-confirmed inventory information

[0511] Step 13:

[0512] The terminal sends a request for inventory information to the cloud server.

[0513] Input: User-confirmed inventory information

[0514] Specific operation: The terminal automatically requests current inventory information from the server.

[0515] Output: Request sent to cloud server

[0516] Step 14:

[0517] The server generates a list of dishes based on the inventory information and sends it to the terminal.

[0518] Input: Requested inventory information

[0519] Specific operation: The server generates a list of dishes that can be suggested based on inventory information (e.g., 1 / 2 cabbage, 3 carrots, 500g of chicken) and sends it to the terminal.

[0520] Output: The recipe list sent to the user

[0521] Step 15:

[0522] The user selects the dish they want to make from the suggested dishes.

[0523] Input: A list of suggested dishes

[0524] Specific action: The user selects "Stir-fried cabbage and chicken" from a list of suggested dishes on a smartphone app.

[0525] Output: Selected dish items

[0526] Step 16:

[0527] The server compares the ingredients required for the selected dish with stock information and generates a list of ingredients that are in short supply.

[0528] Input: Selected dish item

[0529] Specific operation: The server compares the ingredients required for the selected dish (e.g., "Stir-fried cabbage and chicken") (1 cabbage, 300g chicken) with the current inventory information and adds the missing ingredients (1 / 2 cabbage) to the list.

[0530] Output: List of missing ingredients

[0531] Step 17:

[0532] The terminal displays a list of ingredients that are in short supply to the user.

[0533] Input: List of ingredients in short supply

[0534] Specific operation: The device displays a list of ingredients that are in short supply (e.g., 1 / 2 a cabbage) to the user, and the user checks and edits the shopping list.

[0535] Output: A list of missing ingredients displayed to the user

[0536] (Application example 2)

[0537] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0538] In today's world, users are expected to select foods and products that reflect their individual emotions and conditions. However, conventional systems were unable to provide emotion-based suggestions or inventory management, and the optimization of shopping lists and recipe suggestions did not reflect the user's emotions. This made shopping and recipe selection inconvenient, making it difficult to improve user satisfaction.

[0539] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0540] In this invention, the server includes: [a photographing means for the user to photograph ingredients;] [a transmitting means for transmitting the photographed image data to the cloud server;] [an analysis means for analyzing the image data in the cloud server and identifying the type and amount of ingredients;] [a storing means for saving the ingredient information identified as a result of the analysis;] [an update means for inputting the amount of ingredients used by the user based on the saved ingredient information and updating inventory information;] [an emotion analysis means for inputting user emotion data and analyzing it with an emotion engine;] [a suggestion means for suggesting dishes and products suitable for the user based on the emotion analysis results;] [a generation means for generating a shopping list based on current inventory information and suggested dishes; and] [a display means for displaying the generated shopping list to the user. This makes it possible to generate an optimal shopping list based on the user's emotions and inventory information, and to suggest dishes and products according to emotions.

[0541] The "photography means for the user to photograph ingredients" refers to a camera or other photographic device that records images of ingredients purchased by the user.

[0542] The "transmission means for transmitting the captured image data to the cloud server" refers to a communication device and software for uploading the captured image data to a server on the cloud via the Internet.

[0543] The "analysis means for analyzing image data on a cloud server and identifying the type and quantity of ingredients" refers to a program and hardware for recognizing the type and quantity of ingredients using an image analysis algorithm on a cloud server.

[0544] The "storage means for storing information on ingredients identified as analysis results" refers to a device and software for storing information on analyzed ingredients in a storage medium such as a database.

[0545] The "update means for inputting the amount of ingredients used by the user based on the stored ingredient information and updating inventory information" refers to an interface for inputting the amount of ingredients used by the user, and a program and hardware for updating inventory information based on that data.

[0546] "Emotion analysis means for inputting user emotion data and analyzing it with an emotion engine" refers to an emotion analysis engine and related software and hardware for inputting and interpreting a user's emotional state.

[0547] The "means for suggesting dishes and products suitable for the user based on the results of emotion analysis" refers to an algorithm and related hardware for selecting dishes and products that are likely to be preferred by the user based on the results of emotion analysis.

[0548] "Means for generating a shopping list based on current inventory information and suggested dishes" refers to a program and hardware for identifying ingredients that are in short supply based on current inventory information and automatically creating a shopping list.

[0549] The "display means for displaying the generated shopping list to the user" refers to a device and software for visually displaying the generated shopping list on a display device such as the user's smartphone or personal computer.

[0550] This invention is a system that combines an ingredient management system with an emotion engine that recognizes the user's emotions, making recipe suggestions and optimizing the shopping list according to the user's emotions. In addition to purchasing and inventory management and recipe suggestions, this system is characterized by recognizing the user's emotions with the emotion engine and making suggestions based on the user's emotions.

[0551] Basic system configuration

[0552] Photographing and storing ingredients

[0553] The user takes a photo of the ingredients they have purchased using the camera on their smartphone. The captured image data is sent to a cloud server via the smartphone's transmission means. The cloud server then uses an image analysis algorithm (e.g., Google Cloud Vision, Amazon Rekognition) to identify the type and volume of the ingredients. The ingredient information identified as a result of the analysis is then stored in a database.

[0554] Inventory management

[0555] Users input the amounts of ingredients used for dinner into a smartphone app, and the data is sent to a cloud server, which updates the inventory information in the database.

[0556] Emotion Recognition and Suggestions

[0557] The user inputs their emotional state for the day into the emotion engine via a smartphone app. The emotion data is sent to a cloud server and analyzed by an emotion analysis engine (e.g., Microsoft Azure Cognitive Services, IBM Watson (registered trademark)). Based on the emotion analysis results, the system makes recommendations for dishes and products suitable for the user.

[0558] Generate a shopping list

[0559] The user launches the smartphone app and generates a shopping list based on current inventory information. The server then generates a list of ingredients that are in short supply and displays that information to the user.

[0560] Specific examples

[0561] The user follows the steps described above and after shopping, purchases 1 cabbage, 3 carrots, and 500g of chicken, and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database (1 cabbage, 3 carrots, 500g of chicken). The user eats 1 / 2 a cabbage for dinner and enters that information into the app. The device sends the input information to the server, and the server updates the inventory to "1 / 2 a cabbage."

[0562] Prompt Sentence Examples

[0563] Prompt input for generative AI model:

[0564] 1. "Imagine a flow where a user inputs emotional data, sends that data to a server, and suggests dishes based on the user's emotions."

[0565] 2. "When a user takes a picture of an ingredient, generate a program that analyzes the image and stores it in a database."

[0566] In this way, by combining emotion engines, it becomes possible to make optimal suggestions based on the user's emotions, enabling more personalized ingredient management and cooking suggestions.

[0567] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0568] Step 1:

[0569] The user takes a photo of the ingredients they have purchased using their smartphone camera.

[0570] Input: Image of purchased ingredients

[0571] Output: Captured image data

[0572] How it works: The user takes a photo of an ingredient with their smartphone camera and obtains image data.

[0573] Step 2:

[0574] The device sends the captured image data to a cloud server.

[0575] Input: Captured image data

[0576] Output: Image data transferred to the cloud server

[0577] Operation: Using the device's transmission means, the captured image data is uploaded to a cloud server via the Internet.

[0578] Step 3:

[0579] The server analyzes the image data sent to the cloud and identifies the type and volume of ingredients.

[0580] Input: Image data sent to the cloud server

[0581] Output: Ingredient type and quantity information

[0582] How it works: Using image analysis algorithms on a cloud server, the type and volume of ingredients are recognized from image data (e.g., Google Cloud Vision, Amazon Rekognition).

[0583] Step 4:

[0584] The server stores the ingredient information identified as a result of the analysis in a database.

[0585] Input: Ingredient type and quantity information

[0586] Output: Ingredient information stored in a database

[0587] Operation: The server records the identified ingredient information using a storage means that stores it in a database.

[0588] Step 5:

[0589] The user enters the amount of ingredients used into a smartphone app.

[0590] Input: Amount of ingredients used

[0591] Output: Usage data sent from the smartphone

[0592] How it works: The user uses the app's interface to input the amounts of ingredients used.

[0593] Step 6:

[0594] The terminal transmits the input usage information to the cloud server.

[0595] Input: Usage data

[0596] Output: Usage data transferred to the cloud server

[0597] Operation: The terminal uses the transmission means to upload the input data to the cloud server.

[0598] Step 7:

[0599] The server updates the inventory information in the database.

[0600] Input: Usage data

[0601] Output: Updated inventory information

[0602] How it works: The server updates the inventory information in the database based on the usage data.

[0603] Step 8:

[0604] The user enters their emotional state for the day into a smartphone app.

[0605] Input: Emotional state data

[0606] Output: Emotional state data sent from the smartphone

[0607] How it works: The user uses the app's interface to input their emotional state for the day.

[0608] Step 9:

[0609] The device transmits the emotion data to the cloud server.

[0610] Input: Emotional state data

[0611] Output: Emotional state data transferred to the cloud server

[0612] Operation: The device uses the transmission means to upload emotion data to the cloud server.

[0613] Step 10:

[0614] The server analyzes the user's emotions using an emotion engine.

[0615] Input: Emotional state data

[0616] Output: Emotion analysis results

[0617] How it works: The server uses an emotion analysis engine (e.g., Microsoft Azure Cognitive Services, IBM Watson) to analyze the emotion data.

[0618] Step 11:

[0619] The server will suggest dishes and products suitable for the user based on the results of the emotion analysis.

[0620] Input: Sentiment analysis results, inventory information

[0621] Output: A list of suggested dishes and products

[0622] How it works: The server creates a list of suitable dishes and products based on the results of sentiment analysis and inventory information.

[0623] Step 12:

[0624] The server generates a shopping list based on current inventory information and suggested dishes.

[0625] Input: Proposal list, inventory information

[0626] Output: Shopping list

[0627] How it works: The server compares the suggested list with inventory information and generates a list of ingredients or products that are in short supply.

[0628] Step 13:

[0629] The terminal displays the generated shopping list to the user.

[0630] Input: Shopping list

[0631] Output: A shopping list visually displayed to the user

[0632] Operation: The terminal uses a display means to visually display the shopping list to the user.

[0633] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0634] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0635] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0636] [Second embodiment]

[0637] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0638] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0639] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0640] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0641] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0642] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0643] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0644] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0645] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0646] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0647] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0648] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0649] overview

[0650] This invention is a system that streamlines food inventory management within the home and prevents forgetting to use ingredients or purchasing duplicates. Users take photos of purchased ingredients with their smartphones and send the images to a cloud server, where AI identifies and records the type and volume of ingredients. The data is stored in the cloud and used for inventory management, creating shopping lists, and suggesting recipes.

[0651] Program processing overview

[0652] Food Recognition and Preservation

[0653] User: Takes a photo of the ingredients he / she has purchased using his / her smartphone camera.

[0654] Terminal: Sends captured image data to a cloud server.

[0655] Server: Analyzes the received image data using AI and recognizes the type and volume of ingredients.

[0656] Server: Saves the analysis results in a database.

[0657] Example: A user purchases one cabbage, three carrots, and 500g of chicken, and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database.

[0658] Inventory management

[0659] User: Enters the amount of ingredients used for dinner into the app.

[0660] Terminal: Sends the entered usage information to the cloud server.

[0661] Server: Updates the inventory information in the database.

[0662] Example: A user makes a cabbage salad and uses 1 / 2 a cabbage. The user enters "1 / 2 a cabbage used" into the app, and the server updates the cabbage inventory in the database to "1 / 2 a cabbage."

[0663] Shopping suggestions

[0664] User: Launches the app and creates a shopping list.

[0665] Terminal: Retrieves current inventory information from the cloud.

[0666] Server: Generates a list of dishes that can be suggested based on inventory information and sends it to the device.

[0667] Terminal: Shows the user a list of suggested dishes.

[0668] User: Select the dish they want to make from the suggested dishes.

[0669] Server: Compares the ingredients required for the selected dish with the current inventory information and generates a list of ingredients that are missing.

[0670] Terminal: Displays a list of missing ingredients and allows the user to review and edit the shopping list.

[0671] Example: A user launches an app, and the app retrieves inventory information for "1 / 2 cabbage," "3 carrots," and "500g chicken" from the server. Based on this inventory, the app suggests "stir-fried cabbage and chicken," which the user selects. The server compares the required ingredients (1 cabbage, 300g chicken) with the inventory, and adds the missing 1 / 2 cabbage to the list. The app displays this list to the user, who can then confirm and edit the shopping list.

[0672] Implementation flow

[0673] To implement this system, users use a smartphone app to take a photo of the ingredients they have purchased and send the image to a cloud server via the app. The cloud server then analyzes the image, identifies the type and quantity of ingredients, and stores the information in a database. The app updates inventory information in real time and also reflects the amount of ingredients entered by the user. When shopping, the app also displays suggested dishes based on current inventory information and automatically generates a list of ingredients that are in short supply.

[0674] This allows users to efficiently manage ingredients and receive planned cooking suggestions, thereby reducing food waste and enabling efficient household budget management.

[0675] The processing flow will be explained below.

[0676] Food Recognition and Preservation

[0677] Step 1:

[0678] The user takes a photo of the purchased ingredients using the smartphone camera.

[0679] Step 2:

[0680] The device saves the captured image data and calls an API to send it to the cloud server.

[0681] Step 3:

[0682] The server temporarily stores the received image data and passes the data to an AI model for image analysis.

[0683] Step 4:

[0684] The server uses an AI model to analyze the image data and identify the type and quantity of ingredients, for example, "1 cabbage," "3 carrots," and "500g of chicken."

[0685] Step 5:

[0686] The server saves the analysis results in a database and records them as "1 cabbage," "3 carrots," and "500g of chicken."

[0687] Inventory management

[0688] Step 6:

[0689] The user enters the amount of ingredients used for dinner into the app.

[0690] Step 7:

[0691] The device calls an API to send the entered usage information to the cloud server.

[0692] Step 8:

[0693] The server temporarily stores the received usage data and updates the inventory information in the database.

[0694] Step 9:

[0695] The server reduces the inventory of specific ingredients in the database and updates it to "1 / 2 cabbage," "3 carrots," and "500g chicken."

[0696] Shopping suggestions

[0697] Step 10:

[0698] A user launches the app and checks current inventory information to create a shopping list.

[0699] Step 11:

[0700] The device requests current inventory information from the cloud server.

[0701] Step 12:

[0702] The server retrieves current inventory information from the database and sends it to the terminal.

[0703] Step 13:

[0704] Based on the inventory information received by the terminal, a list of dishes that can be suggested to the user is generated and displayed.

[0705] Step 14:

[0706] The user selects the dish they want to make from the suggested dishes.

[0707] Step 15:

[0708] The server compares all ingredients required for the selected dish with current inventory information and generates a list of ingredients that are in short supply.

[0709] Step 16:

[0710] The server transmits a list of ingredients that are in short supply to the terminal, which then displays it to the user.

[0711] Step 17:

[0712] The user checks the displayed list of ingredients that are in short supply and creates and edits a shopping list.

[0713] Specific examples of implementation

[0714] Step 18:

[0715] After shopping, the user purchases one cabbage, three carrots, and 500g of chicken and takes a photo of it with their smartphone.

[0716] Step 19:

[0717] The device sends the image to the server, which analyzes it and records it in a database ("1 cabbage," "3 carrots," "500g chicken").

[0718] Step 20:

[0719] A user eats 1 / 2 a cabbage for dinner and enters that information into the app.

[0720] Step 21:

[0721] The terminal sends the input information to the server, and the server updates the inventory to "1 / 2 cabbage."

[0722] Step 22:

[0723] Before the next shopping trip, the user launches the app, which retrieves the current inventory information. ("1 / 2 cabbage")

[0724] Step 23:

[0725] The server sends inventory information for "1 / 2 cabbage," "3 carrots," and "500g of chicken" to the terminal, and the terminal then suggests dishes based on that information.

[0726] Step 24:

[0727] The user selects "Stir-fried cabbage and chicken" and adds "1 / 2 cabbage" to the list, for which the server is short.

[0728] Step 25:

[0729] The device displays a list of ingredients that are in short supply, and the user can check and edit the shopping list.

[0730] These are the specific processing steps, from recognizing and storing ingredients to inventory management and suggestions when shopping.

[0731] Example 1

[0732] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0733] In most cases, food inventory management in the home is done manually, resulting in frequent forgetting to use ingredients or duplicate purchases. Furthermore, determining what to buy next requires knowing the current inventory status and recipes for the necessary dishes, which is time-consuming. Furthermore, there are often times when people are out of ideas for cooking, so an efficient way to solve these problems is needed.

[0734] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0735] In this invention, the server includes: [a photographing means for the user to photograph ingredients;] [a transmitting means for transmitting the photographed image data to a network server;] [an analyzing means for analyzing the image data in the network server and identifying the type and amount of ingredients;] [a storing means for saving the ingredient information identified as a result of the analysis;] [an updating means for inputting the amount of ingredients used by the user based on the stored ingredient information and updating inventory information;] [a generating means for generating a shopping list based on current inventory information;] [a display means for displaying the generated shopping list to the user;] [an automatic output means for generating a list of suggested dishes; and] [a list generating means for generating a list of missing ingredients based on the suggested dishes. This enables ingredient inventory management, prevention of forgotten ingredients and duplicate purchases, efficient shopping list creation, and dish suggestions.

[0736] The term "photography means for a user to photograph ingredients" refers to a device or application that a user uses to record the ingredients they have purchased as digital images.

[0737] The "transmission means for transmitting the photographed image data to the network server" refers to a device or application for transmitting the photographed image data of ingredients to a server on the cloud via the Internet.

[0738] "Analysis means for analyzing image data on a network server and identifying the type and volume of ingredients" refers to software or algorithms for analyzing received image data and automatically identifying the type and volume of ingredients.

[0739] "Storage means for storing information on ingredients identified as a result of the analysis" refers to a device or application for recording data on the type and volume of ingredients obtained through the analysis in storage such as a database.

[0740] "Means for updating inventory information by inputting the amount of ingredients used by the user based on the stored ingredient information" refers to a device or application for correcting and updating inventory data in real time based on the amount of ingredients used input by the user through the application.

[0741] "Means for generating a shopping list based on current inventory information" refers to a device or application that references recorded inventory information and automatically creates a list of new ingredients to be purchased.

[0742] The "display means for displaying the generated shopping list to the user" refers to a device or application for visually displaying the automatically generated shopping list on the display of the user's device, for example.

[0743] "Automatic output means for generating a list of suggested dishes" refers to a device or application that automatically generates and presents a list of dishes that can be made based on stored inventory information and data such as user preferences.

[0744] "List generation means for generating a list of missing ingredients based on a proposed dish" refers to a device or application that compares ingredients required for the proposed dish with inventory data and lists missing ingredients.

[0745] MODE FOR CARRYING OUT THE INVENTION

[0746] This invention is a system designed to improve the efficiency of food ingredient inventory management in the home, prevent forgetting to use ingredients or purchasing the same ingredients twice, and provide recipe suggestions based on a well-planned recipe. Specific embodiments of this system are described below.

[0747] A user uses a smartphone to take a photo of the ingredients they have purchased. The smartphone's camera (photography means) is used to capture high-resolution images of the ingredients. For example, imagine a case where a user takes a photo of one cabbage, three carrots, and 500g of chicken upon returning home from the supermarket. The captured image data is sent to a network server on the cloud via an application on the smartphone (transmission means).

[0748] The network server analyzes the received image data. Specifically, it uses image recognition services such as Google Cloud Vision API and Amazon Rekognition to identify the type and volume of ingredients in the image (analysis means). The data resulting from the analysis is stored in a cloud database (e.g., Google Firebase or Amazon DynamoDB) (storage means).

[0749] When a user uses an ingredient, they input the amount used into the smartphone application. For example, if they input "1 / 2 cabbage used," the application sends this information to the cloud server, and the inventory information in the cloud database is updated (update means).

[0750] Furthermore, when a user launches the app to go shopping, the application retrieves current inventory information from the cloud. A shopping list is automatically generated based on the current inventory information (generation means). The generated shopping list is displayed to the user, allowing the user to use the list as a reference when shopping (display means).

[0751] The system also has the ability to suggest dishes using a generative AI model. For example, it uses a language model such as OpenAI's GPT-3 to generate a list of suggested dishes based on inventory information and displays it to the user (automatic output means). When the user selects one of the suggested dishes, the system compares the ingredients needed for that dish with current inventory information and automatically generates a list of ingredients that are missing (list generation means).

[0752] The above process allows users to efficiently manage ingredients and plan their shopping and cooking, thereby reducing food waste and enabling efficient management of household finances.

[0753] For example, suppose the user enters the prompt text as follows:

[0754] "I have half a cabbage, three carrots, and 500g of chicken left in my fridge. What kind of dish can I make with these ingredients? Can you also tell me how to make it?"

[0755] Based on these prompts, the generative AI model will suggest optimal dishes and cooking methods to the user, allowing them to expand the range of cooking they can do at home.

[0756] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0757] Program processing flow

[0758] Step 1:

[0759] The user takes a photo of the ingredients

[0760] Specific operation: The user uses the smartphone camera to take a picture of the ingredients they purchased. For example, they take a picture of one cabbage, three carrots, and 500g of chicken lined up.

[0761] Input: An image of the food captured by the camera.

[0762] Output: Image data of ingredients stored on the smartphone.

[0763] Step 2:

[0764] The device sends the image to the cloud server.

[0765] Specific operation: A smartphone app sends captured image data to a cloud server via the Internet. For example, the app uploads the images to AWS S3.

[0766] Input: Image data of ingredients stored on your smartphone.

[0767] Output: Image data sent to the cloud server.

[0768] Step 3:

[0769] The server analyzes the image and recognizes the type and volume of ingredients.

[0770] Specific operation: The cloud server analyzes the received image data using Google Cloud Vision API and Amazon Rekognition to identify the type and quantity of ingredients. For example, it might recognize it as one cabbage, three carrots, and 500g of chicken.

[0771] Input: Image data stored on a cloud server.

[0772] Output: Analysis results including ingredient type and volume.

[0773] Step 4:

[0774] The server stores the analysis results in a database

[0775] Specific operation: The food ingredient information from the analysis results is saved in a cloud database (e.g., Google Firebase or Amazon DynamoDB).

[0776] Input: Analysis results including ingredient type and volume.

[0777] Output: Ingredient information recorded in a cloud database.

[0778] Step 5:

[0779] The user enters the amount of ingredients used into the app.

[0780] Specific operation: The user inputs the amount of ingredients used for dinner into the smartphone app. For example, the user inputs that half a cabbage was used.

[0781] Input: Amount of ingredients used.

[0782] Output: Usage information entered into the smartphone app.

[0783] Step 6:

[0784] The device sends usage information to the cloud server.

[0785] Specific operation: The smartphone app sends the entered usage information to a cloud server. For example, the app sends usage data to Firebase Firestore.

[0786] Input: Usage information stored in the smartphone app.

[0787] Output: Usage information sent to the cloud server.

[0788] Step 7:

[0789] The server updates the inventory information in the database

[0790] Specific operation: The cloud server updates the inventory information in the cloud database in real time based on the received usage information. For example, the inventory of cabbage is updated to 1 / 2.

[0791] Input: Usage information sent to the cloud server.

[0792] Output: Updated inventory information from the cloud database.

[0793] Step 8:

[0794] The user launches the app and creates a shopping list

[0795] Specific action: The user launches the smartphone app and accesses the screen for creating a shopping list.

[0796] Input: The app launch operation.

[0797] Output: Display of the app's shopping list creation screen.

[0798] Step 9:

[0799] The device retrieves current inventory information from the cloud.

[0800] Specific operation: The smartphone app accesses a cloud database to get the latest inventory information, for example, from Firebase Firestore.

[0801] Input: A retrieval request to the cloud database.

[0802] Output: The latest inventory information obtained.

[0803] Step 10:

[0804] The server generates recipe suggestions based on inventory information and sends them to the device.

[0805] Specific operation: The cloud server uses the acquired inventory information to generate a list of dishes that are best suited to the user using a language model such as GPT-3, and sends this list to the smartphone app.

[0806] Input: Latest inventory information.

[0807] Output: A list of suggested dishes.

[0808] Step 11:

[0809] The device displays suggested dishes

[0810] Specific operation: The smartphone app displays the list of suggested dishes received from the server to the user. For example, it displays a list including "stir-fried cabbage and chicken."

[0811] Input: A list of dish suggestions received from the server.

[0812] Output: A list of cooking suggestions displayed on a smartphone app.

[0813] Step 12:

[0814] The user selects the dish they want to make.

[0815] Specific operation: The user selects the dish they want to make from the list of suggested dishes. For example, they select "Stir-fried cabbage and chicken."

[0816] Input: User's food selection.

[0817] Output: Selected dish information.

[0818] Step 13:

[0819] The server compares the required ingredients with the available stock and generates a list of ingredients that are in short supply.

[0820] Specific operation: The cloud server compares the list of ingredients needed for the selected dish with the current inventory information and lists any ingredients that are missing. For example, it recognizes that 1 / 2 a cabbage is missing.

[0821] Input: Selected dish information and current inventory information.

[0822] Output: A list of missing ingredients.

[0823] Step 14:

[0824] The device displays a list of ingredients that are missing, and the user can check and edit the shopping list.

[0825] Specific operation: The smartphone app displays the list of ingredients that are in short supply received from the server to the user, who then checks and edits the list.

[0826] Input: Missing ingredient list.

[0827] Output: A list of ingredients that are in short supply, displayed on the smartphone app. Users can confirm and edit the list.

[0828] (Application example 1)

[0829] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0830] Inventory management of ingredients and products at home or in a brick-and-mortar store is complicated, and manual management can easily lead to forgetting to use ingredients or purchasing duplicates. Furthermore, if product inventory is not managed properly, there is a high risk of unsold items or shortages. This increases food waste and opportunity losses, hindering efficient operations.

[0831] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0832] In this invention, the server includes [a photographing means for the user to photograph ingredients], [a transmission means for transmitting the photographed image data to the cloud server], [an analysis means for analyzing the image data in the cloud server and identifying the type and amount of ingredients], [a storage means for saving the ingredient information identified as a result of the analysis], [an update means for inputting the amount of ingredients used by the user based on the saved ingredient information and updating inventory information], [a generation means for generating a shopping list based on current inventory information], [a display means for displaying the generated shopping list to the user], [a suggestion means for suggesting products to the user in conjunction with inventory], and [a detection means for detecting missing ingredients or products]. This makes it possible to automatically manage ingredient and product inventory and make appropriate suggestions to the user.

[0833] "Photographing means" refers to a device or method that allows a user to photograph ingredients or products.

[0834] The "transmission means" is a device or method for transmitting captured image data to a cloud server.

[0835] The "analysis means" is a device or method for analyzing image data on a cloud server and identifying the type and volume of ingredients or products.

[0836] The "storage means" refers to a device or method for storing the ingredients and product information identified as analysis results in a database.

[0837] The "update means" is a device or method for inputting the amount of ingredients or products used by the user based on the stored ingredient and product information and updating the inventory information.

[0838] The "generation means" is a device or method for generating a shopping list or a recommended product list based on current inventory information.

[0839] The "display means" refers to a device or method for displaying the generated shopping list or recommended product list to the user.

[0840] The "suggestion means" is a device or method for suggesting products to users in conjunction with inventory.

[0841] "Detection means" refers to a device or method for detecting missing ingredients or products.

[0842] Overall system configuration

[0843] The system of this invention is broadly composed of a photographing means, a transmitting means, a analyzing means, a saving means, a updating means, a generating means, a displaying means, a suggesting means, and a detecting means. These means work together to realize inventory management of ingredients and products and shopping support based on that inventory management.

[0844] Program Overview

[0845] Filming method

[0846] The user uses a smartphone to take a photo of the ingredients or products they have purchased, and the device captures the image using the smartphone's camera.

[0847] Transmission method

[0848] The captured image data is sent to a cloud server. This device or software uses the smartphone's communication functions (Wi-Fi, mobile data communication, etc.).

[0849] Analysis means

[0850] The cloud server analyzes the captured image data using an AI model (for example, a deep learning model such as TensorFlow) to identify the type and volume of specific ingredients or products. This analysis uses image recognition technology.

[0851] Preservation means

[0852] The analyzed ingredient and product information is stored in a cloud database (such as AWS RDS or Google Cloud Firestore). The type, size, and photo date and time of each ingredient or product are recorded in the database.

[0853] Update method

[0854] Users input the quantities of ingredients and products used into the application, and the information is sent to a cloud server, which updates the inventory information in a database based on the input.

[0855] generation means

[0856] Based on the current inventory information, the cloud server generates a shopping list and a list of recommended products, allowing users to efficiently create lists of ingredients and products they need.

[0857] Display means

[0858] The generated shopping list and recommended product list are displayed to the user through a smartphone application. The user interface can be built using a cross-platform framework such as React Native.

[0859] Proposal means

[0860] The server connects with current inventory information and suggests suitable products and recipes to users based on the latest inventory information and the user's past purchase history.

[0861] Detection Method

[0862] Every time inventory data is updated, the cloud server compares it with the existing inventory data to detect any missing ingredients or products, which are then automatically added to the user's shopping list.

[0863] Specific examples

[0864] For example, when a store employee places new tomatoes on a shelf, they take a picture of the tomato using the app, which then uploads the image to the cloud. An AI model on the cloud server analyzes the image, identifies the type and quantity of tomatoes, and stores the information in a database. When a customer opens the app, fresh tomatoes are recommended based on the latest inventory information.

[0865] Prompt Sentence Examples

[0866] "Please upload images of newly arrived tomatoes to the cloud server. The AI ​​on the server side will identify the quantity and quality of the tomatoes, update the inventory information, and reflect it as recommendations on the customer app."

[0867] As a result, the system of the present invention can improve the efficiency of inventory management of food ingredients and products at home or in a physical store, and can support customers in making appropriate product suggestions and generating shopping lists.

[0868] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0869] Step 1:

[0870] The user takes a photo of the food or product using a smartphone.

[0871] Input: Image of ingredients or products

[0872] Output: Image data stored on a smartphone

[0873] Specific actions: The user launches the smartphone camera app, frames the food or product they are purchasing, and presses the shutter button. When doing so, they take as clear an image as possible, taking care to ensure that the food or product is clearly identifiable.

[0874] Step 2:

[0875] The device sends the captured image data to a cloud server.

[0876] Input: Captured image data

[0877] Output: Image data transferred to the cloud server

[0878] What happens: The smartphone app compresses the captured image file and sends an HTTP POST request to a cloud server over the internet, including user authentication information.

[0879] Step 3:

[0880] The server receives the image data at the cloud server and analyzes it using the analysis means.

[0881] Input: Image data transferred to the cloud server

[0882] Output: Analyzed ingredients and product types and volume data

[0883] How it works: The server stores the received image data on disk and performs image recognition using an AI model (e.g., TensorFlow). This AI model identifies the type and volume of ingredients or products from the image and extracts this information as metadata.

[0884] Step 4:

[0885] The server stores the analysis results in a database.

[0886] Input: Analyzed ingredient and product type and volume data

[0887] Output: Analysis result data stored in a database

[0888] Specific operation: The server converts the analysis results into structured data and executes an INSERT query to a cloud database (e.g., AWS RDS, Google Cloud Firestore). The stored data includes the type of ingredients or products, volume, analysis date and time, etc.

[0889] Step 5:

[0890] The user enters the amounts of ingredients and products used into the application.

[0891] Input: Usage information

[0892] Output: Usage data transferred to the cloud server

[0893] Specific operation: The user opens a form on the smartphone application to input the amount of ingredients and products used, enters the necessary information, for example, "1 / 2 cabbage used," and presses the send button. This information is sent to the cloud server.

[0894] Step 6:

[0895] The server updates the inventory information in the database based on the input usage data.

[0896] Input: Usage data transferred to the cloud server

[0897] Output: Updated inventory information

[0898] Specific operation: The server analyzes the received usage data and reduces the corresponding inventory data. It updates the inventory information in the database using an UPDATE query. The updated inventory information reflects the remaining amount of the ingredient or product.

[0899] Step 7:

[0900] The server generates shopping lists and recommended product lists based on current inventory information.

[0901] Input: Updated inventory information

[0902] Output: Generated shopping list and recommended products list

[0903] Specific operation: The server identifies ingredients and products that are in short supply based on the latest inventory data, creates a list of items to purchase, and adds recommended products and recipes to the list, taking into account the user's past purchase history and trend data.

[0904] Step 8:

[0905] The terminal displays the generated shopping list and recommended product list to the user.

[0906] Input: Generated shopping list or recommended product list

[0907] Output: List displayed on smartphone

[0908] What it does: The smartphone application analyzes the list data received from the server and displays it in an easy-to-read format on the user interface, allowing the user to plan their shopping needs.

[0909] Step 9:

[0910] The server works in conjunction with inventory information to suggest products to users.

[0911] Input: Current inventory information, user's past purchase history

[0912] Output: Suggested products and recipe information

[0913] What it does: The server cross-references the inventory database with the user's purchasing history to generate product and recipe recommendations that are best suited to the user. These recommendations are listed in order of most relevant and notified to the user.

[0914] Step 10:

[0915] The server detects ingredients or products that are in short supply and adds them to the shopping list.

[0916] Input: Current inventory information

[0917] Output: Updated shopping list

[0918] What it does: The server periodically scans inventory data to identify ingredients or products that are running low, automatically adding them to the shopping list and notifying the user.

[0919] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0920] overview

[0921] This invention is a system that combines an ingredient management system with an emotion engine that recognizes the user's emotions, making recipe suggestions and optimizing the shopping list according to the user's emotions. In addition to purchasing and inventory management and recipe suggestions, this system is characterized by recognizing the user's emotions with the emotion engine and making suggestions based on the user's emotions.

[0922] Program processing overview

[0923] Food Recognition and Preservation

[0924] User: Takes a photo of the purchased ingredients using the smartphone camera.

[0925] Terminal: Sends captured image data to a cloud server.

[0926] Server: Analyzes the received image data using AI and recognizes the type and volume of ingredients.

[0927] Server: Saves the analysis results in a database.

[0928] Example: A user purchases one cabbage, three carrots, and 500g of chicken, and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database.

[0929] Inventory management

[0930] User: Enters the amount of ingredients used for dinner into the app.

[0931] Terminal: Sends the entered usage information to the cloud server.

[0932] Server: Updates the inventory information in the database.

[0933] Example: A user makes a cabbage salad and uses 1 / 2 a cabbage. The user enters "1 / 2 a cabbage used" into the app, and the server updates the cabbage inventory in the database to "1 / 2 a cabbage."

[0934] Emotion Recognition and Suggestions

[0935] User: Accesses the emotion engine through the app and inputs their emotional state for the day.

[0936] Terminal: Sends the user's emotional data to the cloud server.

[0937] Server: The emotion engine analyzes the user's emotions and suggests dishes and ingredients based on those emotions.

[0938] For example, if the user is tired, easy meals will be suggested, and if the user is energetic, more challenging meals will be suggested.

[0939] Shopping suggestions

[0940] User: Launches the app and checks current inventory information to create a shopping list.

[0941] Terminal: Requests current inventory information from the cloud server.

[0942] Server: Generates a list of dishes that can be suggested based on inventory information and sends it to the device.

[0943] Terminal: Shows the user a list of suggested dishes.

[0944] User: Select the dish they want to make from the suggested dishes.

[0945] Server: Compares all ingredients required for the selected dish with current inventory and generates a list of ingredients that are missing.

[0946] Terminal: Displays a list of missing ingredients and allows the user to review and edit the shopping list.

[0947] Example: A user launches the app and retrieves current inventory information (1 / 2 cabbage, 3 carrots, 500g chicken). The emotion engine recognizes that the user is tired and suggests a simple "stir-fried cabbage and chicken." The server adds the missing 1 / 2 cabbage to the list, and the device displays the shopping list.

[0948] Specific examples of implementation

[0949] After shopping, a user purchases 1 cabbage, 3 carrots, and 500g of chicken and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database (1 cabbage, 3 carrots, 500g of chicken). The user eats 1 / 2 a cabbage for dinner and enters that information into the app. The device sends the input information to the server, which updates the inventory to "1 / 2 a cabbage."

[0950] Next, the user launches the app and checks the current inventory information (1 / 2 cabbage, 3 carrots, 500g of chicken) to create a shopping list. The emotion engine recognizes the user's emotion (e.g., tiredness) and suggests an easy-to-make dish, "Stir-fried cabbage and chicken." The server compares the required ingredients (1 cabbage, 300g of chicken) with the inventory and adds the missing 1 / 2 cabbage to the list. The device displays the list of missing ingredients, and the user can check and edit the shopping list.

[0951] In this way, by combining emotion engines, it becomes possible to make optimal suggestions based on the user's emotions, enabling more personalized ingredient management and cooking suggestions.

[0952] The processing flow will be explained below.

[0953] Processing flow of a system that combines emotion engines

[0954] Emotion Recognition and Suggestions

[0955] Step 1:

[0956] The user launches the app and enters their emotional state for the day.

[0957] Step 2:

[0958] The device transmits the user's emotional data to a cloud server.

[0959] Step 3:

[0960] The server passes the received emotion data to the emotion engine, which analyzes the user's emotional state.

[0961] Step 4:

[0962] The server generates a list of dishes according to the emotion based on the analysis results of the emotion engine.

[0963] Food Recognition and Preservation

[0964] Step 5:

[0965] The user takes a photo of the purchased ingredients using the smartphone camera.

[0966] Step 6:

[0967] The device saves the captured image data and calls an API to send it to the cloud server.

[0968] Step 7:

[0969] The server temporarily stores the received image data and passes the data to an AI model for image analysis.

[0970] Step 8:

[0971] The server uses an AI model to analyze the image data and identify the type and quantity of ingredients, for example, "1 cabbage," "3 carrots," and "500g of chicken."

[0972] Step 9:

[0973] The server saves the analysis results in a database and records them as "1 cabbage," "3 carrots," and "500g of chicken."

[0974] Inventory management

[0975] Step 10:

[0976] The user enters the amount of ingredients used for dinner into the app.

[0977] Step 11:

[0978] The device calls an API to send the entered usage information to the cloud server.

[0979] Step 12:

[0980] The server temporarily stores the received usage data and updates the inventory information in the database.

[0981] Step 13:

[0982] The server reduces the inventory of specific ingredients in the database and updates it to "1 / 2 cabbage," "3 carrots," and "500g chicken."

[0983] Shopping suggestions

[0984] Step 14:

[0985] A user launches the app and checks current inventory information to create a shopping list.

[0986] Step 15:

[0987] The device requests current inventory information from the cloud server.

[0988] Step 16:

[0989] The server retrieves current inventory information from the database and sends it to the terminal.

[0990] Step 17:

[0991] Based on the inventory information received by the terminal, a list of dishes that can be suggested to the user is generated and displayed.

[0992] Step 18:

[0993] The user selects the dish they want to make from the suggested dishes.

[0994] Step 19:

[0995] The server compares all ingredients required for the selected dish with current inventory information and generates a list of ingredients that are in short supply.

[0996] Step 20:

[0997] The server transmits a list of ingredients that are in short supply to the terminal, which then displays it to the user.

[0998] Step 21:

[0999] The user checks the displayed list of ingredients that are in short supply and creates and edits a shopping list.

[1000] Specific examples of implementation

[1001] Step 22:

[1002] After shopping, the user purchases one cabbage, three carrots, and 500g of chicken and takes a photo of it with their smartphone.

[1003] Step 23:

[1004] The device sends the image to the server, which analyzes it and records it in a database ("1 cabbage," "3 carrots," "500g chicken").

[1005] Step 24:

[1006] A user eats 1 / 2 a cabbage for dinner and enters that information into the app.

[1007] Step 25:

[1008] The terminal sends the input information to the server, and the server updates the inventory to "1 / 2 cabbage."

[1009] Step 26:

[1010] The user launches the app before their next shopping trip, and the app retrieves the current inventory information ("1 / 2 cabbage," "3 carrots," and "500g chicken").

[1011] Step 27:

[1012] The device inputs the user's emotional state into a cloud server, which then analyzes the data.

[1013] Step 28:

[1014] The emotion engine recognizes that the user is tired and suggests an easy-to-make dish: stir-fried cabbage and chicken.

[1015] Step 29:

[1016] The server compares the ingredients needed ("1 cabbage" and "300g chicken") with the inventory and adds the missing 1 / 2 cabbage to the list.

[1017] Step 30:

[1018] The device displays a list of ingredients that are in short supply, and the user can check and edit the shopping list.

[1019] The above is the specific processing flow of a system that combines an emotion engine. By taking the user's emotions into consideration, more personalized suggestions become possible, which is expected to improve user satisfaction.

[1020] Example 2

[1021] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1022] In today's busy lifestyles, it is difficult for users to efficiently manage ingredient inventory, select dishes, and receive recipe suggestions tailored to their mood of the day. Conventional systems require users to manually update inventory information and manage ingredient usage, which is tedious and often inaccurate. Furthermore, they lack the ability to suggest dishes based on the user's emotions, resulting in a non-personalized user experience. Therefore, there is a need for a system that allows users to efficiently manage ingredients, inventory, recognize emotions, and receive optimal recipe suggestions.

[1023] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1024] In this invention, the server includes: [a photographing means for the user to photograph ingredients;] [a transmission means for sending the photographed image data to the cloud server;] [an analysis means for analyzing the image data in the cloud server and identifying the type and amount of ingredients;] [a storage means for saving ingredient information identified as a result of the analysis;] [an update means for inputting the amount of ingredients used by the user based on the saved ingredient information and updating inventory information;] [an emotion recognition and suggestion means for recognizing the user's emotions and suggesting dishes and ingredients based on the emotions;] [a notification means for notifying the user of the suggestions generated based on the emotions;] [a generation means for generating a shopping list based on current inventory information;] [a display means for displaying the generated shopping list to the user; and] [an emotion input means for inputting the user's emotional state and sending it to the cloud server. This enables users to efficiently manage ingredients and enables personalized dish suggestions based on the user's emotions.

[1025] "Photographing means" refers to the equipment or method used by the user to photograph the ingredients they have purchased.

[1026] "Transmission means" refers to the equipment or method used to transmit captured image data to the cloud server.

[1027] The "analysis means" refers to the equipment or method used to analyze the image data sent to the cloud server and identify the type and volume of ingredients.

[1028] "Storage means" refers to the equipment or method used to store the ingredient information identified as the analysis result in a database or the like.

[1029] The "update means" refers to a device or method used to input the amount of ingredients used by the user based on the stored ingredient information and update the inventory information.

[1030] "Emotion recognition and suggestion means" refers to devices and methods used to recognize a user's emotions and suggest dishes and ingredients based on those emotions.

[1031] "Notification means" refers to the device or method used to notify the user of suggestions generated based on emotions.

[1032] A "generator" is a device or method used to generate a shopping list based on current inventory information.

[1033] "Display means" refers to the device or method used to display the generated shopping list to the user.

[1034] "Emotion input means" refers to the device or method used to input the user's emotional state and transmit it to the cloud server.

[1035] MODE FOR CARRYING OUT THE INVENTION

[1036] This invention is a system that combines an ingredient management system with an emotion engine that recognizes the user's emotions, making recipe suggestions and optimizing the shopping list according to the user's emotions. In addition to purchasing and inventory management and recipe suggestions, this system is characterized by recognizing the user's emotions with the emotion engine and making suggestions based on the user's emotions.

[1037] Food Recognition and Preservation

[1038] User: Take a photo of the ingredients they purchased using their smartphone camera. The user launches the smartphone app, opens the camera function, and takes a photo of the ingredients. For example, they take a photo of one cabbage, three carrots, and 500g of chicken.

[1039] Device: Sends the captured image data to the cloud server. When the user taps the "Send" button, the image data is uploaded to the cloud server via the Internet.

[1040] Server: The received image data is analyzed using AI to identify the type and quantity of ingredients. The server uses the Google Cloud Vision API to analyze the image and extract information such as "1 cabbage, 3 carrots, 500g of chicken."

[1041] Server: The analysis results are saved in a database. The server saves the acquired ingredient information in a MySQL database and reflects it in each user's inventory list.

[1042] Inventory management

[1043] User: Enter the amount of ingredients used for dinner into the app. The user opens the smartphone app and enters the amount of ingredients used (for example, "1 / 2 head of cabbage used").

[1044] Device: Sends the entered usage information to the cloud server. When the user taps the "Send" button, the usage information is sent to the cloud server.

[1045] Server: Updates the inventory information in the database. The server updates the database and changes the amount of cabbage in the inventory list to "1 / 2 piece."

[1046] Emotion Recognition and Suggestions

[1047] User: Accesses the emotion engine through the app and inputs the emotional state of the day. The user opens the smartphone app, accesses the emotion engine, and inputs the emotion of the day (e.g., "tired").

[1048] Device: Sends the user's emotional data to the cloud server. When the user taps the "Send" button, the emotional data is sent to the cloud server.

[1049] Server: The emotion engine analyzes the user's emotions and suggests dishes and ingredients based on those emotions. The server uses the Microsoft Azure Emotion API to analyze emotions and determines that "simple dishes should be suggested," suggesting, for example, "stir-fried cabbage and chicken."

[1050] Shopping suggestions

[1051] User: Launches the app and checks current inventory information to create a shopping list. The user opens the smartphone app and taps the "Check inventory" button.

[1052] Device: Requests current inventory information from the cloud server. The device automatically requests current inventory information from the server.

[1053] Server: Generates a list of dishes that can be suggested based on inventory information and sends it to the device.The server generates a list of dishes that can be suggested based on inventory information (e.g., 1 / 2 cabbage, 3 carrots, 500g of chicken) and sends it to the device.

[1054] Device: Shows the user a list of suggested dishes. The device shows the user a suggestion for "Stir-fried cabbage and chicken."

[1055] User: Selects the dish they want to make from the suggested dishes. The user selects "Stir-fried cabbage and chicken" from the list of suggested dishes.

[1056] Server: Compares all ingredients needed for the selected dish with the current inventory and generates a list of ingredients that are missing. The server compares the required ingredients (1 cabbage, 300g chicken) with the inventory and adds the missing ingredients (1 / 2 cabbage) to the list.

[1057] Device: Displays a list of ingredients that are missing, and the user can check and edit the shopping list. The device displays a list of ingredients that are missing (1 / 2 a cabbage) to the user, and the user can check and edit the shopping list.

[1058] For example, after a user purchases a cabbage, three carrots, and 500g of chicken, the user takes a photo of the item with their smartphone, and the device sends the image to a server, which analyzes it and records it in a database. Next, the user uses half a cabbage for dinner and enters that information into the app. The device sends the input information to the server, and the server updates the inventory to "half a cabbage."

[1059] Next, the user launches the app and checks their current inventory (1 / 2 cabbage, 3 carrots, 500g of chicken) to create a shopping list. The emotion engine recognizes the user's emotion (e.g., tiredness) and suggests an easy-to-make dish, "Stir-fried cabbage and chicken." The server compares the required ingredients (1 cabbage, 300g of chicken) with the inventory and adds the missing 1 / 2 cabbage to the list. The device displays the list of missing ingredients to the user, who can then check and edit the shopping list.

[1060] In this way, by combining emotion engines, it becomes possible to make optimal suggestions based on the user's emotions, enabling more personalized ingredient management and cooking suggestions.

[1061] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1062] Step 1:

[1063] The user takes a photo of the food.

[1064] Input: Purchased ingredients

[1065] Specific operation: The user launches the smartphone app and uses the camera function to take a picture of ingredients (e.g., 1 cabbage, 3 carrots, and 500g of chicken).

[1066] Output: Captured image data

[1067] Step 2:

[1068] The device sends the image data to the cloud server.

[1069] Input: Photographed image data

[1070] Specific operation: When the user taps the "Send" button, the image data is uploaded to a cloud server via the Internet.

[1071] Output: Image data sent to the cloud server

[1072] Step 3:

[1073] The server analyzes the image data.

[1074] Input: Image data sent to the cloud server

[1075] Specific operation: The server analyzes the image data using the Google Cloud Vision API and identifies the type and quantity of ingredients (e.g., "1 cabbage, 3 carrots, 500g chicken").

[1076] Output: Parsed ingredient information

[1077] Step 4:

[1078] The server stores the analysis results in a database.

[1079] Input: Parsed ingredient information

[1080] Specific operation: The server saves ingredient information in a MySQL database and updates each user's inventory list.

[1081] Output: Updated database

[1082] Step 5:

[1083] The user enters the amount of ingredients used into the app.

[1084] Input: Amount of ingredients used for dinner

[1085] Specific operation: The user opens the smartphone app and enters the ingredients used (e.g., "1 / 2 cabbage used").

[1086] Output: Input usage information

[1087] Step 6:

[1088] The device sends usage information to the cloud server.

[1089] Input: Entered usage information

[1090] Specific operation: When the user taps the "Send" button, the usage information is sent to the cloud server.

[1091] Output: Usage information sent to the cloud server

[1092] Step 7:

[1093] The server updates the inventory information in the database.

[1094] Input: Usage information sent to the cloud server

[1095] Specific operation: The server updates the database and corrects the quantity of the relevant ingredient in the inventory list (e.g., changing the quantity of cabbage to "1 / 2 piece").

[1096] Output: Updated inventory information

[1097] Step 8:

[1098] A user accesses the emotion recognition engine and inputs an emotional state.

[1099] Input: Emotional state of the day

[1100] Specific operation: The user opens the smartphone app, accesses the emotion engine, and inputs their emotion for the day (e.g., "tired").

[1101] Output: Input emotion data

[1102] Step 9:

[1103] The device transmits the emotion data to the cloud server.

[1104] Input: Input emotion data

[1105] Specific operation: When the user taps the "Send" button, the emotion data is sent to the cloud server.

[1106] Output: Emotion data sent to the cloud server

[1107] Step 10:

[1108] The server analyzes emotions using an emotion engine and generates suggestions.

[1109] Input: Emotion data sent to the cloud server

[1110] What it does: The server uses the Microsoft Azure Emotion API to analyze emotions and generate dish and ingredient suggestions (e.g., "Stir-fried cabbage and chicken") based on those emotions.

[1111] Output: Emotion-based food and ingredient suggestions

[1112] Step 11:

[1113] The terminal notifies the user of the generated proposal.

[1114] Input: Emotion-based dish and ingredient suggestions

[1115] Specific behavior: The device notifies the user and suggests "Stir-fried cabbage and chicken."

[1116] Output: User notification

[1117] Step 12:

[1118] The user launches the app and checks inventory information.

[1119] Input: Current inventory information

[1120] Specific operation: When a user launches the smartphone app and taps the "Check stock" button, the app displays current stock information.

[1121] Output: User-confirmed inventory information

[1122] Step 13:

[1123] The terminal sends a request for inventory information to the cloud server.

[1124] Input: User-confirmed inventory information

[1125] Specific operation: The terminal automatically requests current inventory information from the server.

[1126] Output: Request sent to cloud server

[1127] Step 14:

[1128] The server generates a list of dishes based on the inventory information and sends it to the terminal.

[1129] Input: Requested inventory information

[1130] Specific operation: The server generates a list of dishes that can be suggested based on inventory information (e.g., 1 / 2 cabbage, 3 carrots, 500g of chicken) and sends it to the terminal.

[1131] Output: The recipe list sent to the user

[1132] Step 15:

[1133] The user selects the dish they want to make from the suggested dishes.

[1134] Input: A list of suggested dishes

[1135] Specific action: The user selects "Stir-fried cabbage and chicken" from a list of suggested dishes on a smartphone app.

[1136] Output: Selected dish items

[1137] Step 16:

[1138] The server compares the ingredients required for the selected dish with stock information and generates a list of ingredients that are in short supply.

[1139] Input: Selected dish item

[1140] Specific operation: The server compares the ingredients required for the selected dish (e.g., "Stir-fried cabbage and chicken") (1 cabbage, 300g chicken) with the current inventory information and adds the missing ingredients (1 / 2 cabbage) to the list.

[1141] Output: List of missing ingredients

[1142] Step 17:

[1143] The terminal displays a list of ingredients that are in short supply to the user.

[1144] Input: List of ingredients in short supply

[1145] Specific operation: The device displays a list of ingredients that are in short supply (e.g., 1 / 2 a cabbage) to the user, and the user checks and edits the shopping list.

[1146] Output: A list of missing ingredients displayed to the user

[1147] (Application example 2)

[1148] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1149] In today's world, users are expected to select foods and products that reflect their individual emotions and conditions. However, conventional systems were unable to provide emotion-based suggestions or inventory management, and the optimization of shopping lists and recipe suggestions did not reflect the user's emotions. This made shopping and recipe selection inconvenient, making it difficult to improve user satisfaction.

[1150] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1151] In this invention, the server includes: [a photographing means for the user to photograph ingredients;] [a transmitting means for transmitting the photographed image data to the cloud server;] [an analysis means for analyzing the image data in the cloud server and identifying the type and amount of ingredients;] [a storing means for saving the ingredient information identified as a result of the analysis;] [an update means for inputting the amount of ingredients used by the user based on the saved ingredient information and updating inventory information;] [an emotion analysis means for inputting user emotion data and analyzing it with an emotion engine;] [a suggestion means for suggesting dishes and products suitable for the user based on the emotion analysis results;] [a generation means for generating a shopping list based on current inventory information and suggested dishes; and] [a display means for displaying the generated shopping list to the user. This makes it possible to generate an optimal shopping list based on the user's emotions and inventory information, and to suggest dishes and products according to emotions.

[1152] The "photography means for the user to photograph ingredients" refers to a camera or other photographic device that records images of ingredients purchased by the user.

[1153] The "transmission means for transmitting the captured image data to the cloud server" refers to a communication device and software for uploading the captured image data to a server on the cloud via the Internet.

[1154] The "analysis means for analyzing image data on a cloud server and identifying the type and quantity of ingredients" refers to a program and hardware for recognizing the type and quantity of ingredients using an image analysis algorithm on a cloud server.

[1155] The "storage means for storing information on ingredients identified as analysis results" refers to a device and software for storing information on analyzed ingredients in a storage medium such as a database.

[1156] The "update means for inputting the amount of ingredients used by the user based on the stored ingredient information and updating inventory information" refers to an interface for inputting the amount of ingredients used by the user, and a program and hardware for updating inventory information based on that data.

[1157] "Emotion analysis means for inputting user emotion data and analyzing it with an emotion engine" refers to an emotion analysis engine and related software and hardware for inputting and interpreting a user's emotional state.

[1158] The "means for suggesting dishes and products suitable for the user based on the results of emotion analysis" refers to an algorithm and related hardware for selecting dishes and products that are likely to be preferred by the user based on the results of emotion analysis.

[1159] "Means for generating a shopping list based on current inventory information and suggested dishes" refers to a program and hardware for identifying ingredients that are in short supply based on current inventory information and automatically creating a shopping list.

[1160] The "display means for displaying the generated shopping list to the user" refers to a device and software for visually displaying the generated shopping list on a display device such as the user's smartphone or personal computer.

[1161] This invention is a system that combines an ingredient management system with an emotion engine that recognizes the user's emotions, making recipe suggestions and optimizing the shopping list according to the user's emotions. In addition to purchasing and inventory management and recipe suggestions, this system is characterized by recognizing the user's emotions with the emotion engine and making suggestions based on the user's emotions.

[1162] Basic system configuration

[1163] Photographing and storing ingredients

[1164] The user takes a photo of the ingredients they have purchased using the camera on their smartphone. The captured image data is sent to a cloud server via the smartphone's transmission means. The cloud server then uses an image analysis algorithm (e.g., Google Cloud Vision, Amazon Rekognition) to identify the type and volume of the ingredients. The ingredient information identified as a result of the analysis is then stored in a database.

[1165] Inventory management

[1166] Users input the amounts of ingredients used for dinner into a smartphone app, and the data is sent to a cloud server, which updates the inventory information in the database.

[1167] Emotion Recognition and Suggestions

[1168] The user inputs their emotional state for the day into the emotion engine via a smartphone app. The emotion data is sent to a cloud server and analyzed by an emotion analysis engine (e.g., Microsoft Azure Cognitive Services, IBM Watson). Based on the results of the emotion analysis, the system makes recommendations for dishes and products that are suitable for the user.

[1169] Generate a shopping list

[1170] The user launches the smartphone app and generates a shopping list based on current inventory information. The server then generates a list of ingredients that are in short supply and displays that information to the user.

[1171] Specific examples

[1172] The user follows the steps described above and after shopping, purchases 1 cabbage, 3 carrots, and 500g of chicken, and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database (1 cabbage, 3 carrots, 500g of chicken). The user eats 1 / 2 a cabbage for dinner and enters that information into the app. The device sends the input information to the server, and the server updates the inventory to "1 / 2 a cabbage."

[1173] Prompt Sentence Examples

[1174] Prompt input for generative AI model:

[1175] 1. "Imagine a flow where a user inputs emotional data, sends that data to a server, and suggests dishes based on the user's emotions."

[1176] 2. "When a user takes a picture of an ingredient, generate a program that analyzes the image and stores it in a database."

[1177] In this way, by combining emotion engines, it becomes possible to make optimal suggestions based on the user's emotions, enabling more personalized ingredient management and cooking suggestions.

[1178] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1179] Step 1:

[1180] The user takes a photo of the ingredients they have purchased using their smartphone camera.

[1181] Input: Image of purchased ingredients

[1182] Output: Captured image data

[1183] How it works: The user takes a photo of an ingredient with their smartphone camera and obtains image data.

[1184] Step 2:

[1185] The device sends the captured image data to a cloud server.

[1186] Input: Captured image data

[1187] Output: Image data transferred to the cloud server

[1188] Operation: Using the device's transmission means, the captured image data is uploaded to a cloud server via the Internet.

[1189] Step 3:

[1190] The server analyzes the image data sent to the cloud and identifies the type and volume of ingredients.

[1191] Input: Image data sent to the cloud server

[1192] Output: Ingredient type and quantity information

[1193] How it works: Using image analysis algorithms on a cloud server, the type and volume of ingredients are recognized from image data (e.g., Google Cloud Vision, Amazon Rekognition).

[1194] Step 4:

[1195] The server stores the ingredient information identified as a result of the analysis in a database.

[1196] Input: Ingredient type and quantity information

[1197] Output: Ingredient information stored in a database

[1198] Operation: The server records the identified ingredient information using a storage means that stores it in a database.

[1199] Step 5:

[1200] The user enters the amount of ingredients used into a smartphone app.

[1201] Input: Amount of ingredients used

[1202] Output: Usage data sent from the smartphone

[1203] How it works: The user uses the app's interface to input the amounts of ingredients used.

[1204] Step 6:

[1205] The terminal transmits the input usage information to the cloud server.

[1206] Input: Usage data

[1207] Output: Usage data transferred to the cloud server

[1208] Operation: The terminal uses the transmission means to upload the input data to the cloud server.

[1209] Step 7:

[1210] The server updates the inventory information in the database.

[1211] Input: Usage data

[1212] Output: Updated inventory information

[1213] How it works: The server updates the inventory information in the database based on the usage data.

[1214] Step 8:

[1215] The user enters their emotional state for the day into a smartphone app.

[1216] Input: Emotional state data

[1217] Output: Emotional state data sent from the smartphone

[1218] How it works: The user uses the app's interface to input their emotional state for the day.

[1219] Step 9:

[1220] The device transmits the emotion data to the cloud server.

[1221] Input: Emotional state data

[1222] Output: Emotional state data transferred to the cloud server

[1223] Operation: The device uses the transmission means to upload emotion data to the cloud server.

[1224] Step 10:

[1225] The server analyzes the user's emotions using an emotion engine.

[1226] Input: Emotional state data

[1227] Output: Emotion analysis results

[1228] How it works: The server uses an emotion analysis engine (e.g., Microsoft Azure Cognitive Services, IBM Watson) to analyze the emotion data.

[1229] Step 11:

[1230] The server will suggest dishes and products suitable for the user based on the results of the emotion analysis.

[1231] Input: Sentiment analysis results, inventory information

[1232] Output: A list of suggested dishes and products

[1233] How it works: The server creates a list of suitable dishes and products based on the results of sentiment analysis and inventory information.

[1234] Step 12:

[1235] The server generates a shopping list based on current inventory information and suggested dishes.

[1236] Input: Proposal list, inventory information

[1237] Output: Shopping list

[1238] How it works: The server compares the suggested list with inventory information and generates a list of ingredients or products that are in short supply.

[1239] Step 13:

[1240] The terminal displays the generated shopping list to the user.

[1241] Input: Shopping list

[1242] Output: A shopping list visually displayed to the user

[1243] Operation: The terminal uses a display means to visually display the shopping list to the user.

[1244] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1245] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1246] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1247] [Third embodiment]

[1248] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1249] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1250] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1251] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1252] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1253] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1254] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1255] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1256] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1257] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1258] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1259] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1260] overview

[1261] This invention is a system that streamlines food inventory management within the home and prevents forgetting to use ingredients or purchasing duplicates. Users take photos of purchased ingredients with their smartphones and send the images to a cloud server, where AI identifies and records the type and volume of ingredients. The data is stored in the cloud and used for inventory management, creating shopping lists, and suggesting recipes.

[1262] Program processing overview

[1263] Food Recognition and Preservation

[1264] User: Takes a photo of the ingredients he / she has purchased using his / her smartphone camera.

[1265] Terminal: Sends captured image data to a cloud server.

[1266] Server: Analyzes the received image data using AI and recognizes the type and volume of ingredients.

[1267] Server: Saves the analysis results in a database.

[1268] Example: A user purchases one cabbage, three carrots, and 500g of chicken, and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database.

[1269] Inventory management

[1270] User: Enters the amount of ingredients used for dinner into the app.

[1271] Terminal: Sends the entered usage information to the cloud server.

[1272] Server: Updates the inventory information in the database.

[1273] Example: A user makes a cabbage salad and uses 1 / 2 a cabbage. The user enters "1 / 2 a cabbage used" into the app, and the server updates the cabbage inventory in the database to "1 / 2 a cabbage."

[1274] Shopping suggestions

[1275] User: Launches the app and creates a shopping list.

[1276] Terminal: Retrieves current inventory information from the cloud.

[1277] Server: Generates a list of dishes that can be suggested based on inventory information and sends it to the device.

[1278] Terminal: Shows the user a list of suggested dishes.

[1279] User: Select the dish they want to make from the suggested dishes.

[1280] Server: Compares the ingredients required for the selected dish with the current inventory information and generates a list of ingredients that are missing.

[1281] Terminal: Displays a list of missing ingredients and allows the user to review and edit the shopping list.

[1282] Example: A user launches an app, and the app retrieves inventory information for "1 / 2 cabbage," "3 carrots," and "500g chicken" from the server. Based on this inventory, the app suggests "stir-fried cabbage and chicken," which the user selects. The server compares the required ingredients (1 cabbage, 300g chicken) with the inventory, and adds the missing 1 / 2 cabbage to the list. The app displays this list to the user, who can then confirm and edit the shopping list.

[1283] Implementation flow

[1284] To implement this system, users use a smartphone app to take a photo of the ingredients they have purchased and send the image to a cloud server via the app. The cloud server then analyzes the image, identifies the type and quantity of ingredients, and stores the information in a database. The app updates inventory information in real time and also reflects the amount of ingredients entered by the user. When shopping, the app also displays suggested dishes based on current inventory information and automatically generates a list of ingredients that are in short supply.

[1285] This allows users to efficiently manage ingredients and receive planned cooking suggestions, thereby reducing food waste and enabling efficient household budget management.

[1286] The processing flow will be explained below.

[1287] Food Recognition and Preservation

[1288] Step 1:

[1289] The user takes a photo of the purchased ingredients using the smartphone camera.

[1290] Step 2:

[1291] The device saves the captured image data and calls an API to send it to the cloud server.

[1292] Step 3:

[1293] The server temporarily stores the received image data and passes the data to an AI model for image analysis.

[1294] Step 4:

[1295] The server uses an AI model to analyze the image data and identify the type and quantity of ingredients, for example, "1 cabbage," "3 carrots," and "500g of chicken."

[1296] Step 5:

[1297] The server saves the analysis results in a database and records them as "1 cabbage," "3 carrots," and "500g of chicken."

[1298] Inventory management

[1299] Step 6:

[1300] The user enters the amount of ingredients used for dinner into the app.

[1301] Step 7:

[1302] The device calls an API to send the entered usage information to the cloud server.

[1303] Step 8:

[1304] The server temporarily stores the received usage data and updates the inventory information in the database.

[1305] Step 9:

[1306] The server reduces the inventory of specific ingredients in the database and updates it to "1 / 2 cabbage," "3 carrots," and "500g chicken."

[1307] Shopping suggestions

[1308] Step 10:

[1309] A user launches the app and checks current inventory information to create a shopping list.

[1310] Step 11:

[1311] The device requests current inventory information from the cloud server.

[1312] Step 12:

[1313] The server retrieves current inventory information from the database and sends it to the terminal.

[1314] Step 13:

[1315] Based on the inventory information received by the terminal, a list of dishes that can be suggested to the user is generated and displayed.

[1316] Step 14:

[1317] The user selects the dish they want to make from the suggested dishes.

[1318] Step 15:

[1319] The server compares all ingredients required for the selected dish with current inventory information and generates a list of ingredients that are in short supply.

[1320] Step 16:

[1321] The server transmits a list of ingredients that are in short supply to the terminal, which then displays it to the user.

[1322] Step 17:

[1323] The user checks the displayed list of ingredients that are in short supply and creates and edits a shopping list.

[1324] Specific examples of implementation

[1325] Step 18:

[1326] After shopping, the user purchases one cabbage, three carrots, and 500g of chicken and takes a photo of it with their smartphone.

[1327] Step 19:

[1328] The device sends the image to the server, which analyzes it and records it in a database ("1 cabbage," "3 carrots," "500g chicken").

[1329] Step 20:

[1330] A user eats 1 / 2 a cabbage for dinner and enters that information into the app.

[1331] Step 21:

[1332] The terminal sends the input information to the server, and the server updates the inventory to "1 / 2 cabbage."

[1333] Step 22:

[1334] Before the next shopping trip, the user launches the app, which retrieves the current inventory information. ("1 / 2 cabbage")

[1335] Step 23:

[1336] The server sends inventory information for "1 / 2 cabbage," "3 carrots," and "500g of chicken" to the terminal, and the terminal then suggests dishes based on that information.

[1337] Step 24:

[1338] The user selects "Stir-fried cabbage and chicken" and adds "1 / 2 cabbage" to the list, for which the server is short.

[1339] Step 25:

[1340] The device displays a list of ingredients that are in short supply, and the user can check and edit the shopping list.

[1341] These are the specific processing steps, from recognizing and storing ingredients to inventory management and suggestions when shopping.

[1342] Example 1

[1343] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1344] In most cases, food inventory management in the home is done manually, resulting in frequent forgetting to use ingredients or duplicate purchases. Furthermore, determining what to buy next requires knowing the current inventory status and recipes for the necessary dishes, which is time-consuming. Furthermore, there are often times when people are out of ideas for cooking, so an efficient way to solve these problems is needed.

[1345] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1346] In this invention, the server includes: [a photographing means for the user to photograph ingredients;] [a transmitting means for transmitting the photographed image data to a network server;] [an analyzing means for analyzing the image data in the network server and identifying the type and amount of ingredients;] [a storing means for saving the ingredient information identified as a result of the analysis;] [an updating means for inputting the amount of ingredients used by the user based on the stored ingredient information and updating inventory information;] [a generating means for generating a shopping list based on current inventory information;] [a display means for displaying the generated shopping list to the user;] [an automatic output means for generating a list of suggested dishes; and] [a list generating means for generating a list of missing ingredients based on the suggested dishes. This enables ingredient inventory management, prevention of forgotten ingredients and duplicate purchases, efficient shopping list creation, and dish suggestions.

[1347] The term "photography means for a user to photograph ingredients" refers to a device or application that a user uses to record the ingredients they have purchased as digital images.

[1348] The "transmission means for transmitting the photographed image data to the network server" refers to a device or application for transmitting the photographed image data of ingredients to a server on the cloud via the Internet.

[1349] "Analysis means for analyzing image data on a network server and identifying the type and volume of ingredients" refers to software or algorithms for analyzing received image data and automatically identifying the type and volume of ingredients.

[1350] "Storage means for storing information on ingredients identified as a result of the analysis" refers to a device or application for recording data on the type and volume of ingredients obtained through the analysis in storage such as a database.

[1351] "Means for updating inventory information by inputting the amount of ingredients used by the user based on the stored ingredient information" refers to a device or application for correcting and updating inventory data in real time based on the amount of ingredients used input by the user through the application.

[1352] "Means for generating a shopping list based on current inventory information" refers to a device or application that references recorded inventory information and automatically creates a list of new ingredients to be purchased.

[1353] The "display means for displaying the generated shopping list to the user" refers to a device or application for visually displaying the automatically generated shopping list on the display of the user's device, for example.

[1354] "Automatic output means for generating a list of suggested dishes" refers to a device or application that automatically generates and presents a list of dishes that can be made based on stored inventory information and data such as user preferences.

[1355] "List generation means for generating a list of missing ingredients based on a proposed dish" refers to a device or application that compares ingredients required for the proposed dish with inventory data and lists missing ingredients.

[1356] MODE FOR CARRYING OUT THE INVENTION

[1357] This invention is a system designed to improve the efficiency of food ingredient inventory management in the home, prevent forgetting to use ingredients or purchasing the same ingredients twice, and provide recipe suggestions based on a well-planned recipe. Specific embodiments of this system are described below.

[1358] A user uses a smartphone to take a photo of the ingredients they have purchased. The smartphone's camera (photography means) is used to capture high-resolution images of the ingredients. For example, imagine a case where a user takes a photo of one cabbage, three carrots, and 500g of chicken upon returning home from the supermarket. The captured image data is sent to a network server on the cloud via an application on the smartphone (transmission means).

[1359] The network server analyzes the received image data. Specifically, it uses image recognition services such as Google Cloud Vision API and Amazon Rekognition to identify the type and volume of ingredients in the image (analysis means). The data resulting from the analysis is stored in a cloud database (e.g., Google Firebase or Amazon DynamoDB) (storage means).

[1360] When a user uses an ingredient, they input the amount used into the smartphone application. For example, if they input "1 / 2 cabbage used," the application sends this information to the cloud server, and the inventory information in the cloud database is updated (update means).

[1361] Furthermore, when a user launches the app to go shopping, the application retrieves current inventory information from the cloud. A shopping list is automatically generated based on the current inventory information (generation means). The generated shopping list is displayed to the user, allowing the user to use the list as a reference when shopping (display means).

[1362] The system also has the ability to suggest dishes using a generative AI model. For example, it uses a language model such as OpenAI's GPT-3 to generate a list of suggested dishes based on inventory information and displays it to the user (automatic output means). When the user selects one of the suggested dishes, the system compares the ingredients needed for that dish with current inventory information and automatically generates a list of ingredients that are missing (list generation means).

[1363] The above process allows users to efficiently manage ingredients and plan their shopping and cooking, thereby reducing food waste and enabling efficient management of household finances.

[1364] For example, suppose the user enters the prompt text as follows:

[1365] "I have half a cabbage, three carrots, and 500g of chicken left in my fridge. What kind of dish can I make with these ingredients? Can you also tell me how to make it?"

[1366] Based on these prompts, the generative AI model will suggest optimal dishes and cooking methods to the user, allowing them to expand the range of cooking they can do at home.

[1367] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1368] Program processing flow

[1369] Step 1:

[1370] The user takes a photo of the ingredients

[1371] Specific operation: The user uses the smartphone camera to take a picture of the ingredients they purchased. For example, they take a picture of one cabbage, three carrots, and 500g of chicken lined up.

[1372] Input: An image of the food captured by the camera.

[1373] Output: Image data of ingredients stored on the smartphone.

[1374] Step 2:

[1375] The device sends the image to the cloud server.

[1376] Specific operation: A smartphone app sends captured image data to a cloud server via the Internet. For example, the app uploads the images to AWS S3.

[1377] Input: Image data of ingredients stored on your smartphone.

[1378] Output: Image data sent to the cloud server.

[1379] Step 3:

[1380] The server analyzes the image and recognizes the type and volume of ingredients.

[1381] Specific operation: The cloud server analyzes the received image data using Google Cloud Vision API and Amazon Rekognition to identify the type and quantity of ingredients. For example, it might recognize it as one cabbage, three carrots, and 500g of chicken.

[1382] Input: Image data stored on a cloud server.

[1383] Output: Analysis results including ingredient type and volume.

[1384] Step 4:

[1385] The server stores the analysis results in a database

[1386] Specific operation: The food ingredient information from the analysis results is saved in a cloud database (e.g., Google Firebase or Amazon DynamoDB).

[1387] Input: Analysis results including ingredient type and volume.

[1388] Output: Ingredient information recorded in a cloud database.

[1389] Step 5:

[1390] The user enters the amount of ingredients used into the app.

[1391] Specific operation: The user inputs the amount of ingredients used for dinner into the smartphone app. For example, the user inputs that half a cabbage was used.

[1392] Input: Amount of ingredients used.

[1393] Output: Usage information entered into the smartphone app.

[1394] Step 6:

[1395] The device sends usage information to the cloud server.

[1396] Specific operation: The smartphone app sends the entered usage information to a cloud server. For example, the app sends usage data to Firebase Firestore.

[1397] Input: Usage information stored in the smartphone app.

[1398] Output: Usage information sent to the cloud server.

[1399] Step 7:

[1400] The server updates the inventory information in the database

[1401] Specific operation: The cloud server updates the inventory information in the cloud database in real time based on the received usage information. For example, the inventory of cabbage is updated to 1 / 2.

[1402] Input: Usage information sent to the cloud server.

[1403] Output: Updated inventory information from the cloud database.

[1404] Step 8:

[1405] The user launches the app and creates a shopping list

[1406] Specific action: The user launches the smartphone app and accesses the screen for creating a shopping list.

[1407] Input: The app launch operation.

[1408] Output: Display of the app's shopping list creation screen.

[1409] Step 9:

[1410] The device retrieves current inventory information from the cloud.

[1411] Specific operation: The smartphone app accesses a cloud database to get the latest inventory information, for example, from Firebase Firestore.

[1412] Input: A retrieval request to the cloud database.

[1413] Output: The latest inventory information obtained.

[1414] Step 10:

[1415] The server generates recipe suggestions based on inventory information and sends them to the device.

[1416] Specific operation: The cloud server uses the acquired inventory information to generate a list of dishes that are best suited to the user using a language model such as GPT-3, and sends this list to the smartphone app.

[1417] Input: Latest inventory information.

[1418] Output: A list of suggested dishes.

[1419] Step 11:

[1420] The device displays suggested dishes

[1421] Specific operation: The smartphone app displays the list of suggested dishes received from the server to the user. For example, it displays a list including "stir-fried cabbage and chicken."

[1422] Input: A list of dish suggestions received from the server.

[1423] Output: A list of cooking suggestions displayed on a smartphone app.

[1424] Step 12:

[1425] The user selects the dish they want to make.

[1426] Specific operation: The user selects the dish they want to make from the list of suggested dishes. For example, they select "Stir-fried cabbage and chicken."

[1427] Input: User's food selection.

[1428] Output: Selected dish information.

[1429] Step 13:

[1430] The server compares the required ingredients with the available stock and generates a list of ingredients that are in short supply.

[1431] Specific operation: The cloud server compares the list of ingredients needed for the selected dish with the current inventory information and lists any ingredients that are missing. For example, it recognizes that 1 / 2 a cabbage is missing.

[1432] Input: Selected dish information and current inventory information.

[1433] Output: A list of missing ingredients.

[1434] Step 14:

[1435] The device displays a list of ingredients that are missing, and the user can check and edit the shopping list.

[1436] Specific operation: The smartphone app displays the list of ingredients that are in short supply received from the server to the user, who then checks and edits the list.

[1437] Input: Missing ingredient list.

[1438] Output: A list of ingredients that are in short supply, displayed on the smartphone app. Users can confirm and edit the list.

[1439] (Application example 1)

[1440] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1441] Inventory management of ingredients and products at home or in a brick-and-mortar store is complicated, and manual management can easily lead to forgetting to use ingredients or purchasing duplicates. Furthermore, if product inventory is not managed properly, there is a high risk of unsold items or shortages. This increases food waste and opportunity losses, hindering efficient operations.

[1442] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1443] In this invention, the server includes [a photographing means for the user to photograph ingredients], [a transmission means for transmitting the photographed image data to the cloud server], [an analysis means for analyzing the image data in the cloud server and identifying the type and amount of ingredients], [a storage means for saving the ingredient information identified as a result of the analysis], [an update means for inputting the amount of ingredients used by the user based on the saved ingredient information and updating inventory information], [a generation means for generating a shopping list based on current inventory information], [a display means for displaying the generated shopping list to the user], [a suggestion means for suggesting products to the user in conjunction with inventory], and [a detection means for detecting missing ingredients or products]. This makes it possible to automatically manage ingredient and product inventory and make appropriate suggestions to the user.

[1444] "Photographing means" refers to a device or method that allows a user to photograph ingredients or products.

[1445] The "transmission means" is a device or method for transmitting captured image data to a cloud server.

[1446] The "analysis means" is a device or method for analyzing image data on a cloud server and identifying the type and volume of ingredients or products.

[1447] The "storage means" refers to a device or method for storing the ingredients and product information identified as analysis results in a database.

[1448] The "update means" is a device or method for inputting the amount of ingredients or products used by the user based on the stored ingredient and product information and updating the inventory information.

[1449] The "generation means" is a device or method for generating a shopping list or a recommended product list based on current inventory information.

[1450] The "display means" refers to a device or method for displaying the generated shopping list or recommended product list to the user.

[1451] The "suggestion means" is a device or method for suggesting products to users in conjunction with inventory.

[1452] "Detection means" refers to a device or method for detecting missing ingredients or products.

[1453] Overall system configuration

[1454] The system of this invention is broadly composed of a photographing means, a transmitting means, a analyzing means, a saving means, a updating means, a generating means, a displaying means, a suggesting means, and a detecting means. These means work together to realize inventory management of ingredients and products and shopping support based on that inventory management.

[1455] Program Overview

[1456] Filming method

[1457] The user uses a smartphone to take a photo of the ingredients or products they have purchased, and the device captures the image using the smartphone's camera.

[1458] Transmission method

[1459] The captured image data is sent to a cloud server. This device or software uses the smartphone's communication functions (Wi-Fi, mobile data communication, etc.).

[1460] Analysis means

[1461] The cloud server analyzes the captured image data using an AI model (for example, a deep learning model such as TensorFlow) to identify the type and volume of specific ingredients or products. This analysis uses image recognition technology.

[1462] Preservation means

[1463] The analyzed ingredient and product information is stored in a cloud database (such as AWS RDS or Google Cloud Firestore). The type, size, and photo date and time of each ingredient or product are recorded in the database.

[1464] Update method

[1465] Users input the quantities of ingredients and products used into the application, and the information is sent to a cloud server, which updates the inventory information in a database based on the input.

[1466] generation means

[1467] Based on the current inventory information, the cloud server generates a shopping list and a list of recommended products, allowing users to efficiently create lists of ingredients and products they need.

[1468] Display means

[1469] The generated shopping list and recommended product list are displayed to the user through a smartphone application. The user interface can be built using a cross-platform framework such as React Native.

[1470] Proposal means

[1471] The server connects with current inventory information and suggests suitable products and recipes to users based on the latest inventory information and the user's past purchase history.

[1472] Detection Method

[1473] Every time inventory data is updated, the cloud server compares it with the existing inventory data to detect any missing ingredients or products, which are then automatically added to the user's shopping list.

[1474] Specific examples

[1475] For example, when a store employee places new tomatoes on a shelf, they take a picture of the tomato using the app, which then uploads the image to the cloud. An AI model on the cloud server analyzes the image, identifies the type and quantity of tomatoes, and stores the information in a database. When a customer opens the app, fresh tomatoes are recommended based on the latest inventory information.

[1476] Prompt Sentence Examples

[1477] "Please upload images of newly arrived tomatoes to the cloud server. The AI ​​on the server side will identify the quantity and quality of the tomatoes, update the inventory information, and reflect it as recommendations on the customer app."

[1478] As a result, the system of the present invention can improve the efficiency of inventory management of food ingredients and products at home or in a physical store, and can support customers in making appropriate product suggestions and generating shopping lists.

[1479] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1480] Step 1:

[1481] The user takes a photo of the food or product using a smartphone.

[1482] Input: Image of ingredients or products

[1483] Output: Image data stored on a smartphone

[1484] Specific actions: The user launches the smartphone camera app, frames the food or product they are purchasing, and presses the shutter button. When doing so, they take as clear an image as possible, taking care to ensure that the food or product is clearly identifiable.

[1485] Step 2:

[1486] The device sends the captured image data to a cloud server.

[1487] Input: Captured image data

[1488] Output: Image data transferred to the cloud server

[1489] What happens: The smartphone app compresses the captured image file and sends an HTTP POST request to a cloud server over the internet, including user authentication information.

[1490] Step 3:

[1491] The server receives the image data at the cloud server and analyzes it using the analysis means.

[1492] Input: Image data transferred to the cloud server

[1493] Output: Analyzed ingredients and product types and volume data

[1494] How it works: The server stores the received image data on disk and performs image recognition using an AI model (e.g., TensorFlow). This AI model identifies the type and volume of ingredients or products from the image and extracts this information as metadata.

[1495] Step 4:

[1496] The server stores the analysis results in a database.

[1497] Input: Analyzed ingredient and product type and volume data

[1498] Output: Analysis result data stored in a database

[1499] Specific operation: The server converts the analysis results into structured data and executes an INSERT query to a cloud database (e.g., AWS RDS, Google Cloud Firestore). The stored data includes the type of ingredients or products, volume, analysis date and time, etc.

[1500] Step 5:

[1501] The user enters the amounts of ingredients and products used into the application.

[1502] Input: Usage information

[1503] Output: Usage data transferred to the cloud server

[1504] Specific operation: The user opens a form on the smartphone application to input the amount of ingredients and products used, enters the necessary information, for example, "1 / 2 cabbage used," and presses the send button. This information is sent to the cloud server.

[1505] Step 6:

[1506] The server updates the inventory information in the database based on the input usage data.

[1507] Input: Usage data transferred to the cloud server

[1508] Output: Updated inventory information

[1509] Specific operation: The server analyzes the received usage data and reduces the corresponding inventory data. It updates the inventory information in the database using an UPDATE query. The updated inventory information reflects the remaining amount of the ingredient or product.

[1510] Step 7:

[1511] The server generates shopping lists and recommended product lists based on current inventory information.

[1512] Input: Updated inventory information

[1513] Output: Generated shopping list and recommended products list

[1514] Specific operation: The server identifies ingredients and products that are in short supply based on the latest inventory data, creates a list of items to purchase, and adds recommended products and recipes to the list, taking into account the user's past purchase history and trend data.

[1515] Step 8:

[1516] The terminal displays the generated shopping list and recommended product list to the user.

[1517] Input: Generated shopping list or recommended product list

[1518] Output: List displayed on smartphone

[1519] What it does: The smartphone application analyzes the list data received from the server and displays it in an easy-to-read format on the user interface, allowing the user to plan their shopping needs.

[1520] Step 9:

[1521] The server works in conjunction with inventory information to suggest products to users.

[1522] Input: Current inventory information, user's past purchase history

[1523] Output: Suggested products and recipe information

[1524] What it does: The server cross-references the inventory database with the user's purchasing history to generate product and recipe recommendations that are best suited to the user. These recommendations are listed in order of most relevant and notified to the user.

[1525] Step 10:

[1526] The server detects ingredients or products that are in short supply and adds them to the shopping list.

[1527] Input: Current inventory information

[1528] Output: Updated shopping list

[1529] What it does: The server periodically scans inventory data to identify ingredients or products that are running low, automatically adding them to the shopping list and notifying the user.

[1530] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1531] overview

[1532] This invention is a system that combines an ingredient management system with an emotion engine that recognizes the user's emotions, making recipe suggestions and optimizing the shopping list according to the user's emotions. In addition to purchasing and inventory management and recipe suggestions, this system is characterized by recognizing the user's emotions with the emotion engine and making suggestions based on the user's emotions.

[1533] Program processing overview

[1534] Food Recognition and Preservation

[1535] User: Takes a photo of the purchased ingredients using the smartphone camera.

[1536] Terminal: Sends captured image data to a cloud server.

[1537] Server: Analyzes the received image data using AI and recognizes the type and volume of ingredients.

[1538] Server: Saves the analysis results in a database.

[1539] Example: A user purchases one cabbage, three carrots, and 500g of chicken, and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database.

[1540] Inventory management

[1541] User: Enters the amount of ingredients used for dinner into the app.

[1542] Terminal: Sends the entered usage information to the cloud server.

[1543] Server: Updates the inventory information in the database.

[1544] Example: A user makes a cabbage salad and uses 1 / 2 a cabbage. The user enters "1 / 2 a cabbage used" into the app, and the server updates the cabbage inventory in the database to "1 / 2 a cabbage."

[1545] Emotion Recognition and Suggestions

[1546] User: Accesses the emotion engine through the app and inputs their emotional state for the day.

[1547] Terminal: Sends the user's emotional data to the cloud server.

[1548] Server: The emotion engine analyzes the user's emotions and suggests dishes and ingredients based on those emotions.

[1549] For example, if the user is tired, easy meals will be suggested, and if the user is energetic, more challenging meals will be suggested.

[1550] Shopping suggestions

[1551] User: Launches the app and checks current inventory information to create a shopping list.

[1552] Terminal: Requests current inventory information from the cloud server.

[1553] Server: Generates a list of dishes that can be suggested based on inventory information and sends it to the device.

[1554] Terminal: Shows the user a list of suggested dishes.

[1555] User: Select the dish they want to make from the suggested dishes.

[1556] Server: Compares all ingredients required for the selected dish with current inventory and generates a list of ingredients that are missing.

[1557] Terminal: Displays a list of missing ingredients and allows the user to review and edit the shopping list.

[1558] Example: A user launches the app and retrieves current inventory information (1 / 2 cabbage, 3 carrots, 500g chicken). The emotion engine recognizes that the user is tired and suggests a simple "stir-fried cabbage and chicken." The server adds the missing 1 / 2 cabbage to the list, and the device displays the shopping list.

[1559] Specific examples of implementation

[1560] After shopping, a user purchases 1 cabbage, 3 carrots, and 500g of chicken and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database (1 cabbage, 3 carrots, 500g of chicken). The user eats 1 / 2 a cabbage for dinner and enters that information into the app. The device sends the input information to the server, which updates the inventory to "1 / 2 a cabbage."

[1561] Next, the user launches the app and checks the current inventory information (1 / 2 cabbage, 3 carrots, 500g of chicken) to create a shopping list. The emotion engine recognizes the user's emotion (e.g., tiredness) and suggests an easy-to-make dish, "Stir-fried cabbage and chicken." The server compares the required ingredients (1 cabbage, 300g of chicken) with the inventory and adds the missing 1 / 2 cabbage to the list. The device displays the list of missing ingredients, and the user can check and edit the shopping list.

[1562] In this way, by combining emotion engines, it becomes possible to make optimal suggestions based on the user's emotions, enabling more personalized ingredient management and cooking suggestions.

[1563] The processing flow will be explained below.

[1564] Processing flow of a system that combines emotion engines

[1565] Emotion Recognition and Suggestions

[1566] Step 1:

[1567] The user launches the app and enters their emotional state for the day.

[1568] Step 2:

[1569] The device transmits the user's emotional data to a cloud server.

[1570] Step 3:

[1571] The server passes the received emotion data to the emotion engine, which analyzes the user's emotional state.

[1572] Step 4:

[1573] The server generates a list of dishes according to the emotion based on the analysis results of the emotion engine.

[1574] Food Recognition and Preservation

[1575] Step 5:

[1576] The user takes a photo of the purchased ingredients using the smartphone camera.

[1577] Step 6:

[1578] The device saves the captured image data and calls an API to send it to the cloud server.

[1579] Step 7:

[1580] The server temporarily stores the received image data and passes the data to an AI model for image analysis.

[1581] Step 8:

[1582] The server uses an AI model to analyze the image data and identify the type and quantity of ingredients, for example, "1 cabbage," "3 carrots," and "500g of chicken."

[1583] Step 9:

[1584] The server saves the analysis results in a database and records them as "1 cabbage," "3 carrots," and "500g of chicken."

[1585] Inventory management

[1586] Step 10:

[1587] The user enters the amount of ingredients used for dinner into the app.

[1588] Step 11:

[1589] The device calls an API to send the entered usage information to the cloud server.

[1590] Step 12:

[1591] The server temporarily stores the received usage data and updates the inventory information in the database.

[1592] Step 13:

[1593] The server reduces the inventory of specific ingredients in the database and updates it to "1 / 2 cabbage," "3 carrots," and "500g chicken."

[1594] Shopping suggestions

[1595] Step 14:

[1596] A user launches the app and checks current inventory information to create a shopping list.

[1597] Step 15:

[1598] The device requests current inventory information from the cloud server.

[1599] Step 16:

[1600] The server retrieves current inventory information from the database and sends it to the terminal.

[1601] Step 17:

[1602] Based on the inventory information received by the terminal, a list of dishes that can be suggested to the user is generated and displayed.

[1603] Step 18:

[1604] The user selects the dish they want to make from the suggested dishes.

[1605] Step 19:

[1606] The server compares all ingredients required for the selected dish with current inventory information and generates a list of ingredients that are in short supply.

[1607] Step 20:

[1608] The server transmits a list of ingredients that are in short supply to the terminal, which then displays it to the user.

[1609] Step 21:

[1610] The user checks the displayed list of ingredients that are in short supply and creates and edits a shopping list.

[1611] Specific examples of implementation

[1612] Step 22:

[1613] After shopping, the user purchases one cabbage, three carrots, and 500g of chicken and takes a photo of it with their smartphone.

[1614] Step 23:

[1615] The device sends the image to the server, which analyzes it and records it in a database ("1 cabbage," "3 carrots," "500g chicken").

[1616] Step 24:

[1617] A user eats 1 / 2 a cabbage for dinner and enters that information into the app.

[1618] Step 25:

[1619] The terminal sends the input information to the server, and the server updates the inventory to "1 / 2 cabbage."

[1620] Step 26:

[1621] The user launches the app before their next shopping trip, and the app retrieves the current inventory information ("1 / 2 cabbage," "3 carrots," and "500g chicken").

[1622] Step 27:

[1623] The device inputs the user's emotional state into a cloud server, which then analyzes the data.

[1624] Step 28:

[1625] The emotion engine recognizes that the user is tired and suggests an easy-to-make dish: stir-fried cabbage and chicken.

[1626] Step 29:

[1627] The server compares the ingredients needed ("1 cabbage" and "300g chicken") with the inventory and adds the missing 1 / 2 cabbage to the list.

[1628] Step 30:

[1629] The device displays a list of ingredients that are in short supply, and the user can check and edit the shopping list.

[1630] The above is the specific processing flow of a system that combines an emotion engine. By taking the user's emotions into consideration, more personalized suggestions become possible, which is expected to improve user satisfaction.

[1631] Example 2

[1632] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1633] In today's busy lifestyles, it is difficult for users to efficiently manage ingredient inventory, select dishes, and receive recipe suggestions tailored to their mood of the day. Conventional systems require users to manually update inventory information and manage ingredient usage, which is tedious and often inaccurate. Furthermore, they lack the ability to suggest dishes based on the user's emotions, resulting in a non-personalized user experience. Therefore, there is a need for a system that allows users to efficiently manage ingredients, inventory, recognize emotions, and receive optimal recipe suggestions.

[1634] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1635] In this invention, the server includes: [a photographing means for the user to photograph ingredients;] [a transmission means for sending the photographed image data to the cloud server;] [an analysis means for analyzing the image data in the cloud server and identifying the type and amount of ingredients;] [a storage means for saving ingredient information identified as a result of the analysis;] [an update means for inputting the amount of ingredients used by the user based on the saved ingredient information and updating inventory information;] [an emotion recognition and suggestion means for recognizing the user's emotions and suggesting dishes and ingredients based on the emotions;] [a notification means for notifying the user of the suggestions generated based on the emotions;] [a generation means for generating a shopping list based on current inventory information;] [a display means for displaying the generated shopping list to the user; and] [an emotion input means for inputting the user's emotional state and sending it to the cloud server. This enables users to efficiently manage ingredients and enables personalized dish suggestions based on the user's emotions.

[1636] "Photographing means" refers to the equipment or method used by the user to photograph the ingredients they have purchased.

[1637] "Transmission means" refers to the equipment or method used to transmit captured image data to the cloud server.

[1638] The "analysis means" refers to the equipment or method used to analyze the image data sent to the cloud server and identify the type and volume of ingredients.

[1639] "Storage means" refers to the equipment or method used to store the ingredient information identified as the analysis result in a database or the like.

[1640] The "update means" refers to a device or method used to input the amount of ingredients used by the user based on the stored ingredient information and update the inventory information.

[1641] "Emotion recognition and suggestion means" refers to devices and methods used to recognize a user's emotions and suggest dishes and ingredients based on those emotions.

[1642] "Notification means" refers to the device or method used to notify the user of suggestions generated based on emotions.

[1643] A "generator" is a device or method used to generate a shopping list based on current inventory information.

[1644] "Display means" refers to the device or method used to display the generated shopping list to the user.

[1645] "Emotion input means" refers to the device or method used to input the user's emotional state and transmit it to the cloud server.

[1646] MODE FOR CARRYING OUT THE INVENTION

[1647] This invention is a system that combines an ingredient management system with an emotion engine that recognizes the user's emotions, making recipe suggestions and optimizing the shopping list according to the user's emotions. In addition to purchasing and inventory management and recipe suggestions, this system is characterized by recognizing the user's emotions with the emotion engine and making suggestions based on the user's emotions.

[1648] Food Recognition and Preservation

[1649] User: Take a photo of the ingredients they purchased using their smartphone camera. The user launches the smartphone app, opens the camera function, and takes a photo of the ingredients. For example, they take a photo of one cabbage, three carrots, and 500g of chicken.

[1650] Device: Sends the captured image data to the cloud server. When the user taps the "Send" button, the image data is uploaded to the cloud server via the Internet.

[1651] Server: The received image data is analyzed using AI to identify the type and quantity of ingredients. The server uses the Google Cloud Vision API to analyze the image and extract information such as "1 cabbage, 3 carrots, 500g of chicken."

[1652] Server: The analysis results are saved in a database. The server saves the acquired ingredient information in a MySQL database and reflects it in each user's inventory list.

[1653] Inventory management

[1654] User: Enter the amount of ingredients used for dinner into the app. The user opens the smartphone app and enters the amount of ingredients used (for example, "1 / 2 head of cabbage used").

[1655] Device: Sends the entered usage information to the cloud server. When the user taps the "Send" button, the usage information is sent to the cloud server.

[1656] Server: Updates the inventory information in the database. The server updates the database and changes the amount of cabbage in the inventory list to "1 / 2 piece."

[1657] Emotion Recognition and Suggestions

[1658] User: Accesses the emotion engine through the app and inputs the emotional state of the day. The user opens the smartphone app, accesses the emotion engine, and inputs the emotion of the day (e.g., "tired").

[1659] Device: Sends the user's emotional data to the cloud server. When the user taps the "Send" button, the emotional data is sent to the cloud server.

[1660] Server: The emotion engine analyzes the user's emotions and suggests dishes and ingredients based on those emotions. The server uses the Microsoft Azure Emotion API to analyze emotions and determines that "simple dishes should be suggested," suggesting, for example, "stir-fried cabbage and chicken."

[1661] Shopping suggestions

[1662] User: Launches the app and checks current inventory information to create a shopping list. The user opens the smartphone app and taps the "Check inventory" button.

[1663] Device: Requests current inventory information from the cloud server. The device automatically requests current inventory information from the server.

[1664] Server: Generates a list of dishes that can be suggested based on inventory information and sends it to the device.The server generates a list of dishes that can be suggested based on inventory information (e.g., 1 / 2 cabbage, 3 carrots, 500g of chicken) and sends it to the device.

[1665] Device: Shows the user a list of suggested dishes. The device shows the user a suggestion for "Stir-fried cabbage and chicken."

[1666] User: Selects the dish they want to make from the suggested dishes. The user selects "Stir-fried cabbage and chicken" from the list of suggested dishes.

[1667] Server: Compares all ingredients needed for the selected dish with the current inventory and generates a list of ingredients that are missing. The server compares the required ingredients (1 cabbage, 300g chicken) with the inventory and adds the missing ingredients (1 / 2 cabbage) to the list.

[1668] Device: Displays a list of ingredients that are missing, and the user can check and edit the shopping list. The device displays a list of ingredients that are missing (1 / 2 a cabbage) to the user, and the user can check and edit the shopping list.

[1669] For example, after a user purchases a cabbage, three carrots, and 500g of chicken, the user takes a photo of the item with their smartphone, and the device sends the image to a server, which analyzes it and records it in a database. Next, the user uses half a cabbage for dinner and enters that information into the app. The device sends the input information to the server, and the server updates the inventory to "half a cabbage."

[1670] Next, the user launches the app and checks their current inventory (1 / 2 cabbage, 3 carrots, 500g of chicken) to create a shopping list. The emotion engine recognizes the user's emotion (e.g., tiredness) and suggests an easy-to-make dish, "Stir-fried cabbage and chicken." The server compares the required ingredients (1 cabbage, 300g of chicken) with the inventory and adds the missing 1 / 2 cabbage to the list. The device displays the list of missing ingredients to the user, who can then check and edit the shopping list.

[1671] In this way, by combining emotion engines, it becomes possible to make optimal suggestions based on the user's emotions, enabling more personalized ingredient management and cooking suggestions.

[1672] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1673] Step 1:

[1674] The user takes a photo of the food.

[1675] Input: Purchased ingredients

[1676] Specific operation: The user launches the smartphone app and uses the camera function to take a picture of ingredients (e.g., 1 cabbage, 3 carrots, and 500g of chicken).

[1677] Output: Captured image data

[1678] Step 2:

[1679] The device sends the image data to the cloud server.

[1680] Input: Photographed image data

[1681] Specific operation: When the user taps the "Send" button, the image data is uploaded to a cloud server via the Internet.

[1682] Output: Image data sent to the cloud server

[1683] Step 3:

[1684] The server analyzes the image data.

[1685] Input: Image data sent to the cloud server

[1686] Specific operation: The server analyzes the image data using the Google Cloud Vision API and identifies the type and quantity of ingredients (e.g., "1 cabbage, 3 carrots, 500g chicken").

[1687] Output: Parsed ingredient information

[1688] Step 4:

[1689] The server stores the analysis results in a database.

[1690] Input: Parsed ingredient information

[1691] Specific operation: The server saves ingredient information in a MySQL database and updates each user's inventory list.

[1692] Output: Updated database

[1693] Step 5:

[1694] The user enters the amount of ingredients used into the app.

[1695] Input: Amount of ingredients used for dinner

[1696] Specific operation: The user opens the smartphone app and enters the ingredients used (e.g., "1 / 2 cabbage used").

[1697] Output: Input usage information

[1698] Step 6:

[1699] The device sends usage information to the cloud server.

[1700] Input: Entered usage information

[1701] Specific operation: When the user taps the "Send" button, the usage information is sent to the cloud server.

[1702] Output: Usage information sent to the cloud server

[1703] Step 7:

[1704] The server updates the inventory information in the database.

[1705] Input: Usage information sent to the cloud server

[1706] Specific operation: The server updates the database and corrects the quantity of the relevant ingredient in the inventory list (e.g., changing the quantity of cabbage to "1 / 2 piece").

[1707] Output: Updated inventory information

[1708] Step 8:

[1709] A user accesses the emotion recognition engine and inputs an emotional state.

[1710] Input: Emotional state of the day

[1711] Specific operation: The user opens the smartphone app, accesses the emotion engine, and inputs their emotion for the day (e.g., "tired").

[1712] Output: Input emotion data

[1713] Step 9:

[1714] The device transmits the emotion data to the cloud server.

[1715] Input: Input emotion data

[1716] Specific operation: When the user taps the "Send" button, the emotion data is sent to the cloud server.

[1717] Output: Emotion data sent to the cloud server

[1718] Step 10:

[1719] The server analyzes emotions using an emotion engine and generates suggestions.

[1720] Input: Emotion data sent to the cloud server

[1721] What it does: The server uses the Microsoft Azure Emotion API to analyze emotions and generate dish and ingredient suggestions (e.g., "Stir-fried cabbage and chicken") based on those emotions.

[1722] Output: Emotion-based food and ingredient suggestions

[1723] Step 11:

[1724] The terminal notifies the user of the generated proposal.

[1725] Input: Emotion-based dish and ingredient suggestions

[1726] Specific behavior: The device notifies the user and suggests "Stir-fried cabbage and chicken."

[1727] Output: User notification

[1728] Step 12:

[1729] The user launches the app and checks inventory information.

[1730] Input: Current inventory information

[1731] Specific operation: When a user launches the smartphone app and taps the "Check stock" button, the app displays current stock information.

[1732] Output: User-confirmed inventory information

[1733] Step 13:

[1734] The terminal sends a request for inventory information to the cloud server.

[1735] Input: User-confirmed inventory information

[1736] Specific operation: The terminal automatically requests current inventory information from the server.

[1737] Output: Request sent to cloud server

[1738] Step 14:

[1739] The server generates a list of dishes based on the inventory information and sends it to the terminal.

[1740] Input: Requested inventory information

[1741] Specific operation: The server generates a list of dishes that can be suggested based on inventory information (e.g., 1 / 2 cabbage, 3 carrots, 500g of chicken) and sends it to the terminal.

[1742] Output: The recipe list sent to the user

[1743] Step 15:

[1744] The user selects the dish they want to make from the suggested dishes.

[1745] Input: A list of suggested dishes

[1746] Specific action: The user selects "Stir-fried cabbage and chicken" from a list of suggested dishes on a smartphone app.

[1747] Output: Selected dish items

[1748] Step 16:

[1749] The server compares the ingredients required for the selected dish with stock information and generates a list of ingredients that are in short supply.

[1750] Input: Selected dish item

[1751] Specific operation: The server compares the ingredients required for the selected dish (e.g., "Stir-fried cabbage and chicken") (1 cabbage, 300g chicken) with the current inventory information and adds the missing ingredients (1 / 2 cabbage) to the list.

[1752] Output: List of missing ingredients

[1753] Step 17:

[1754] The terminal displays a list of ingredients that are in short supply to the user.

[1755] Input: List of ingredients in short supply

[1756] Specific operation: The device displays a list of ingredients that are in short supply (e.g., 1 / 2 a cabbage) to the user, and the user checks and edits the shopping list.

[1757] Output: A list of missing ingredients displayed to the user

[1758] (Application example 2)

[1759] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1760] In today's world, users are expected to select foods and products that reflect their individual emotions and conditions. However, conventional systems were unable to provide emotion-based suggestions or inventory management, and the optimization of shopping lists and recipe suggestions did not reflect the user's emotions. This made shopping and recipe selection inconvenient, making it difficult to improve user satisfaction.

[1761] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1762] In this invention, the server includes: [a photographing means for the user to photograph ingredients;] [a transmitting means for transmitting the photographed image data to the cloud server;] [an analysis means for analyzing the image data in the cloud server and identifying the type and amount of ingredients;] [a storing means for saving the ingredient information identified as a result of the analysis;] [an update means for inputting the amount of ingredients used by the user based on the saved ingredient information and updating inventory information;] [an emotion analysis means for inputting user emotion data and analyzing it with an emotion engine;] [a suggestion means for suggesting dishes and products suitable for the user based on the emotion analysis results;] [a generation means for generating a shopping list based on current inventory information and suggested dishes; and] [a display means for displaying the generated shopping list to the user. This makes it possible to generate an optimal shopping list based on the user's emotions and inventory information, and to suggest dishes and products according to emotions.

[1763] The "photography means for the user to photograph ingredients" refers to a camera or other photographic device that records images of ingredients purchased by the user.

[1764] The "transmission means for transmitting the captured image data to the cloud server" refers to a communication device and software for uploading the captured image data to a server on the cloud via the Internet.

[1765] The "analysis means for analyzing image data on a cloud server and identifying the type and quantity of ingredients" refers to a program and hardware for recognizing the type and quantity of ingredients using an image analysis algorithm on a cloud server.

[1766] The "storage means for storing information on ingredients identified as analysis results" refers to a device and software for storing information on analyzed ingredients in a storage medium such as a database.

[1767] The "update means for inputting the amount of ingredients used by the user based on the stored ingredient information and updating inventory information" refers to an interface for inputting the amount of ingredients used by the user, and a program and hardware for updating inventory information based on that data.

[1768] "Emotion analysis means for inputting user emotion data and analyzing it with an emotion engine" refers to an emotion analysis engine and related software and hardware for inputting and interpreting a user's emotional state.

[1769] The "means for suggesting dishes and products suitable for the user based on the results of emotion analysis" refers to an algorithm and related hardware for selecting dishes and products that are likely to be preferred by the user based on the results of emotion analysis.

[1770] "Means for generating a shopping list based on current inventory information and suggested dishes" refers to a program and hardware for identifying ingredients that are in short supply based on current inventory information and automatically creating a shopping list.

[1771] The "display means for displaying the generated shopping list to the user" refers to a device and software for visually displaying the generated shopping list on a display device such as the user's smartphone or personal computer.

[1772] This invention is a system that combines an ingredient management system with an emotion engine that recognizes the user's emotions, making recipe suggestions and optimizing the shopping list according to the user's emotions. In addition to purchasing and inventory management and recipe suggestions, this system is characterized by recognizing the user's emotions with the emotion engine and making suggestions based on the user's emotions.

[1773] Basic system configuration

[1774] Photographing and storing ingredients

[1775] The user takes a photo of the ingredients they have purchased using the camera on their smartphone. The captured image data is sent to a cloud server via the smartphone's transmission means. The cloud server then uses an image analysis algorithm (e.g., Google Cloud Vision, Amazon Rekognition) to identify the type and volume of the ingredients. The ingredient information identified as a result of the analysis is then stored in a database.

[1776] Inventory management

[1777] Users input the amounts of ingredients used for dinner into a smartphone app, and the data is sent to a cloud server, which updates the inventory information in the database.

[1778] Emotion Recognition and Suggestions

[1779] The user inputs their emotional state for the day into the emotion engine via a smartphone app. The emotion data is sent to a cloud server and analyzed by an emotion analysis engine (e.g., Microsoft Azure Cognitive Services, IBM Watson). Based on the results of the emotion analysis, the system makes recommendations for dishes and products that are suitable for the user.

[1780] Generate a shopping list

[1781] The user launches the smartphone app and generates a shopping list based on current inventory information. The server then generates a list of ingredients that are in short supply and displays that information to the user.

[1782] Specific examples

[1783] The user follows the steps described above and after shopping, purchases 1 cabbage, 3 carrots, and 500g of chicken, and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database (1 cabbage, 3 carrots, 500g of chicken). The user eats 1 / 2 a cabbage for dinner and enters that information into the app. The device sends the input information to the server, and the server updates the inventory to "1 / 2 a cabbage."

[1784] Prompt Sentence Examples

[1785] Prompt input for generative AI model:

[1786] 1. "Imagine a flow where a user inputs emotional data, sends that data to a server, and suggests dishes based on the user's emotions."

[1787] 2. "When a user takes a picture of an ingredient, generate a program that analyzes the image and stores it in a database."

[1788] In this way, by combining emotion engines, it becomes possible to make optimal suggestions based on the user's emotions, enabling more personalized ingredient management and cooking suggestions.

[1789] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1790] Step 1:

[1791] The user takes a photo of the ingredients they have purchased using their smartphone camera.

[1792] Input: Image of purchased ingredients

[1793] Output: Captured image data

[1794] How it works: The user takes a photo of an ingredient with their smartphone camera and obtains image data.

[1795] Step 2:

[1796] The device sends the captured image data to a cloud server.

[1797] Input: Captured image data

[1798] Output: Image data transferred to the cloud server

[1799] Operation: Using the device's transmission means, the captured image data is uploaded to a cloud server via the Internet.

[1800] Step 3:

[1801] The server analyzes the image data sent to the cloud and identifies the type and volume of ingredients.

[1802] Input: Image data sent to the cloud server

[1803] Output: Ingredient type and quantity information

[1804] How it works: Using image analysis algorithms on a cloud server, the type and volume of ingredients are recognized from image data (e.g., Google Cloud Vision, Amazon Rekognition).

[1805] Step 4:

[1806] The server stores the ingredient information identified as a result of the analysis in a database.

[1807] Input: Ingredient type and quantity information

[1808] Output: Ingredient information stored in a database

[1809] Operation: The server records the identified ingredient information using a storage means that stores it in a database.

[1810] Step 5:

[1811] The user enters the amount of ingredients used into a smartphone app.

[1812] Input: Amount of ingredients used

[1813] Output: Usage data sent from the smartphone

[1814] How it works: The user uses the app's interface to input the amounts of ingredients used.

[1815] Step 6:

[1816] The terminal transmits the input usage information to the cloud server.

[1817] Input: Usage data

[1818] Output: Usage data transferred to the cloud server

[1819] Operation: The terminal uses the transmission means to upload the input data to the cloud server.

[1820] Step 7:

[1821] The server updates the inventory information in the database.

[1822] Input: Usage data

[1823] Output: Updated inventory information

[1824] How it works: The server updates the inventory information in the database based on the usage data.

[1825] Step 8:

[1826] The user enters their emotional state for the day into a smartphone app.

[1827] Input: Emotional state data

[1828] Output: Emotional state data sent from the smartphone

[1829] How it works: The user uses the app's interface to input their emotional state for the day.

[1830] Step 9:

[1831] The device transmits the emotion data to the cloud server.

[1832] Input: Emotional state data

[1833] Output: Emotional state data transferred to the cloud server

[1834] Operation: The device uses the transmission means to upload emotion data to the cloud server.

[1835] Step 10:

[1836] The server analyzes the user's emotions using an emotion engine.

[1837] Input: Emotional state data

[1838] Output: Emotion analysis results

[1839] How it works: The server uses an emotion analysis engine (e.g., Microsoft Azure Cognitive Services, IBM Watson) to analyze the emotion data.

[1840] Step 11:

[1841] The server will suggest dishes and products suitable for the user based on the results of the emotion analysis.

[1842] Input: Sentiment analysis results, inventory information

[1843] Output: A list of suggested dishes and products

[1844] How it works: The server creates a list of suitable dishes and products based on the results of sentiment analysis and inventory information.

[1845] Step 12:

[1846] The server generates a shopping list based on current inventory information and suggested dishes.

[1847] Input: Proposal list, inventory information

[1848] Output: Shopping list

[1849] How it works: The server compares the suggested list with inventory information and generates a list of ingredients or products that are in short supply.

[1850] Step 13:

[1851] The terminal displays the generated shopping list to the user.

[1852] Input: Shopping list

[1853] Output: A shopping list visually displayed to the user

[1854] Operation: The terminal uses a display means to visually display the shopping list to the user.

[1855] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1856] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1857] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1858] [Fourth embodiment]

[1859] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1860] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1861] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1862] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1863] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1864] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1865] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1866] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1867] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1868] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1869] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1870] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1871] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1872] overview

[1873] This invention is a system that streamlines food inventory management within the home and prevents forgetting to use ingredients or purchasing duplicates. Users take photos of purchased ingredients with their smartphones and send the images to a cloud server, where AI identifies and records the type and volume of ingredients. The data is stored in the cloud and used for inventory management, creating shopping lists, and suggesting recipes.

[1874] Program processing overview

[1875] Food Recognition and Preservation

[1876] User: Takes a photo of the ingredients he / she has purchased using his / her smartphone camera.

[1877] Terminal: Sends captured image data to a cloud server.

[1878] Server: Analyzes the received image data using AI and recognizes the type and volume of ingredients.

[1879] Server: Saves the analysis results in a database.

[1880] Example: A user purchases one cabbage, three carrots, and 500g of chicken, and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database.

[1881] Inventory management

[1882] User: Enters the amount of ingredients used for dinner into the app.

[1883] Terminal: Sends the entered usage information to the cloud server.

[1884] Server: Updates the inventory information in the database.

[1885] Example: A user makes a cabbage salad and uses 1 / 2 a cabbage. The user enters "1 / 2 a cabbage used" into the app, and the server updates the cabbage inventory in the database to "1 / 2 a cabbage."

[1886] Shopping suggestions

[1887] User: Launches the app and creates a shopping list.

[1888] Terminal: Retrieves current inventory information from the cloud.

[1889] Server: Generates a list of dishes that can be suggested based on inventory information and sends it to the device.

[1890] Terminal: Shows the user a list of suggested dishes.

[1891] User: Select the dish they want to make from the suggested dishes.

[1892] Server: Compares the ingredients required for the selected dish with the current inventory information and generates a list of ingredients that are missing.

[1893] Terminal: Displays a list of missing ingredients and allows the user to review and edit the shopping list.

[1894] Example: A user launches an app, and the app retrieves inventory information for "1 / 2 cabbage," "3 carrots," and "500g chicken" from the server. Based on this inventory, the app suggests "stir-fried cabbage and chicken," which the user selects. The server compares the required ingredients (1 cabbage, 300g chicken) with the inventory, and adds the missing 1 / 2 cabbage to the list. The app displays this list to the user, who can then confirm and edit the shopping list.

[1895] Implementation flow

[1896] To implement this system, users use a smartphone app to take a photo of the ingredients they have purchased and send the image to a cloud server via the app. The cloud server then analyzes the image, identifies the type and quantity of ingredients, and stores the information in a database. The app updates inventory information in real time and also reflects the amount of ingredients entered by the user. When shopping, the app also displays suggested dishes based on current inventory information and automatically generates a list of ingredients that are in short supply.

[1897] This allows users to efficiently manage ingredients and receive planned cooking suggestions, thereby reducing food waste and enabling efficient household budget management.

[1898] The processing flow will be explained below.

[1899] Food Recognition and Preservation

[1900] Step 1:

[1901] The user takes a photo of the purchased ingredients using the smartphone camera.

[1902] Step 2:

[1903] The device saves the captured image data and calls an API to send it to the cloud server.

[1904] Step 3:

[1905] The server temporarily stores the received image data and passes the data to an AI model for image analysis.

[1906] Step 4:

[1907] The server uses an AI model to analyze the image data and identify the type and quantity of ingredients, for example, "1 cabbage," "3 carrots," and "500g of chicken."

[1908] Step 5:

[1909] The server saves the analysis results in a database and records them as "1 cabbage," "3 carrots," and "500g of chicken."

[1910] Inventory management

[1911] Step 6:

[1912] The user enters the amount of ingredients used for dinner into the app.

[1913] Step 7:

[1914] The device calls an API to send the entered usage information to the cloud server.

[1915] Step 8:

[1916] The server temporarily stores the received usage data and updates the inventory information in the database.

[1917] Step 9:

[1918] The server reduces the inventory of specific ingredients in the database and updates it to "1 / 2 cabbage," "3 carrots," and "500g chicken."

[1919] Shopping suggestions

[1920] Step 10:

[1921] A user launches the app and checks current inventory information to create a shopping list.

[1922] Step 11:

[1923] The device requests current inventory information from the cloud server.

[1924] Step 12:

[1925] The server retrieves current inventory information from the database and sends it to the terminal.

[1926] Step 13:

[1927] Based on the inventory information received by the terminal, a list of dishes that can be suggested to the user is generated and displayed.

[1928] Step 14:

[1929] The user selects the dish they want to make from the suggested dishes.

[1930] Step 15:

[1931] The server compares all ingredients required for the selected dish with current inventory information and generates a list of ingredients that are in short supply.

[1932] Step 16:

[1933] The server transmits a list of ingredients that are in short supply to the terminal, which then displays it to the user.

[1934] Step 17:

[1935] The user checks the displayed list of ingredients that are in short supply and creates and edits a shopping list.

[1936] Specific examples of implementation

[1937] Step 18:

[1938] After shopping, the user purchases one cabbage, three carrots, and 500g of chicken and takes a photo of it with their smartphone.

[1939] Step 19:

[1940] The device sends the image to the server, which analyzes it and records it in a database ("1 cabbage," "3 carrots," "500g chicken").

[1941] Step 20:

[1942] A user eats 1 / 2 a cabbage for dinner and enters that information into the app.

[1943] Step 21:

[1944] The terminal sends the input information to the server, and the server updates the inventory to "1 / 2 cabbage."

[1945] Step 22:

[1946] Before the next shopping trip, the user launches the app, which retrieves the current inventory information. ("1 / 2 cabbage")

[1947] Step 23:

[1948] The server sends inventory information for "1 / 2 cabbage," "3 carrots," and "500g of chicken" to the terminal, and the terminal then suggests dishes based on that information.

[1949] Step 24:

[1950] The user selects "Stir-fried cabbage and chicken" and adds "1 / 2 cabbage" to the list, for which the server is short.

[1951] Step 25:

[1952] The device displays a list of ingredients that are in short supply, and the user can check and edit the shopping list.

[1953] These are the specific processing steps, from recognizing and storing ingredients to inventory management and suggestions when shopping.

[1954] Example 1

[1955] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1956] In most cases, food inventory management in the home is done manually, resulting in frequent forgetting to use ingredients or duplicate purchases. Furthermore, determining what to buy next requires knowing the current inventory status and recipes for the necessary dishes, which is time-consuming. Furthermore, there are often times when people are out of ideas for cooking, so an efficient way to solve these problems is needed.

[1957] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1958] In this invention, the server includes: [a photographing means for the user to photograph ingredients;] [a transmitting means for transmitting the photographed image data to a network server;] [an analyzing means for analyzing the image data in the network server and identifying the type and amount of ingredients;] [a storing means for saving the ingredient information identified as a result of the analysis;] [an updating means for inputting the amount of ingredients used by the user based on the stored ingredient information and updating inventory information;] [a generating means for generating a shopping list based on current inventory information;] [a display means for displaying the generated shopping list to the user;] [an automatic output means for generating a list of suggested dishes; and] [a list generating means for generating a list of missing ingredients based on the suggested dishes. This enables ingredient inventory management, prevention of forgotten ingredients and duplicate purchases, efficient shopping list creation, and dish suggestions.

[1959] The term "photography means for a user to photograph ingredients" refers to a device or application that a user uses to record the ingredients they have purchased as digital images.

[1960] The "transmission means for transmitting the photographed image data to the network server" refers to a device or application for transmitting the photographed image data of ingredients to a server on the cloud via the Internet.

[1961] "Analysis means for analyzing image data on a network server and identifying the type and volume of ingredients" refers to software or algorithms for analyzing received image data and automatically identifying the type and volume of ingredients.

[1962] "Storage means for storing information on ingredients identified as a result of the analysis" refers to a device or application for recording data on the type and volume of ingredients obtained through the analysis in storage such as a database.

[1963] "Means for updating inventory information by inputting the amount of ingredients used by the user based on the stored ingredient information" refers to a device or application for correcting and updating inventory data in real time based on the amount of ingredients used input by the user through the application.

[1964] "Means for generating a shopping list based on current inventory information" refers to a device or application that references recorded inventory information and automatically creates a list of new ingredients to be purchased.

[1965] The "display means for displaying the generated shopping list to the user" refers to a device or application for visually displaying the automatically generated shopping list on the display of the user's device, for example.

[1966] "Automatic output means for generating a list of suggested dishes" refers to a device or application that automatically generates and presents a list of dishes that can be made based on stored inventory information and data such as user preferences.

[1967] "List generation means for generating a list of missing ingredients based on a proposed dish" refers to a device or application that compares ingredients required for the proposed dish with inventory data and lists missing ingredients.

[1968] MODE FOR CARRYING OUT THE INVENTION

[1969] This invention is a system designed to improve the efficiency of food ingredient inventory management in the home, prevent forgetting to use ingredients or purchasing the same ingredients twice, and provide recipe suggestions based on a well-planned recipe. Specific embodiments of this system are described below.

[1970] A user uses a smartphone to take a photo of the ingredients they have purchased. The smartphone's camera (photography means) is used to capture high-resolution images of the ingredients. For example, imagine a case where a user takes a photo of one cabbage, three carrots, and 500g of chicken upon returning home from the supermarket. The captured image data is sent to a network server on the cloud via an application on the smartphone (transmission means).

[1971] The network server analyzes the received image data. Specifically, it uses image recognition services such as Google Cloud Vision API and Amazon Rekognition to identify the type and volume of ingredients in the image (analysis means). The data resulting from the analysis is stored in a cloud database (e.g., Google Firebase or Amazon DynamoDB) (storage means).

[1972] When a user uses an ingredient, they input the amount used into the smartphone application. For example, if they input "1 / 2 cabbage used," the application sends this information to the cloud server, and the inventory information in the cloud database is updated (update means).

[1973] Furthermore, when a user launches the app to go shopping, the application retrieves current inventory information from the cloud. A shopping list is automatically generated based on the current inventory information (generation means). The generated shopping list is displayed to the user, allowing the user to use the list as a reference when shopping (display means).

[1974] The system also has the ability to suggest dishes using a generative AI model. For example, it uses a language model such as OpenAI's GPT-3 to generate a list of suggested dishes based on inventory information and displays it to the user (automatic output means). When the user selects one of the suggested dishes, the system compares the ingredients needed for that dish with current inventory information and automatically generates a list of ingredients that are missing (list generation means).

[1975] The above process allows users to efficiently manage ingredients and plan their shopping and cooking, thereby reducing food waste and enabling efficient management of household finances.

[1976] For example, suppose the user enters the prompt text as follows:

[1977] "I have half a cabbage, three carrots, and 500g of chicken left in my fridge. What kind of dish can I make with these ingredients? Can you also tell me how to make it?"

[1978] Based on these prompts, the generative AI model will suggest optimal dishes and cooking methods to the user, allowing them to expand the range of cooking they can do at home.

[1979] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1980] Program processing flow

[1981] Step 1:

[1982] The user takes a photo of the ingredients

[1983] Specific operation: The user uses the smartphone camera to take a picture of the ingredients they purchased. For example, they take a picture of one cabbage, three carrots, and 500g of chicken lined up.

[1984] Input: An image of the food captured by the camera.

[1985] Output: Image data of ingredients stored on the smartphone.

[1986] Step 2:

[1987] The device sends the image to the cloud server.

[1988] Specific operation: A smartphone app sends captured image data to a cloud server via the Internet. For example, the app uploads the images to AWS S3.

[1989] Input: Image data of ingredients stored on your smartphone.

[1990] Output: Image data sent to the cloud server.

[1991] Step 3:

[1992] The server analyzes the image and recognizes the type and volume of ingredients.

[1993] Specific operation: The cloud server analyzes the received image data using Google Cloud Vision API and Amazon Rekognition to identify the type and quantity of ingredients. For example, it might recognize it as one cabbage, three carrots, and 500g of chicken.

[1994] Input: Image data stored on a cloud server.

[1995] Output: Analysis results including ingredient type and volume.

[1996] Step 4:

[1997] The server stores the analysis results in a database

[1998] Specific operation: The food ingredient information from the analysis results is saved in a cloud database (e.g., Google Firebase or Amazon DynamoDB).

[1999] Input: Analysis results including ingredient type and volume.

[2000] Output: Ingredient information recorded in a cloud database.

[2001] Step 5:

[2002] The user enters the amount of ingredients used into the app.

[2003] Specific operation: The user inputs the amount of ingredients used for dinner into the smartphone app. For example, the user inputs that half a cabbage was used.

[2004] Input: Amount of ingredients used.

[2005] Output: Usage information entered into the smartphone app.

[2006] Step 6:

[2007] The device sends usage information to the cloud server.

[2008] Specific operation: The smartphone app sends the entered usage information to a cloud server. For example, the app sends usage data to Firebase Firestore.

[2009] Input: Usage information stored in the smartphone app.

[2010] Output: Usage information sent to the cloud server.

[2011] Step 7:

[2012] The server updates the inventory information in the database

[2013] Specific operation: The cloud server updates the inventory information in the cloud database in real time based on the received usage information. For example, the inventory of cabbage is updated to 1 / 2.

[2014] Input: Usage information sent to the cloud server.

[2015] Output: Updated inventory information from the cloud database.

[2016] Step 8:

[2017] The user launches the app and creates a shopping list

[2018] Specific action: The user launches the smartphone app and accesses the screen for creating a shopping list.

[2019] Input: The app launch operation.

[2020] Output: Display of the app's shopping list creation screen.

[2021] Step 9:

[2022] The device retrieves current inventory information from the cloud.

[2023] Specific operation: The smartphone app accesses a cloud database to get the latest inventory information, for example, from Firebase Firestore.

[2024] Input: A retrieval request to the cloud database.

[2025] Output: The latest inventory information obtained.

[2026] Step 10:

[2027] The server generates recipe suggestions based on inventory information and sends them to the device.

[2028] Specific operation: The cloud server uses the acquired inventory information to generate a list of dishes that are best suited to the user using a language model such as GPT-3, and sends this list to the smartphone app.

[2029] Input: Latest inventory information.

[2030] Output: A list of suggested dishes.

[2031] Step 11:

[2032] The device displays suggested dishes

[2033] Specific operation: The smartphone app displays the list of suggested dishes received from the server to the user. For example, it displays a list including "stir-fried cabbage and chicken."

[2034] Input: A list of dish suggestions received from the server.

[2035] Output: A list of cooking suggestions displayed on a smartphone app.

[2036] Step 12:

[2037] The user selects the dish they want to make.

[2038] Specific operation: The user selects the dish they want to make from the list of suggested dishes. For example, they select "Stir-fried cabbage and chicken."

[2039] Input: User's food selection.

[2040] Output: Selected dish information.

[2041] Step 13:

[2042] The server compares the required ingredients with the available stock and generates a list of ingredients that are in short supply.

[2043] Specific operation: The cloud server compares the list of ingredients needed for the selected dish with the current inventory information and lists any ingredients that are missing. For example, it recognizes that 1 / 2 a cabbage is missing.

[2044] Input: Selected dish information and current inventory information.

[2045] Output: A list of missing ingredients.

[2046] Step 14:

[2047] The device displays a list of ingredients that are missing, and the user can check and edit the shopping list.

[2048] Specific operation: The smartphone app displays the list of ingredients that are in short supply received from the server to the user, who then checks and edits the list.

[2049] Input: Missing ingredient list.

[2050] Output: A list of ingredients that are in short supply, displayed on the smartphone app. Users can confirm and edit the list.

[2051] (Application example 1)

[2052] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2053] Inventory management of ingredients and products at home or in a brick-and-mortar store is complicated, and manual management can easily lead to forgetting to use ingredients or purchasing duplicates. Furthermore, if product inventory is not managed properly, there is a high risk of unsold items or shortages. This increases food waste and opportunity losses, hindering efficient operations.

[2054] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2055] In this invention, the server includes [a photographing means for the user to photograph ingredients], [a transmission means for transmitting the photographed image data to the cloud server], [an analysis means for analyzing the image data in the cloud server and identifying the type and amount of ingredients], [a storage means for saving the ingredient information identified as a result of the analysis], [an update means for inputting the amount of ingredients used by the user based on the saved ingredient information and updating inventory information], [a generation means for generating a shopping list based on current inventory information], [a display means for displaying the generated shopping list to the user], [a suggestion means for suggesting products to the user in conjunction with inventory], and [a detection means for detecting missing ingredients or products]. This makes it possible to automatically manage ingredient and product inventory and make appropriate suggestions to the user.

[2056] "Photographing means" refers to a device or method that allows a user to photograph ingredients or products.

[2057] The "transmission means" is a device or method for transmitting captured image data to a cloud server.

[2058] The "analysis means" is a device or method for analyzing image data on a cloud server and identifying the type and volume of ingredients or products.

[2059] The "storage means" refers to a device or method for storing the ingredients and product information identified as analysis results in a database.

[2060] The "update means" is a device or method for inputting the amount of ingredients or products used by the user based on the stored ingredient and product information and updating the inventory information.

[2061] The "generation means" is a device or method for generating a shopping list or a recommended product list based on current inventory information.

[2062] The "display means" refers to a device or method for displaying the generated shopping list or recommended product list to the user.

[2063] The "suggestion means" is a device or method for suggesting products to users in conjunction with inventory.

[2064] "Detection means" refers to a device or method for detecting missing ingredients or products.

[2065] Overall system configuration

[2066] The system of this invention is broadly composed of a photographing means, a transmitting means, a analyzing means, a saving means, a updating means, a generating means, a displaying means, a suggesting means, and a detecting means. These means work together to realize inventory management of ingredients and products and shopping support based on that inventory management.

[2067] Program Overview

[2068] Filming method

[2069] The user uses a smartphone to take a photo of the ingredients or products they have purchased, and the device captures the image using the smartphone's camera.

[2070] Transmission method

[2071] The captured image data is sent to a cloud server. This device or software uses the smartphone's communication functions (Wi-Fi, mobile data communication, etc.).

[2072] Analysis means

[2073] The cloud server analyzes the captured image data using an AI model (for example, a deep learning model such as TensorFlow) to identify the type and volume of specific ingredients or products. This analysis uses image recognition technology.

[2074] Preservation means

[2075] The analyzed ingredient and product information is stored in a cloud database (such as AWS RDS or Google Cloud Firestore). The type, size, and photo date and time of each ingredient or product are recorded in the database.

[2076] Update method

[2077] Users input the quantities of ingredients and products used into the application, and the information is sent to a cloud server, which updates the inventory information in a database based on the input.

[2078] generation means

[2079] Based on the current inventory information, the cloud server generates a shopping list and a list of recommended products, allowing users to efficiently create lists of ingredients and products they need.

[2080] Display means

[2081] The generated shopping list and recommended product list are displayed to the user through a smartphone application. The user interface can be built using a cross-platform framework such as React Native.

[2082] Proposal means

[2083] The server connects with current inventory information and suggests suitable products and recipes to users based on the latest inventory information and the user's past purchase history.

[2084] Detection Method

[2085] Every time inventory data is updated, the cloud server compares it with the existing inventory data to detect any missing ingredients or products, which are then automatically added to the user's shopping list.

[2086] Specific examples

[2087] For example, when a store employee places new tomatoes on a shelf, they take a picture of the tomato using the app, which then uploads the image to the cloud. An AI model on the cloud server analyzes the image, identifies the type and quantity of tomatoes, and stores the information in a database. When a customer opens the app, fresh tomatoes are recommended based on the latest inventory information.

[2088] Prompt Sentence Examples

[2089] "Please upload images of newly arrived tomatoes to the cloud server. The AI ​​on the server side will identify the quantity and quality of the tomatoes, update the inventory information, and reflect it as recommendations on the customer app."

[2090] As a result, the system of the present invention can improve the efficiency of inventory management of food ingredients and products at home or in a physical store, and can support customers in making appropriate product suggestions and generating shopping lists.

[2091] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2092] Step 1:

[2093] The user takes a photo of the food or product using a smartphone.

[2094] Input: Image of ingredients or products

[2095] Output: Image data stored on a smartphone

[2096] Specific actions: The user launches the smartphone camera app, frames the food or product they are purchasing, and presses the shutter button. When doing so, they take as clear an image as possible, taking care to ensure that the food or product is clearly identifiable.

[2097] Step 2:

[2098] The device sends the captured image data to a cloud server.

[2099] Input: Captured image data

[2100] Output: Image data transferred to the cloud server

[2101] What happens: The smartphone app compresses the captured image file and sends an HTTP POST request to a cloud server over the internet, including user authentication information.

[2102] Step 3:

[2103] The server receives the image data at the cloud server and analyzes it using the analysis means.

[2104] Input: Image data transferred to the cloud server

[2105] Output: Analyzed ingredients and product types and volume data

[2106] How it works: The server stores the received image data on disk and performs image recognition using an AI model (e.g., TensorFlow). This AI model identifies the type and volume of ingredients or products from the image and extracts this information as metadata.

[2107] Step 4:

[2108] The server stores the analysis results in a database.

[2109] Input: Analyzed ingredient and product type and volume data

[2110] Output: Analysis result data stored in a database

[2111] Specific operation: The server converts the analysis results into structured data and executes an INSERT query to a cloud database (e.g., AWS RDS, Google Cloud Firestore). The stored data includes the type of ingredients or products, volume, analysis date and time, etc.

[2112] Step 5:

[2113] The user enters the amounts of ingredients and products used into the application.

[2114] Input: Usage information

[2115] Output: Usage data transferred to the cloud server

[2116] Specific operation: The user opens a form on the smartphone application to input the amount of ingredients and products used, enters the necessary information, for example, "1 / 2 cabbage used," and presses the send button. This information is sent to the cloud server.

[2117] Step 6:

[2118] The server updates the inventory information in the database based on the input usage data.

[2119] Input: Usage data transferred to the cloud server

[2120] Output: Updated inventory information

[2121] Specific operation: The server analyzes the received usage data and reduces the corresponding inventory data. It updates the inventory information in the database using an UPDATE query. The updated inventory information reflects the remaining amount of the ingredient or product.

[2122] Step 7:

[2123] The server generates shopping lists and recommended product lists based on current inventory information.

[2124] Input: Updated inventory information

[2125] Output: Generated shopping list and recommended products list

[2126] Specific operation: The server identifies ingredients and products that are in short supply based on the latest inventory data, creates a list of items to purchase, and adds recommended products and recipes to the list, taking into account the user's past purchase history and trend data.

[2127] Step 8:

[2128] The terminal displays the generated shopping list and recommended product list to the user.

[2129] Input: Generated shopping list or recommended product list

[2130] Output: List displayed on smartphone

[2131] What it does: The smartphone application analyzes the list data received from the server and displays it in an easy-to-read format on the user interface, allowing the user to plan their shopping needs.

[2132] Step 9:

[2133] The server works in conjunction with inventory information to suggest products to users.

[2134] Input: Current inventory information, user's past purchase history

[2135] Output: Suggested products and recipe information

[2136] What it does: The server cross-references the inventory database with the user's purchasing history to generate product and recipe recommendations that are best suited to the user. These recommendations are listed in order of most relevant and notified to the user.

[2137] Step 10:

[2138] The server detects ingredients or products that are in short supply and adds them to the shopping list.

[2139] Input: Current inventory information

[2140] Output: Updated shopping list

[2141] What it does: The server periodically scans inventory data to identify ingredients or products that are running low, automatically adding them to the shopping list and notifying the user.

[2142] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2143] overview

[2144] This invention is a system that combines an ingredient management system with an emotion engine that recognizes the user's emotions, making recipe suggestions and optimizing the shopping list according to the user's emotions. In addition to purchasing and inventory management and recipe suggestions, this system is characterized by recognizing the user's emotions with the emotion engine and making suggestions based on the user's emotions.

[2145] Program processing overview

[2146] Food Recognition and Preservation

[2147] User: Takes a photo of the purchased ingredients using the smartphone camera.

[2148] Terminal: Sends captured image data to a cloud server.

[2149] Server: Analyzes the received image data using AI and recognizes the type and volume of ingredients.

[2150] Server: Saves the analysis results in a database.

[2151] Example: A user purchases one cabbage, three carrots, and 500g of chicken, and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database.

[2152] Inventory management

[2153] User: Enters the amount of ingredients used for dinner into the app.

[2154] Terminal: Sends the entered usage information to the cloud server.

[2155] Server: Updates the inventory information in the database.

[2156] Example: A user makes a cabbage salad and uses 1 / 2 a cabbage. The user enters "1 / 2 a cabbage used" into the app, and the server updates the cabbage inventory in the database to "1 / 2 a cabbage."

[2157] Emotion Recognition and Suggestions

[2158] User: Accesses the emotion engine through the app and inputs their emotional state for the day.

[2159] Terminal: Sends the user's emotional data to the cloud server.

[2160] Server: The emotion engine analyzes the user's emotions and suggests dishes and ingredients based on those emotions.

[2161] For example, if the user is tired, easy meals will be suggested, and if the user is energetic, more challenging meals will be suggested.

[2162] Shopping suggestions

[2163] User: Launches the app and checks current inventory information to create a shopping list.

[2164] Terminal: Requests current inventory information from the cloud server.

[2165] Server: Generates a list of dishes that can be suggested based on inventory information and sends it to the device.

[2166] Terminal: Shows the user a list of suggested dishes.

[2167] User: Select the dish they want to make from the suggested dishes.

[2168] Server: Compares all ingredients required for the selected dish with current inventory and generates a list of ingredients that are missing.

[2169] Terminal: Displays a list of missing ingredients and allows the user to review and edit the shopping list.

[2170] Example: A user launches the app and retrieves current inventory information (1 / 2 cabbage, 3 carrots, 500g chicken). The emotion engine recognizes that the user is tired and suggests a simple "stir-fried cabbage and chicken." The server adds the missing 1 / 2 cabbage to the list, and the device displays the shopping list.

[2171] Specific examples of implementation

[2172] After shopping, a user purchases 1 cabbage, 3 carrots, and 500g of chicken and takes a photo of it with their smartphone. The device sends the image to the server, which analyzes it and records it in the database (1 cabbage, 3 carrots, 500g of chicken). The user eats 1 / 2 a cabbage for dinner and enters that information into the app. The device sends the input information to the server, which updates the inventory to "1 / 2 a cabbage."

[2173] Next, the user launches the app and checks the current inventory information (1 / 2 cabbage, 3 carrots, 500g of chicken) to create a shopping list. The emotion engine recognizes the user's emotion (e.g., tiredness) and suggests an easy-to-make dish, "Stir-fried cabbage and chicken." The server compares the required ingredients (1 cabbage, 300g of chicken) with the inventory and adds the missing 1 / 2 cabbage to the list. The device displays the list of missing ingredients, and the user can check and edit the shopping list.

[2174] In this way, by combining emotion engines, it becomes possible to make optimal suggestions based on the user's emotions, enabling more personalized ingredient management and cooking suggestions.

[2175] The processing flow will be explained below.

[2176] Processing flow of a system that combines emotion engines

[2177] Emotion Recognition and Suggestions

[2178] Step 1:

[2179] The user launches the app and enters their emotional state for the day.

[2180] Step 2:

[2181] The device transmits the user's emotional data to a cloud server.

[2182] Step 3:

[2183] The server passes the received emotion data to the emotion engine, which analyzes the user's emotional state.

[2184] Step 4:

[2185] The server generates a list of dishes according to the emotion based on the analysis results of the emotion engine.

[2186] Food Recognition and Preservation

[2187] Step 5:

[2188] The user takes a photo of the purchased ingredients using the smartphone camera.

[2189] Step 6:

[2190] The device saves the captured image data and calls an API to send it to the cloud server.

[2191] Step 7:

[2192] The server temporarily stores the received image data and passes the data to an AI model for image analysis.

[2193] Step 8:

[2194] The server uses an AI model to analyze the image data and identify the type and quantity of ingredients, for example, "1 cabbage," "3 carrots," and "500g of chicken."

[2195] Step 9:

[2196] The server saves the analysis results in a database and records them as "1 cabbage," "3 carrots," and "500g of chicken."

[2197] Inventory management

[2198] Step 10:

[2199] The user enters the amount of ingredients used for dinner into the app.

[2200] Step 11:

[2201] The device calls an API to send the entered usage information to the cloud server.

[2202] Step 12:

[2203] The server temporarily stores the received usage data and updates the inventory information in the database.

[2204] Step 13:

[2205] The server reduces the inventory of specific ingredients in the database and updates it to "1 / 2 cabbage," "3 carrots," and "500g chicken."

[2206] Shopping suggestions

[2207] Step 14:

[2208] A user launches the app and checks current inventory information to create a shopping list.

[2209] Step 15:

[2210] The device requests current inventory information from the cloud server.

[2211] Step 16:

[2212] The server retrieves current inventory information from the database and sends it to the terminal.

[2213] Step 17:

[2214] Based on the inventory information received by the terminal, a list of dishes that can be suggested to the user is generated and displayed.

[2215] Step 18:

[2216] The user selects the dish they want to make from the suggested dishes.

[2217] Step 19:

[2218] The server compares all ingredients required for the selected dish with current inventory information and generates a list of ingredients that are in short supply.

[2219] Step 20:

[2220] The server transmits a list of ingredients that are in short supply to the terminal, which then displays it to the user.

[2221] Step 21:

[2222] The user checks the displayed list of ingredients that are in short supply and creates and edits a shopping list.

[2223] Specific examples of implementation

[2224] Step 22:

[2225] After shopping, the user purchases one cabbage, three carrots, and 500g of chicken and takes a photo of it with their smartphone.

[2226] Step 23:

[2227] The device sends the image to the server, which analyzes it and records it in a database ("1 cabbage," "3 carrots," "500g chicken").

[2228] Step 24:

[2229] A user eats 1 / 2 a cabbage for dinner and enters that information into the app.

[2230] Step 25:

[2231] The terminal sends the input information to the server, and the server updates the inventory to "1 / 2 cabbage."

[2232] Step 26:

[2233] The user launches the app before their next shopping trip, and the app retrieves the current inventory information ("1 / 2 cabbage," "3 carrots," and "500g chicken").

[2234] Step 27:

[2235] The device inputs the user's emotional state into a cloud server, which then analyzes the data.

[2236] Step 28:

[2237] The emotion engine recognizes that the user is tired and suggests an easy-to-make dish: stir-fried cabbage and chicken.

[2238] Step 29:

[2239] The server compares the ingredients needed ("1 cabbage" and "300g chicken") with the inventory and adds the missing 1 / 2 cabbage to the list.

[2240] Step 30:

[2241] The device displays a list of ingredients that are in short supply, and the user can check and edit the shopping list.

[2242] The above is the specific processing flow of a system that combines an emotion engine. By taking the user's emotions into consideration, more personalized suggestions become possible, which is expected to improve user satisfaction.

[2243] Example 2

[2244] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2245] In today's busy lifestyles, it is difficult for users to efficiently manage ingredient inventory, select dishes, and receive recipe suggestions tailored to their mood of the day. Conventional systems require users to manually update inventory information and manage ingredient usage, which is tedious and often inaccurate. Furthermore, they lack the ability to suggest dishes based on the user's emotions, resulting in a non-personalized user experience. Therefore, there is a need for a system that allows users to efficiently manage ingredients, inventory, recognize emotions, and receive optimal recipe suggestions.

[2246] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2247] In this invention, the server includes: [a photographing means for the user to photograph ingredients;] [a transmission means for sending the photographed image data to the cloud server;] [an analysis means for analyzing the image data in the cloud server and identifying the type and amount of ingredients;] [a storage means for saving ingredient information identified as a result of the analysis;] [an update means for inputting the amount of ingredients used by the user based on the saved ingredient information and updating inventory information;] [an emotion recognition and suggestion means for recognizing the user's emotions and suggesting dishes and ingredients based on the emotions;] [a notification means for notifying the user of the suggestions generated based on the emotions;] [a generation means for generating a shopping list based on current inventory information;] [a display means for displaying the generated shopping list to the user; and] [an emotion input means for inputting the user's emotional state and sending it to the cloud server. This enables users to efficiently manage ingredients and enables personalized dish suggestions based on the user's emotions.

[2248] "Photographing means" refers to the equipment or method used by the user to photograph the ingredients they have purchased.

[2249] "Transmission means" refers to the equipment or method used to transmit captured image data to the cloud server.

[2250] The "analysis means" refers to the equipment or method used to analyze the image data sent to the cloud server and identify the type and volume of ingredients.

[2251] "Storage means" refers to the equipment or method used to store the ingredient information identified as the analysis result in a database or the like.

[2252] The "update means" refers to a device or method used to input the amount of ingredients used by the user based on the stored ingredient information and update the inventory information.

[2253] "Emotion recognition and suggestion means" refers to devices and methods used to recognize a user's emotions and suggest dishes and ingredients based on those emotions.

[2254] "Notification means" refers to the device or method used to notify the user of suggestions generated based on emotions.

[2255] A "generator" is a device or method used to generate a shopping list based on current inventory information.

[2256] "Display means" refers to the device or method used to display the generated shopping list to the user.

[2257] "Emotion input means" refers to the device or method used to input the user's emotional state and transmit it to the cloud server.

[2258] MODE FOR CARRYING OUT THE INVENTION

[2259] This invention is a system that combines an ingredient management system with an emotion engine that recognizes the user's emotions, making recipe suggestions and optimizing the shopping list according to the user's emotions. In addition to purchasing and inventory management and recipe suggestions, this system is characterized by recognizing the user's emotions with the emotion engine and making suggestions based on the user's emotions.

[2260] Food Recognition and Preservation

[2261] User: Take a photo of the ingredients they purchased using their smartphone camera. The user launches the smartphone app, opens the camera function, and takes a photo of the ingredients. For example, they take a photo of one cabbage, three carrots, and 500g of chicken.

[2262] Device: Sends the captured image data to the cloud server. When the user taps the "Send" button, the image data is uploaded to the cloud server via the Internet.

[2263] Server: The received image data is analyzed using AI to identify the type and quantity of ingredients. The server uses the Google Cloud Vision API to analyze the image and extract information such as "1 cabbage, 3 carrots, 500g of chicken."

[2264] Server: The analysis results are saved in a database. The server saves the acquired ingredient information in a MySQL database and reflects it in each user's inventory list.

[2265] Inventory management

[2266] User: Enter the amount of ingredients used for dinner into the app. The user opens the smartphone app and enters the amount of ingredients used (for example, "1 / 2 head of cabbage used").

[2267] Device: Sends the entered usage information to the cloud server. When the user taps the "Send" button, the usage information is sent to the cloud server.

[2268] Server: Updates the inventory information in the database. The server updates the database and changes the amount of cabbage in the inventory list to "1 / 2 piece."

[2269] Emotion Recognition and Suggestions

[2270] User: Accesses the emotion engine through the app and inputs the emotional state of the day. The user opens the smartphone app, accesses the emotion engine, and inputs the emotion of the day (e.g., "tired").

[2271] Device: Sends the user's emotional data to the cloud server. When the user taps the "Send" button, the emotional data is sent to the cloud server.

[2272] Server: The emotion engine analyzes the user's emotions and suggests dishes and ingredients based on those emotions. The server uses the Microsoft Azure Emotion API to analyze emotions and determines that "simple dishes should be suggested," suggesting, for example, "stir-fried cabbage and chicken."

[2273] Shopping suggestions

[2274] User: Launches the app and checks current inventory information to create a shopping list. The user opens the smartphone app and taps the "Check inventory" button.

[2275] Device: Requests current inventory information from the cloud server. The device automatically requests current inventory information from the server.

[2276] Server: Generates a list of dishes that can be suggested based on inventory information and sends it to the device.The server generates a list of dishes that can be suggested based on inventory information (e.g., 1 / 2 cabbage, 3 carrots, 500g of chicken) and sends it to the device.

[2277] Device: Shows the user a list of suggested dishes. The device shows the user a suggestion for "Stir-fried cabbage and chicken."

[2278] User: Selects the dish they want to make from the suggested dishes. The user selects "Stir-fried cabbage and chicken" from the list of suggested dishes.

[2279] Server: Compares all ingredients needed for the selected dish with the current inventory and generates a list of ingredients that are missing. The server compares the required ingredients (1 cabbage, 300g chicken) with the inventory and adds the missing ingredients (1 / 2 cabbage) to the list.

[2280] Device: Displays a list of ingredients that are missing, and the user can check and edit the shopping list. The device displays a list of ingredients that are missing (1 / 2 a cabbage) to the user, and the user can check and edit the shopping list.

[2281] For example, after a user purchases a cabbage, three carrots, and 500g of chicken, the user takes a photo of the item with their smartphone, and the device sends the image to a server, which analyzes it and records it in a database. Next, the user uses half a cabbage for dinner and enters that information into the app. The device sends the input information to the server, and the server updates the inventory to "half a cabbage."

[2282] Next, the user launches the app and checks their current inventory (1 / 2 cabbage, 3 carrots, 500g of chicken) to create a shopping list. The emotion engine recognizes the user's emotion (e.g., tiredness) and suggests an easy-to-make dish, "Stir-fried cabbage and chicken." The server compares the required ingredients (1 cabbage, 300g of chicken) with the inventory and adds the missing 1 / 2 cabbage to the list. The device displays the list of missing ingredients to the user, who can then check and edit the shopping list.

[2283] In this way, by combining emotion engines, it becomes possible to make optimal suggestions based on the user's emotions, enabling more personalized ingredient management and cooking suggestions.

[2284] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2285] Step 1:

[2286] The user takes a photo of the food.

[2287] Input: Purchased ingredients

[2288] Specific operation: The user launches the smartphone app and uses the camera function to take a picture of ingredients (e.g., 1 cabbage, 3 carrots, and 500g of chicken).

[2289] Output: Captured image data

[2290] Step 2:

[2291] The device sends the image data to the cloud server.

[2292] Input: Photographed image data

[2293] Specific operation: When the user taps the "Send" button, the image data is uploaded to a cloud server via the Internet.

[2294] Output: Image data sent to the cloud server

[2295] Step 3:

[2296] The server analyzes the image data.

[2297] Input: Image data sent to the cloud server

[2298] Specific operation: The server analyzes the image data using the Google Cloud Vision API and identifies the type and quantity of ingredients (e.g., "1 cabbage, 3 carrots, 500g chicken").

[2299] Output: Parsed ingredient information

[2300] Step 4:

[2301] The server stores the analysis results in a database.

[2302] Input: Parsed ingredient information

[2303] Specific operation: The server saves ingredient information in a MySQL database and updates each user's inventory list.

[2304] Output: Updated database

[2305] Step 5:

[2306] The user enters the amount of ingredients used into the app.

[2307] Input: Amount of ingredients used for dinner

[2308] Specific operation: The user opens the smartphone app and enters the ingredients used (e.g., "1 / 2 cabbage used").

[2309] Output: Input usage information

[2310] Step 6:

[2311] The device sends usage information to the cloud server.

[2312] Input: Entered usage information

[2313] Specific operation: When the user taps the "Send" button, the usage information is sent to the cloud server.

[2314] Output: Usage information sent to the cloud server

[2315] Step 7:

[2316] The server updates the inventory information in the database.

[2317] Input: Usage information sent to the cloud server

[2318] Specific operation: The server updates the database and corrects the quantity of the relevant ingredient in the inventory list (e...

Claims

1. a photographing means for a user to photograph ingredients; a transmitting means for transmitting the captured image data to a cloud server; an analysis means for analyzing the image data in the cloud server and identifying the type and volume of the ingredients; a storage means for storing the ingredient information identified as a result of the analysis; an updating means for inputting the amount of ingredients used by the user based on the stored ingredient information and updating the inventory information; a generating means for generating a shopping list based on current inventory information; The system includes a display means for displaying the generated shopping list to a user.

2. 10. The system of claim 1, further comprising suggestion means for generating a list of suggested dishes based on current inventory information.

3. 2. The system according to claim 1, further comprising a shortage ingredient generating means for comparing ingredients required for a dish selected by a user with current inventory information and generating a list of ingredients that are in short supply.

Images