system

Abstract

Provide a system. An information acquisition means disposed in a refrigerated storage device; a data processing means for identifying the state of ingredients based on the acquired information; an identification means for identifying a best-before date and a storage method based on the identified condition; an information presentation means for providing the identified information to a user; a recipe generator for generating suggested recipes using the identified ingredients; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's busy lifestyles, it is difficult to efficiently manage food stored in a home refrigerator and consume it at the appropriate time. Food waste, particularly due to overlooking expiration dates, is a major problem. It is also not easy to find optimal recipes that combine the ingredients available. There is a need for a system that can solve these issues, streamline food management in refrigerators, reduce food waste, and even save money. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides a system that includes information acquisition means arranged in a refrigerated storage device, data processing means that identifies the condition of ingredients based on the acquired information, identification means that identifies expiration dates and storage methods based on the identified conditions, information presentation means that provides the identified information to users, and recipe generation means that generates suggested recipes using the identified ingredients.This makes it possible to appropriately manage the expiration dates of ingredients in the refrigerated storage device, present optimal storage methods for the ingredients, and further provide optimal recipes based on the ingredients in stock.

[0006] A "refrigerated storage device" is an electrical device used to store food at low temperatures.

[0007] "Information acquisition means" refers to a means for acquiring information about food ingredients in a refrigerated storage device, and includes devices that use cameras or scanners as methods for doing so.

[0008] The "data processing means" is a means for identifying the state of ingredients based on the acquired information and performing the necessary analysis and processing.

[0009] The "identification means" is a means for checking and identifying the expiration date and storage method of ingredients based on the information identified by the data processing means.

[0010] The "information presentation means" is a means for presenting the information obtained by the identification means to the user visually or audibly.

[0011] The "recipe generation means" is a means for generating a cooking recipe to be suggested to the user using the identified ingredient information. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] The system of this invention efficiently manages ingredients stored in a refrigerated storage device, provides information on expiration dates and storage methods, and suggests optimal recipes using those ingredients. The system is mainly configured as follows, and each process is executed as follows.

[0034] System configuration

[0035] 1. Terminal

[0036] The terminal is a device installed inside the refrigerated storage device and has the following main functions:

[0037] Information acquisition method: Information on ingredients in the refrigerated storage device is acquired using a camera or scanner.

[0038] Data processing method: Based on the acquired ingredient information, AI technology is used to recognize ingredients and extract the necessary information.

[0039] 2. Server

[0040] The server receives and processes information sent from the terminal, manages the necessary data, and provides it to the user.

[0041] Identification method: Refer to the database and identify the expiration date and storage method from the ingredient information.

[0042] Recipe generation means: Based on parameterized ingredient information, an optimal recipe is generated and proposed to the user.

[0043] 3. User Devices

[0044] The user device is a user interface such as a smartphone or tablet, which displays and operates the information provided by the server.

[0045] Information presentation means: Visually presents information about ingredients, expiration dates, storage methods, recipes, etc. to the user.

[0046] Example of a system

[0047] 1. The device scans the refrigerator

[0048] The device uses a camera to photograph the inside of the refrigerator and recognize barcodes, labels, and shapes. During this process, ingredients such as milk, eggs, and broccoli are detected. The device analyzes and extracts information about these ingredients and sends it to the server.

[0049] 2. The server processes the information

[0050] The server receives the food ingredient information sent from the device and compares it with information in the database. For example, specific information such as "Milk (Best before date: 2023 / 10 / 15, quantity: 1L)," "Eggs (Best before date: 2023 / 10 / 10, quantity: 6 eggs)," and "Broccoli (Best before date: 2023 / 10 / 12, quantity: 1 bunch)" can be identified. It also checks whether these ingredients have been stored properly based on the expiration date.

[0051] The server then analyzes this information and calculates the optimal recipe, suggesting recipes such as "Omelette," "Cream Broccoli Soup," and "Frittata" using these ingredients.

[0052] 3. Provide information to users

[0053] The user opens a dedicated smartphone app and checks the information provided by the server. For example, a list of ingredients currently in the refrigerator and their expiration dates are displayed on the screen. The user can also check which ingredients are close to their expiration date and select a recipe that uses those ingredients. For example, the user can select a recipe for "cream broccoli soup" and proceed with the cooking process while checking the ingredients and cooking instructions in detail.

[0054] The above is a specific embodiment for implementing the system of the present invention. This system improves the efficiency of food management in a refrigerated storage device, reducing food waste and saving household money.

[0055] The processing flow will be explained below.

[0056] Step 1:

[0057] The device scans the inside of the refrigerator. It uses its built-in camera to take multiple images in maximum detail, capturing food labels, barcodes, shapes, and other features.

[0058] Step 2:

[0059] The device uses image processing algorithms to extract ingredient information from the captured image. It recognizes barcodes and labels to obtain the name, quantity, expiration date, and other information of the corresponding ingredients. Shape recognition is used to estimate the type and approximate quantity of ingredients without barcodes.

[0060] Step 3:

[0061] The device converts the extracted ingredient information into JSON format, including data fields such as ingredient name, quantity, and expiration date to ensure the information is formatted correctly.

[0062] Step 4:

[0063] The device sends the ingredient information in JSON format to the server via the network, using communication methods such as Wi-Fi or Bluetooth to send the data quickly and securely.

[0064] Step 5:

[0065] The server receives the JSON data sent from the device. The server parses the data and extracts each field (ingredients, quantity, expiration date, etc.). This analysis verifies the integrity of the data.

[0066] Step 6:

[0067] The server updates the database with the extracted information, adding new ingredient information or overwriting existing data, checking for duplicates or incomplete data and correcting it as necessary.

[0068] Step 7:

[0069] The server identifies the expiration date and storage method of the ingredients. It looks up information about each ingredient in the database (e.g., shelf life, appropriate storage temperature, etc.) to check the expiration date and best storage method of the ingredients.

[0070] Step 8:

[0071] The server uses the ingredient information to calculate and select the optimal recipe. It references recipe information in the database and suggests recipes that are best suited to the ingredients based on combinations and quantities. It also selects recipes that prioritize ingredients that are approaching their expiration date.

[0072] Step 9:

[0073] The server converts the selected recipes and related information into JSON format and sends it back to the user's device, including a list of recommended recipes, the ingredients needed, and cooking instructions.

[0074] Step 10:

[0075] The user opens the smartphone app and checks the information sent from the server. A list of ingredients in the refrigerator, their expiration dates, and storage methods are displayed on the screen, along with suggested recipes.

[0076] Step 11:

[0077] The user selects a suggested recipe and reviews its details. The user then checks the ingredients list and cooking instructions for each recipe and gets to cooking.

[0078] Through the above steps, the system of the present invention can efficiently manage food ingredients stored in a refrigerated storage device and provide optimal information and suggestions to the user.

[0079] Example 1

[0080] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0081] There is a need for a system that can efficiently manage food ingredients stored in refrigerated storage devices, provide information on expiration dates and storage methods, and even suggest optimal recipes using those ingredients. However, conventional systems have incomplete acquisition and processing of ingredient information, making it difficult to efficiently provide useful information to users. For this reason, a new system is needed to reduce food waste and save household budgets.

[0082] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0083] In this invention, the server includes the information acquisition means, data processing means for identifying the condition of ingredients based on the acquired information, identification means for identifying expiration dates and storage methods based on the identified conditions, information presentation means for providing the identified information to users, recipe generation means for generating suggested recipes using the identified ingredients, and means for transmitting the generated recipes to a user device. This makes it possible to accurately and efficiently acquire and process information about ingredients in the refrigerated storage device and provide users with useful information and optimal recipes.

[0084] The "information acquisition means" is a device that is placed inside the refrigerated storage device and acquires information about ingredients using a camera or scanner.

[0085] The "data processing means" is a device that identifies the state of ingredients based on the acquired ingredient information and extracts necessary information.

[0086] The "identification means" is a device that identifies the expiration date and storage method based on the state of the identified food material.

[0087] The "information presentation means" is a device that provides the user with information about identified ingredients, and is primarily displayed on the user's device.

[0088] A "recipe generator" is a device that generates suggested recipes using identified ingredients.

[0089] A "generative AI model" is an artificial intelligence model used to generate optimal recipes based on ingredient information.

[0090] A "prompt sentence" is a sentence used to input ingredient information into the generative AI model.

[0091] The "user device" is an interface device that displays information provided by the information presentation means and is operated by the user, and includes a smartphone, tablet, etc.

[0092] MODE FOR CARRYING OUT THE INVENTION

[0093] This invention is a system that efficiently manages ingredients stored in a refrigerated storage device, provides information on expiration dates and storage methods, and suggests optimal recipes using those ingredients. The system is primarily composed of a terminal, a server, and a user device.

[0094] Information acquisition by terminal

[0095] The terminal is a device placed inside the refrigerated storage unit, and its role is to obtain information about ingredients using a camera or scanner. Specifically, a camera installed inside the refrigerator periodically photographs ingredients and obtains image data. Barcode and label information is extracted from the image data to obtain basic information about the ingredients (product name, expiration date, quantity, etc.). For example, the terminal photographs a milk package and scans the barcode to obtain the information "Milk (Expiration date: 2023 / 10 / 15, quantity: 1L)."

[0096] Data processing and identification by the server

[0097] The ingredient information sent from the device is received by the server. The server analyzes this information and checks the database to identify detailed information about the ingredient (such as expiration date and storage method). For example, the server receives information about "eggs (expiration date: 2023 / 10 / 10, quantity: 6 pieces)" and refers to the database to identify the "refrigerated storage" method. The server then uses an algorithm to evaluate the ingredient's storage status and generate appropriate warnings.

[0098] Recipe Generation Method

[0099] The server also generates optimal recipes based on the identified ingredients. Specifically, it uses a generative AI model (e.g., an open-source generative AI model) to suggest recipes based on ingredient combinations. For example, a user might input a prompt such as "Please tell me a recipe that uses milk, eggs, and broccoli," and the server would suggest recipes such as "omelette," "cream broccoli soup," and "fritatta." The generated recipes are then sent from the server to the user device.

[0100] How to provide information

[0101] Users can check the provided information and recipes through their smartphones, tablets, or other user devices. For example, when a user opens the dedicated app, a list of ingredients currently in the refrigerator and expiration dates are displayed. Users can also check ingredients that are close to their expiration date and select recipes that use those ingredients.

[0102] Specific prompt examples:

[0103] "Can you tell me a recipe that uses the milk, eggs, and broccoli I have in my fridge?"

[0104] "Please suggest a dish that can be made using ingredients that are close to their expiration date."

[0105] The above is a specific embodiment for implementing the system of the present invention. This system allows users to efficiently manage food ingredients in a refrigerated storage device and reduce food waste.

[0106] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0107] Step 1: Obtaining ingredient information on the device

[0108] The terminal periodically photographs ingredients using a camera placed inside the refrigerated storage device.

[0109] Input: Image of the inside of the refrigerator

[0110] Output: Captured image data

[0111] Specific operation: The device takes a full image of the inside of the refrigerated storage device once in the morning and once in the afternoon. Specifically, the timer module in the device activates the camera at a set time to capture images inside the storage device.

[0112] Step 2: Extracting ingredient information on the device

[0113] The device analyzes the image data it acquires and extracts barcode and label information.

[0114] Input: Photographed image data

[0115] Output: Basic information about ingredients (product name, expiration date, quantity, etc.)

[0116] Specific operation: Using image recognition software (e.g., OpenCV) within the device, barcodes and labels are detected from the image. The barcode reader scans the barcode information and extracts data such as the product name, expiration date, and quantity.

[0117] Step 3: Send data from the device to the server

[0118] The terminal transmits the extracted ingredient information to the server.

[0119] Input: Basic information about ingredients (product name, expiration date, quantity, etc.)

[0120] Output: Sending ingredient information to the server

[0121] Specific operation: The communication module in the device formats the extracted data into JSON format and sends it to the server using the HTTP protocol. For example, it contains information such as "Milk (expiration date: 2023 / 10 / 15, quantity: 1L)."

[0122] Step 4: Server receives and stores information

[0123] The server receives the ingredient information sent from the terminal and stores it in a database.

[0124] Input: JSON formatted ingredient information

[0125] Output: Saved ingredient information

[0126] What happens: The server receives the HTTP request, parses the JSON data, and saves it in the database. Specifically, the server's API decodes the ingredient information and saves it in the corresponding fields in the database.

[0127] Step 5: Server analyzes ingredient information

[0128] The server references the database to identify detailed information about the ingredients (such as expiration date and storage method).

[0129] Input: Saved ingredient information

[0130] Output:Detailed information about ingredients

[0131] What happens: The server queries the database to identify the expiration date and storage method for the stored item. For example, searching for "milk" returns "Expiration date: 10 / 15 / 2023, Storage method: Refrigerate."

[0132] Step 6: Server evaluation of preservation status

[0133] The server uses an algorithm to assess the food's storage status and generate appropriate warnings.

[0134] Input:Detailed information about ingredients

[0135] Output: Preservation status evaluation results and warning messages

[0136] Specific operation: The server runs a storage condition assessment algorithm to check whether the food is stored properly. For example, if a food item that needs to be refrigerated is in the freezer, it generates a warning. A warning message is generated and sent to the user device.

[0137] Step 7: Generate the recipe

[0138] The server uses a generative AI model to generate an optimal recipe based on the identified ingredients.

[0139] Input: Identified ingredient information

[0140] Output: The generated recipe

[0141] Specific operation: The server generates a prompt based on the ingredient information and inputs it into the generative AI model. For example, the server generates a prompt such as "Please tell me a recipe using the following ingredients: milk, eggs, and broccoli." The generative AI model then suggests recipes such as "omelette" and "cream broccoli soup."

[0142] Step 8: Provide information to user devices

[0143] The server sends the generated recipe and ingredient information to the user's device, and the user checks the information in the app.

[0144] Input: Generated recipe and ingredient information

[0145] Output: Recipe and ingredient information displayed on the user's device

[0146] How it works: The server sends the generated recipe information through the API, and the user device receives it. The user can open the app and check the list of ingredients in the refrigerator, their expiration dates, and suggested recipes.

[0147] The above are the specific processing steps of the program of this system.

[0148] (Application example 1)

[0149] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0150] Managing ingredients in refrigerated storage devices requires a lot of time and effort, making it difficult for users to use food efficiently. There is also a high possibility of food waste occurring when ingredients that have passed their expiration date are discarded. Furthermore, it is difficult to link information about ingredients within the home with an external recipe database, and there is no system that can easily suggest optimal recipes. This creates the problem of not making effective use of ingredients and making it difficult to save money on household expenses.

[0151] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0152] In this invention, the server includes a recipe suggestion unit that suggests recipes using a generative AI model that generates optimal recipes based on ingredient information, and an input unit for inputting a prompt statement to the recipe suggestion unit. This allows users to easily obtain optimal cooking methods that utilize ingredients within their expiration dates based on the ingredient information in the refrigerated storage device. It also reduces food waste and promotes efficient and economical use of ingredients.

[0153] A "refrigerated storage device" is a device for storing items such as food and beverages at a constant low temperature.

[0154] "Information acquisition means" refers to a device for acquiring information about ingredients in the refrigerated storage device, such as a camera or scanner.

[0155] The "data processing means" is a device or software for analyzing the data obtained by the information acquisition means and identifying the state of the ingredients.

[0156] The "identification means" is a device or program for identifying the expiration date and appropriate storage method based on the condition of the identified food material.

[0157] The "information presentation means" is an interface for providing the identified information to the user, and includes devices such as smartphones and tablets.

[0158] The "recipe generation means" is a device or program for creating an optimal cooking recipe based on the identified ingredient information.

[0159] A "generative AI model" is an artificial intelligence model used to generate optimal recipes based on ingredient information.

[0160] A "recipe suggestion means" is a device or program that uses a generative AI model to suggest optimal recipes to users.

[0161] The "input means" is an interface for inputting a prompt sentence to the recipe suggestion means, and includes a keyboard and a touch screen.

[0162] A "prompt" is a text statement that provides specific instructions to the generative AI model, describing specific conditions or requirements for the desired recipe.

[0163] This system efficiently manages information about ingredients stored in refrigerated storage devices, provides information about expiration dates and storage methods, and even suggests optimal recipes using those ingredients. The specific configuration and processing flow of the system are described below.

[0164] 1. System Configuration

[0165] 1.1 Terminal

[0166] The terminal functions as an information acquisition means placed inside the refrigerated storage device, and acquires information about the ingredients in the refrigerated storage device using a camera or scanner.

[0167] 1.2 Server

[0168] The server is a central device that includes multiple means, specifically, a recipe suggestion means that uses a generative AI model to generate optimal recipes based on ingredient information, and an input means for inputting a prompt sentence to the recipe suggestion means.

[0169] 1.3 User Devices

[0170] The user device acts as an information presentation means, such as a smartphone or tablet, and provides an interface for displaying the identified information and suggested recipes.

[0171] 2. Program Processing

[0172] 2.1 Information acquisition

[0173] A camera installed in the terminal periodically scans and captures images of the food items in the refrigerated storage unit, and image recognition technology is used to identify these items and determine their condition through data processing means.

[0174] 2.2 Data Processing

[0175] The server compares the ingredient information sent from the terminal with the database to identify the expiration date and storage method of the specific ingredient, and provides this identified information to the user through a user interface.

[0176] 2.3 Recipe Generation

[0177] The server generates an optimal recipe using a generative AI model based on the identified ingredient information. The generated recipe can be made more specific by the user inputting a prompt sentence.

[0178] 2.4 Information presentation

[0179] Using a user device such as a smartphone or tablet, the user can check the list of ingredients currently in the refrigerator, their expiration date, storage instructions, and the generated recipe. The information displayed on the user device allows the user to use ingredients efficiently.

[0180] 3. Specific Examples

[0181] For example, suppose you have the following ingredients in your refrigerator:

[0182] Milk (Expiration date: 2023 / 10 / 15, Quantity: 1L)

[0183] Eggs (Best before: 2023 / 10 / 10, Quantity: 6)

[0184] Broccoli (Best before date: 2023 / 10 / 12, Quantity: 1 bunch)

[0185] To generate the best recipe using these ingredients, the user can enter the following prompt:

[0186] Create a recipe that meets the following conditions: You have the following ingredients in your refrigerator.

[0187] Milk (Expiration date: 2023 / 10 / 15, Quantity: 1L)

[0188] Eggs (Best before: 2023 / 10 / 10, Quantity: 6)

[0189] Broccoli (Best before date: 2023 / 10 / 12, Quantity: 1 bunch)

[0190] Prioritize ingredients that are close to their expiration date and suggest recipes that are easy to prepare.

[0191] By inputting such a prompt, the generative AI model generates recipes that make effective use of ingredients and displays them on the user's device.

[0192] Hardware and software used

[0193] Examples of the main hardware and software used in this system include:

[0194] Camera: Used to obtain information about ingredients in the refrigerated storage device.

[0195] Smartphone / Tablet: Provides the user interface.

[0196] Information acquisition method: Ingredient recognition using computer vision technology (e.g., OpenCV).

[0197] Data processing means: Processing and analyzing ingredient information (e.g., TENSORFLOW (registered trademark), Python).

[0198] Recipe suggestion method: Recipes are suggested using a generative AI model (e.g., generative AI model).

[0199] Means of information presentation: Building a user interface (e.g., React Native).

[0200] This configuration allows users to more efficiently manage ingredients in their refrigerators and easily retrieve and execute optimal recipes, thereby reducing food waste and saving money.

[0201] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0202] Step 1:

[0203] The terminal scans the inside of the refrigerated storage unit. A camera installed on the terminal periodically photographs the food inside the refrigerated storage unit and captures images. The captured images are sent to a server for data processing. The input is a video of the food inside the refrigerated storage unit, and the output is the image data required for analysis.

[0204] Step 2:

[0205] The server receives the image data. Based on the received image data, it uses computer vision technology to identify ingredients. Specific data processing involves analyzing barcodes and shapes using image recognition algorithms, and identifying ingredient information by matching it with a database. The input is the acquired image data, and the output is the analyzed ingredient information.

[0206] Step 3:

[0207] The server identifies the expiration date and storage method based on the ingredient information. It references the database to obtain information about the ingredient's expiration date and storage method. The input is the identified ingredient information, and the output is the identified expiration date and storage method data.

[0208] Step 4:

[0209] The server manages the identified ingredient information and sends it to the user device. A display interface on the user device allows the user to visually check the current ingredient status, expiration date, and storage method. The input is the identified expiration date and storage method data, and the output is the display data on the user device.

[0210] Step 5:

[0211] The server generates optimal recipes based on ingredient information. Using a generative AI model, it suggests recipes based on prompts entered by the user. The input is the identified ingredient information and prompts, and the output is the generated recipe information.

[0212] Step 6:

[0213] Users can receive more specific recipe suggestions by entering a prompt using the input means of a smartphone or tablet. For example, they can enter, "Please suggest a simple recipe using milk, eggs, and broccoli." The input is the user's prompt, and the output is a recipe suggestion from the generative AI model.

[0214] Step 7:

[0215] The server provides the generated recipe information to the user device, and the user confirms and selects it. The optimal recipe based on the expiration date is displayed on the user device. The input is the generated recipe information, and the output is the display data on the user device.

[0216] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0217] The system of this invention not only efficiently manages food ingredients in a refrigerated storage device and provides information on expiration dates and storage methods, but also has the function of recognizing the user's emotions and adjusting the way information is presented and recipe suggestions based on those emotions. The system is mainly composed of the following elements, which operate in an integrated manner:

[0218] System configuration

[0219] 1. Terminal

[0220] The terminal is a device installed inside the refrigerated storage unit and has the following functions:

[0221] Information acquisition method: Use cameras and scanners to acquire information about ingredients in the refrigerated storage unit. Use barcode scanning and image recognition technology.

[0222] Data processing means: Identify the type, quantity, expiration date, etc. of ingredients from the acquired ingredient information.

[0223] Emotion engine: Analyzes the user's facial images and voice data to recognize emotions.

[0224] 2. Server

[0225] The server receives and processes information sent from the terminal, manages the necessary data, and provides it to the user.

[0226] Identification method: Refer to the database and identify the expiration date and storage method from the ingredient information.

[0227] Recipe generation means: Generates optimal recipes based on ingredient information and suggests them to the user.

[0228] Emotion-based adjustment: The recipe suggestions and information presentation method are adjusted based on the user's emotional information.

[0229] 3. User Devices

[0230] The user device is a user interface such as a smartphone or tablet, which displays and operates the information provided by the server.

[0231] Information presentation means: Visually presents information about ingredients, expiration dates, storage methods, recipes, etc. to the user.

[0232] Example of a system

[0233] 1. The device scans the refrigerator

[0234] The device uses a camera to take pictures of the inside of the refrigerator and recognizes barcodes, labels, and shapes. For example, milk, eggs, and broccoli are detected. The device analyzes this information and sends it to a server.

[0235] 2. The server processes the information

[0236] The server receives the food ingredient information sent from the device and compares it with a database to identify information such as "Milk (best before date: 2023 / 10 / 15, quantity: 1L)," "Eggs (best before date: 2023 / 10 / 10, quantity: 6 eggs)," and "Broccoli (best before date: 2023 / 10 / 12, quantity: 1 bunch)," and also confirms the storage method.

[0237] The server also calculates and suggests optimal recipes based on the ingredient information, such as "Omelette," "Cream Broccoli Soup," and "Frittata."

[0238] 3. User Emotion Recognition and Information Provision

[0239] When a user opens the smartphone app, the device captures the user's facial image and voice via the camera and microphone, and identifies their emotion using an emotion engine. For example, if the server recognizes that the user is feeling stressed, it will use that information to present simple recipes for relaxation, along with soothing color schemes and audio.

[0240] 4. Customizing the user experience

[0241] The emotion engine continuously monitors the user's emotional state and provides the optimal recipes and information presentation methods depending on the situation. For example, it will suggest quick meals when the user is tired, and new and challenging recipes when the user has time, thereby enriching the user experience.

[0242] The above is a concrete example of how to implement the system of the present invention, which combines an emotion engine. This system further improves the efficiency of food management in refrigerated storage devices and provides personalized support according to the user's emotions.

[0243] The processing flow will be explained below.

[0244] Step 1:

[0245] The device scans the inside of the refrigerator. It uses its built-in camera to take detailed photos of the food inside the refrigerator. It then recognizes barcodes, labels, and shapes from the captured images.

[0246] Step 2:

[0247] The device uses image processing technology to extract ingredient information from the captured image. It scans barcodes to obtain information such as the ingredient name, quantity, and expiration date. Shape recognition is used to estimate the type and approximate quantity of ingredients without barcodes.

[0248] Step 3:

[0249] The device converts the extracted ingredient information into JSON format. The correctly formatted data includes fields such as ingredient name, quantity, and expiration date.

[0250] Step 4:

[0251] The device sends JSON-formatted ingredient information to the server via the network, using communication methods such as Wi-Fi and Bluetooth to send the data securely and quickly.

[0252] Step 5:

[0253] The server receives the JSON data sent from the device. The server analyzes the data and extracts information such as the ingredient name, quantity, and expiration date. The integrity of the data is then confirmed.

[0254] Step 6:

[0255] The server updates the database with the extracted information, adds new ingredient information or overwrites existing data, and checks for duplicates or incomplete data and corrects it as necessary.

[0256] Step 7:

[0257] The server identifies the expiration date and storage method of the ingredients, and checks the expiration date and optimal storage method by looking up information in the database about the storage period and appropriate storage temperature.

[0258] Step 8:

[0259] The server uses the ingredient information to generate the optimal recipe. It searches the recipe information in the database and suggests the optimal recipe based on the combination and quantity. When doing so, it gives priority to ingredients that are approaching their expiration date.

[0260] Step 9:

[0261] The server generates recipes and related information, converts them into JSON format, and sends them to the user's device. The recipes include recommended dishes, required ingredients, and cooking instructions.

[0262] Step 10:

[0263] When a user opens the smartphone app, the emotion engine activates via the camera and microphone. The device captures the user's facial image and voice, which the emotion engine analyzes to recognize the user's emotions.

[0264] Step 11:

[0265] The server receives the user's emotional state and adjusts the information presentation method and recipe suggestions accordingly. For example, if the user is feeling stressed, it will use simple recipes that are relaxing and a visually friendly color scheme.

[0266] Step 12:

[0267] The information presentation means displays the adjusted content to the user. The user browses the suggested recipes and information on ingredients and checks the details. The user selects a recipe that interests them, checks the details, and actually cooks the dish.

[0268] Through these processing steps, the system can efficiently manage ingredients in the refrigerator storage device and provide information and recipes that respond to the user's emotions.

[0269] Example 2

[0270] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0271] Conventional food management systems for refrigerated storage devices mainly focus on managing expiration dates and storage methods for ingredients, making it difficult to adjust the information content and recipe suggestions provided according to the user's emotional state. As a result, uniform information and recipes are provided without taking into account the user's current emotions and state, which creates a problem of not being able to improve the user experience.

[0272] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0273] In this invention, the server includes an information acquisition means, a data processing means for identifying the condition of ingredients based on the acquired information, an identification means for identifying expiration dates and storage methods based on the identified conditions, an information presentation means for providing the identified information to the user, a recipe generation means for generating suggested recipes using the identified ingredients, an emotion recognition means for recognizing the emotion of the user, and an emotion adjustment means for adjusting the recipe suggestions and information presentation method based on the recognized emotion. This not only enables efficient management of ingredients in the refrigerated storage device, but also enables personalized information and recipe suggestions according to the user's emotional state.

[0274] The "information acquisition means" refers to a device such as a camera or scanner for acquiring information about ingredients in the refrigerated storage device.

[0275] The "data processing means" is a combination of software and hardware for identifying the state of ingredients based on the acquired ingredient information.

[0276] The "identification means" is a device or algorithm for identifying the expiration date and storage method based on the condition of the identified food material.

[0277] "Information presentation means" refers to a user interface such as a display or smartphone app that provides the identified information to the user.

[0278] The "recipe generation means" refers to software or algorithms that generate and suggest optimal recipes using the identified ingredients.

[0279] "Emotion recognition means" refers to software or algorithms that analyze a user's facial image and voice data to recognize their emotions.

[0280] An "emotion adjustment means" is a combination of software and hardware for adjusting recipe suggestions and information presentation methods based on recognized emotions.

[0281] The system of this invention not only efficiently manages food ingredients in a refrigerated storage device and provides information on expiration dates and storage methods, but also has the function of recognizing the user's emotions and adjusting the way information is presented and recipe suggestions based on those emotions. This system is composed of the following elements, which operate in an integrated manner.

[0282] System configuration

[0283] 1. Terminal

[0284] The terminal is a device installed inside the refrigerated storage unit and has the following functions: The terminal uses a camera or scanner as an information acquisition means to acquire information about ingredients inside the refrigerated storage unit. For example, it uses barcode scanning and image recognition technology. The terminal processes the acquired ingredient information using a data processing means to identify the type, quantity, expiration date, etc. of the ingredients.

[0285] Furthermore, the device recognizes emotions by analyzing the user's facial images and voice data, using image recognition technology and voice analysis software.

[0286] 2. Server

[0287] The server receives and processes the information sent from the terminal and manages the necessary data. The server's identification means compares the received ingredient information with a database and identifies detailed ingredient information (e.g., expiration date and storage method). The identified information is provided to the user device using the information presentation means.

[0288] The server's recipe generation means generates optimal recipes based on the ingredient information and suggests them to the user. Furthermore, the emotion adjustment means adjusts the content of the recipe suggestions and the way in which the information is presented based on the user's emotion information. For example, if the user is tired, the server will suggest simple recipes that can be prepared in a short time.

[0289] 3. User Devices

[0290] The user device is a user interface such as a smartphone or tablet. The user device displays and operates information provided by the server. Information such as ingredient information, expiration dates, storage methods, and recipes are visually presented to the user through the information presentation means.

[0291] Specific examples

[0292] 1. The device scans the refrigerator

[0293] The device uses a camera to take pictures of the inside of the refrigerator and recognizes barcodes, labels, and shapes. For example, milk, eggs, and broccoli are detected. The device analyzes this information and sends it to a server.

[0294] 2. The server processes the information

[0295] The server receives the ingredient information sent from the device and compares it with the database to identify information such as "Milk (Expiration date: 2023 / 10 / 15, Amount: 1L)," "Eggs (Expiration date: 2023 / 10 / 10, Amount: 6 eggs)," and "Broccoli (Expiration date: 2023 / 10 / 12, Amount: 1 bunch)." The server then calculates and suggests the optimal recipe based on the ingredient information. For example, it might suggest "Omelette," "Cream Broccoli Soup," or "Frittata."

[0296] 3. User Emotion Recognition and Information Provision

[0297] When a user opens the smartphone app, the device captures the user's facial image and voice via the camera and microphone, and identifies their emotion using an emotion engine. For example, if the server recognizes that the user is feeling stressed, it will use that information to present simple recipes for relaxation, along with soothing color schemes and audio.

[0298] Prompt Sentence Examples

[0299] "Imagine a system that not only efficiently manages the ingredients in your refrigerator and tells you expiration dates and storage methods, but also recognizes your emotions and suggests the best recipes based on those emotions. For example, it could suggest quick meals when you're tired, and more challenging recipes when you have the time."

[0300] The system of the present invention makes food management in a refrigerated storage device more efficient and can provide personalized support that responds to the user's emotions.

[0301] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0302] Step 1:

[0303] The device obtains information about ingredients in the refrigerator. When the user opens the refrigerator door, the device's camera and scanner are automatically activated. For example, the camera takes a photo of the food in the refrigerator, and the scanner reads the barcode. The input data are images of the ingredients in the refrigerator and their barcode information. Based on this input data, the device uses image recognition technology to identify the ingredient information and format it as text data. The output is text information that includes the type, quantity, and expiration date of each ingredient.

[0304] Step 2:

[0305] The device sends the acquired ingredient information to the server. The input data is text-format ingredient information generated on the device. The device then sends this information to the server via the network. The data received by the server is text data containing detailed information about the ingredients. Specifically, the data is sent securely using a data transmission protocol (e.g., HTTP, HTTPS).

[0306] Step 3:

[0307] The server processes the ingredient information. The input data is the text-formatted ingredient information sent from the terminal. The server's identification means checks the information against a database to identify detailed information about the ingredient. For example, a database query is used to obtain detailed information such as "Milk (best before date: 2023 / 10 / 15, quantity: 1L)." As output, detailed information about the identified ingredient is generated in text format. Specifically, an SQL query is executed on the server to extract the required information from the database.

[0308] Step 4:

[0309] The server's recipe generation means generates an optimal recipe based on the ingredient information. The input data is detailed information about the identified ingredients. The server uses an algorithm to calculate an optimal recipe based on the current ingredients. For example, dish names such as "Omelette," "Cream Broccoli Soup," and "Frittata" are suggested. The output is a list of suggested recipes. Specifically, the server checks the recipe database programmatically and selects the optimal recipe.

[0310] Step 5:

[0311] A user opens a smartphone app and performs emotion recognition. The input data is the user's facial image and voice data. The device's camera and microphone capture this data and send it to the device's emotion recognition means. The device identifies the emotion using facial expression analysis software and voice analysis software. The output is analyzed emotional information. For example, it may determine that the user is "feeling stressed" based on facial expression or voice tone.

[0312] Step 6:

[0313] The server receives the emotional information and adjusts the information presentation method and recipe suggestions. The input data is the emotional information sent from the device. The server's emotion adjustment means uses this information to adjust the content and method of information provided to the user. For example, a user in a stressed state might be suggested "meals that can be made in a short time" or "easy recipes that will help them relax." The output generates a customized recipe list and visually and audibly adjusted information presentation. In concrete terms, the server compares the information with a recipe database and selects the recipe that best suits the user's emotional state, while also adjusting the color scheme of the information presentation screen and the audio guidance.

[0314] Step 7:

[0315] The user's experience is executed. Input data is a customized recipe list and adjusted information presentation sent from the server. The user receives this information through a smartphone app, selects a dish, and begins cooking. Specific operations include displaying the recipe on the user's smartphone, allowing them to cook according to their mood and situation that day.

[0316] (Application example 2)

[0317] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0318] Conventional refrigerated storage devices make it difficult to efficiently manage expiration dates and storage conditions of ingredients, leading to users often wasting expired ingredients. While recipe suggestions based on ingredient information were offered, they were unable to tailor suggestions to individual users' emotions or circumstances, resulting in low satisfaction. Furthermore, the time required to manually order missing ingredients meant that these systems lacked convenience.

[0319] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0320] In this invention, the server includes an emotion engine that recognizes emotions, an adjustment means that adjusts the content of suggestions and the method of information presentation based on the user's emotion information, and a delivery ordering means that automatically orders ingredients that are in short supply, thereby enabling personalized recipe suggestions and automatic delivery ordering according to the user's emotions.

[0321] The "information acquisition means" is a device that is placed inside the refrigerated storage device and acquires food ingredient information using barcode scanning or image recognition technology.

[0322] The "data processing means" is a device that analyzes the acquired ingredient information and identifies the condition of the ingredients, such as type, quantity, and expiration date.

[0323] The "identification means" is a device that identifies the expiration date and storage method based on the state of the identified food material.

[0324] The "information presentation means" is a device for providing the identified information to the user.

[0325] The "recipe generation means" is a device that generates and proposes optimal recipes based on the identified ingredient information.

[0326] An "emotion engine" is a device that analyzes a user's facial images and voice data to recognize their emotions.

[0327] The "adjustment means" is a device that adjusts the content of recipe suggestions and the method of presenting information based on the user's emotional information.

[0328] The "delivery ordering means" is a device that automatically places an additional order when ingredients are in short supply.

[0329] The system of this invention not only efficiently manages food ingredients in a refrigerated storage device and provides information on expiration dates and storage methods, but also has the function of recognizing the user's emotions and adjusting the way information is presented and recipe suggestions based on those emotions. The main elements of the system are a terminal installed in the refrigerated storage device, a server that processes data, and a user device.

[0330] Terminal

[0331] The terminal is located within the refrigerated storage unit and has the following functions:

[0332] 1. Information acquisition method: Using a smartphone camera, food items in the refrigerator are scanned. Image recognition technology and barcode scanning are used to acquire food information (name, quantity, expiration date).

[0333] 2. Data processing means: Analyze the acquired food ingredient information and identify the type, quantity, expiration date, etc.

[0334] server

[0335] The server receives the information sent by the terminal and performs the following functions:

[0336] 1. Identification method: Based on the acquired data, the expiration date and storage method are checked against a database to identify the product.

[0337] 2. Recipe generation means: Generates optimal recipes using the identified ingredients and suggests them to the user.

[0338] 3. Emotion Engine: Analyzes the user's facial image and voice data to recognize emotions. This is done using EmotionRecognizer (software for emotion recognition).

[0339] 4. Adjustment method: Based on the information obtained from the emotion engine, recipe suggestions and the way information is presented are adjusted.

[0340] 5. Delivery ordering method: Automatically order missing ingredients from a food delivery service.

[0341] User Device

[0342] The user device (smartphone or tablet) is used to display and operate the information provided by the server.

[0343] 1. Information presentation means: Visually present information about ingredients, expiration dates, storage methods, recipes, etc. to the user.

[0344] Specific examples

[0345] When a user scans ingredients in a refrigerator, information such as "Milk (Best before date: 2023 / 10 / 15, Quantity: 1L)," "Eggs (Best before date: 2023 / 10 / 10, Quantity: 6 eggs)," and "Broccoli (Best before date: 2023 / 10 / 12, Quantity: 1 bunch)" is obtained. This information is sent from the device to the server and registered in a database along with storage instructions. Based on this information, the server generates recipes such as "Omelette" and "Cream Broccoli Soup" and suggests them to the user.

[0346] When a user opens the smartphone app and captures their facial images and voice via the camera and microphone, the emotion engine identifies the user's emotions. For example, if the app detects that the user is feeling stressed, it will present information such as simple, relaxing recipes with soothing colors and sounds. If the suggested recipe is missing ingredients, the delivery ordering method will automatically order the missing ingredients.

[0347] Prompt Sentence Examples

[0348] Ingredients needed: 1L milk, 6 eggs, 1 bunch of broccoli, 200g cheese

[0349] Best before date: Milk 2023 / 10 / 15, Eggs 2023 / 10 / 10, Broccoli 2023 / 10 / 12

[0350] User Emotion: Tired

[0351] Generate suggested recipes.

[0352] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0353] Step 1:

[0354] The terminal scans ingredients in the refrigerator. Using the smartphone camera, it obtains ingredient information using barcode scanning and image recognition technology. The data obtained includes the ingredient's name, quantity, and expiration date. For example, the camera scans the barcode on a piece of milk and obtains the information "Milk (quantity: 1L, expiration date: 2023 / 10 / 15)."

[0355] Step 2:

[0356] The device analyzes the acquired ingredient information using a data processing means and sends the identified ingredient information to the server. The input is ingredient information (name, quantity, expiration date), and the output is a data packet containing this information. For example, a data packet such as "Eggs (quantity: 6, expiration date: 2023 / 10 / 10)" is sent to the server.

[0357] Step 3:

[0358] The server receives the data packet and uses an identification means to compare it with the database. The data calculation performed here is to identify the optimal storage method based on the ingredient name, quantity, and expiration date. For example, for "broccoli (quantity: 1 bunch, expiration date: 2023 / 10 / 12)", the storage method is identified as "refrigerated storage."

[0359] Step 4:

[0360] The server sends the identified storage information and expiration date information to the user device, and the information presentation means presents it to the user. The input is the identified storage method and expiration date information, and the output is the display on the user device. For example, the information "Broccoli (storage method: refrigerated, expiration date: 2023 / 10 / 12)" is displayed on a smartphone.

[0361] Step 5:

[0362] The server's recipe generation means generates an optimal recipe based on the identified ingredient information. The input is ingredient information and storage method, and the output is the generated recipe. For example, it generates recipes for "omelette" or "cream broccoli soup" based on "milk (amount: 1L), eggs (amount: 6), broccoli (amount: 1 bunch)".

[0363] Step 6:

[0364] A user opens a smartphone app and sends facial images and voice data to the emotion engine via the camera and microphone. The input is the user's facial image and voice data, and the output is the identified emotion. For example, the emotion engine may identify the user as "tired."

[0365] Step 7:

[0366] The server's adjustment means adjusts the recipe suggestions and information presentation method based on the identified emotional information. The input is the user's emotional information and ingredient information, and the output is the adjusted recipe suggestions and information presentation method. For example, for a tired user, information is presented that includes "easy recipes" and "relaxing color schemes."

[0367] Step 8:

[0368] The user checks the proposed recipe, and if there are missing ingredients, the delivery ordering method automatically orders the missing ingredients from a food delivery service. The input is a list of the missing ingredients, and the output is an order confirmation to the delivery service. For example, "200g of cheese" and "3 tomatoes" can be automatically ordered.

[0369] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0370] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0371] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0372] [Second embodiment]

[0373] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0374] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0375] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0376] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0377] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0378] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0379] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0380] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0381] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0382] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0383] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0384] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0385] The system of this invention efficiently manages ingredients stored in a refrigerated storage device, provides information on expiration dates and storage methods, and suggests optimal recipes using those ingredients. The system is mainly configured as follows, and each process is executed as follows.

[0386] System configuration

[0387] 1. Terminal

[0388] The terminal is a device installed inside the refrigerated storage device and has the following main functions:

[0389] Information acquisition method: Information on ingredients in the refrigerated storage device is acquired using a camera or scanner.

[0390] Data processing method: Based on the acquired ingredient information, AI technology is used to recognize ingredients and extract the necessary information.

[0391] 2. Server

[0392] The server receives and processes information sent from the terminal, manages the necessary data, and provides it to the user.

[0393] Identification method: Refer to the database and identify the expiration date and storage method from the ingredient information.

[0394] Recipe generation means: Based on parameterized ingredient information, an optimal recipe is generated and proposed to the user.

[0395] 3. User Devices

[0396] The user device is a user interface such as a smartphone or tablet, which displays and operates the information provided by the server.

[0397] Information presentation means: Visually presents information about ingredients, expiration dates, storage methods, recipes, etc. to the user.

[0398] Example of a system

[0399] 1. The device scans the refrigerator

[0400] The device uses a camera to photograph the inside of the refrigerator and recognize barcodes, labels, and shapes. During this process, ingredients such as milk, eggs, and broccoli are detected. The device analyzes and extracts information about these ingredients and sends it to the server.

[0401] 2. The server processes the information

[0402] The server receives the food ingredient information sent from the device and compares it with information in the database. For example, specific information such as "Milk (Best before date: 2023 / 10 / 15, quantity: 1L)," "Eggs (Best before date: 2023 / 10 / 10, quantity: 6 eggs)," and "Broccoli (Best before date: 2023 / 10 / 12, quantity: 1 bunch)" can be identified. It also checks whether these ingredients have been stored properly based on the expiration date.

[0403] The server then analyzes this information and calculates the optimal recipe, suggesting recipes such as "Omelette," "Cream Broccoli Soup," and "Frittata" using these ingredients.

[0404] 3. Provide information to users

[0405] The user opens a dedicated smartphone app and checks the information provided by the server. For example, a list of ingredients currently in the refrigerator and their expiration dates are displayed on the screen. The user can also check which ingredients are close to their expiration date and select a recipe that uses those ingredients. For example, the user can select a recipe for "cream broccoli soup" and proceed with the cooking process while checking the ingredients and cooking instructions in detail.

[0406] The above is a specific embodiment for implementing the system of the present invention. This system improves the efficiency of food management in a refrigerated storage device, reducing food waste and saving household money.

[0407] The processing flow will be explained below.

[0408] Step 1:

[0409] The device scans the inside of the refrigerator. It uses its built-in camera to take multiple images in maximum detail, capturing food labels, barcodes, shapes, and other features.

[0410] Step 2:

[0411] The device uses image processing algorithms to extract ingredient information from the captured image. It recognizes barcodes and labels to obtain the name, quantity, expiration date, and other information of the corresponding ingredients. Shape recognition is used to estimate the type and approximate quantity of ingredients without barcodes.

[0412] Step 3:

[0413] The device converts the extracted ingredient information into JSON format, including data fields such as ingredient name, quantity, and expiration date to ensure the information is formatted correctly.

[0414] Step 4:

[0415] The device sends the ingredient information in JSON format to the server via the network, using communication methods such as Wi-Fi or Bluetooth to send the data quickly and securely.

[0416] Step 5:

[0417] The server receives the JSON data sent from the device. The server parses the data and extracts each field (ingredients, quantity, expiration date, etc.). This analysis verifies the integrity of the data.

[0418] Step 6:

[0419] The server updates the database with the extracted information, adding new ingredient information or overwriting existing data, checking for duplicates or incomplete data and correcting it as necessary.

[0420] Step 7:

[0421] The server identifies the expiration date and storage method of the ingredients. It looks up information about each ingredient in the database (e.g., shelf life, appropriate storage temperature, etc.) to check the expiration date and best storage method of the ingredients.

[0422] Step 8:

[0423] The server uses the ingredient information to calculate and select the optimal recipe. It references recipe information in the database and suggests recipes that are best suited to the ingredients based on combinations and quantities. It also selects recipes that prioritize ingredients that are approaching their expiration date.

[0424] Step 9:

[0425] The server converts the selected recipes and related information into JSON format and sends it back to the user's device, including a list of recommended recipes, the ingredients needed, and cooking instructions.

[0426] Step 10:

[0427] The user opens the smartphone app and checks the information sent from the server. A list of ingredients in the refrigerator, their expiration dates, and storage methods are displayed on the screen, along with suggested recipes.

[0428] Step 11:

[0429] The user selects a suggested recipe and reviews its details. The user then checks the ingredients list and cooking instructions for each recipe and gets to cooking.

[0430] Through the above steps, the system of the present invention can efficiently manage food ingredients stored in a refrigerated storage device and provide optimal information and suggestions to the user.

[0431] Example 1

[0432] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0433] There is a need for a system that can efficiently manage food ingredients stored in refrigerated storage devices, provide information on expiration dates and storage methods, and even suggest optimal recipes using those ingredients. However, conventional systems have incomplete acquisition and processing of ingredient information, making it difficult to efficiently provide useful information to users. For this reason, a new system is needed to reduce food waste and save household budgets.

[0434] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0435] In this invention, the server includes the information acquisition means, data processing means for identifying the condition of ingredients based on the acquired information, identification means for identifying expiration dates and storage methods based on the identified conditions, information presentation means for providing the identified information to users, recipe generation means for generating suggested recipes using the identified ingredients, and means for transmitting the generated recipes to a user device. This makes it possible to accurately and efficiently acquire and process information about ingredients in the refrigerated storage device and provide users with useful information and optimal recipes.

[0436] The "information acquisition means" is a device that is placed inside the refrigerated storage device and acquires information about ingredients using a camera or scanner.

[0437] The "data processing means" is a device that identifies the state of ingredients based on the acquired ingredient information and extracts necessary information.

[0438] The "identification means" is a device that identifies the expiration date and storage method based on the state of the identified food material.

[0439] The "information presentation means" is a device that provides the user with information about identified ingredients, and is primarily displayed on the user's device.

[0440] A "recipe generator" is a device that generates suggested recipes using identified ingredients.

[0441] A "generative AI model" is an artificial intelligence model used to generate optimal recipes based on ingredient information.

[0442] A "prompt sentence" is a sentence used to input ingredient information into the generative AI model.

[0443] The "user device" is an interface device that displays information provided by the information presentation means and is operated by the user, and includes a smartphone, tablet, etc.

[0444] MODE FOR CARRYING OUT THE INVENTION

[0445] This invention is a system that efficiently manages ingredients stored in a refrigerated storage device, provides information on expiration dates and storage methods, and suggests optimal recipes using those ingredients. The system is primarily composed of a terminal, a server, and a user device.

[0446] Information acquisition by terminal

[0447] The terminal is a device placed inside the refrigerated storage unit, and its role is to obtain information about ingredients using a camera or scanner. Specifically, a camera installed inside the refrigerator periodically photographs ingredients and obtains image data. Barcode and label information is extracted from the image data to obtain basic information about the ingredients (product name, expiration date, quantity, etc.). For example, the terminal photographs a milk package and scans the barcode to obtain the information "Milk (Expiration date: 2023 / 10 / 15, quantity: 1L)."

[0448] Data processing and identification by the server

[0449] The ingredient information sent from the device is received by the server. The server analyzes this information and checks the database to identify detailed information about the ingredient (such as expiration date and storage method). For example, the server receives information about "eggs (expiration date: 2023 / 10 / 10, quantity: 6 pieces)" and refers to the database to identify the "refrigerated storage" method. The server then uses an algorithm to evaluate the ingredient's storage status and generate appropriate warnings.

[0450] Recipe Generation Method

[0451] The server also generates optimal recipes based on the identified ingredients. Specifically, it uses a generative AI model (e.g., an open-source generative AI model) to suggest recipes based on ingredient combinations. For example, a user might input a prompt such as "Please tell me a recipe that uses milk, eggs, and broccoli," and the server would suggest recipes such as "omelette," "cream broccoli soup," and "fritatta." The generated recipes are then sent from the server to the user device.

[0452] How to provide information

[0453] Users can check the provided information and recipes through their smartphones, tablets, or other user devices. For example, when a user opens the dedicated app, a list of ingredients currently in the refrigerator and expiration dates are displayed. Users can also check ingredients that are close to their expiration date and select recipes that use those ingredients.

[0454] Specific prompt examples:

[0455] "Can you tell me a recipe that uses the milk, eggs, and broccoli I have in my fridge?"

[0456] "Please suggest a dish that can be made using ingredients that are close to their expiration date."

[0457] The above is a specific embodiment for implementing the system of the present invention. This system allows users to efficiently manage food ingredients in a refrigerated storage device and reduce food waste.

[0458] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0459] Step 1: Obtaining ingredient information on the device

[0460] The terminal periodically photographs ingredients using a camera placed inside the refrigerated storage device.

[0461] Input: Image of the inside of the refrigerator

[0462] Output: Captured image data

[0463] Specific operation: The device takes a full image of the inside of the refrigerated storage device once in the morning and once in the afternoon. Specifically, the timer module in the device activates the camera at a set time to capture images inside the storage device.

[0464] Step 2: Extracting ingredient information on the device

[0465] The device analyzes the image data it acquires and extracts barcode and label information.

[0466] Input: Photographed image data

[0467] Output: Basic information about ingredients (product name, expiration date, quantity, etc.)

[0468] Specific operation: Using image recognition software (e.g., OpenCV) within the device, barcodes and labels are detected from the image. The barcode reader scans the barcode information and extracts data such as the product name, expiration date, and quantity.

[0469] Step 3: Send data from the device to the server

[0470] The terminal transmits the extracted ingredient information to the server.

[0471] Input: Basic information about ingredients (product name, expiration date, quantity, etc.)

[0472] Output: Sending ingredient information to the server

[0473] Specific operation: The communication module in the device formats the extracted data into JSON format and sends it to the server using the HTTP protocol. For example, it contains information such as "Milk (expiration date: 2023 / 10 / 15, quantity: 1L)."

[0474] Step 4: Server receives and stores information

[0475] The server receives the ingredient information sent from the terminal and stores it in a database.

[0476] Input: JSON formatted ingredient information

[0477] Output: Saved ingredient information

[0478] What happens: The server receives the HTTP request, parses the JSON data, and saves it in the database. Specifically, the server's API decodes the ingredient information and saves it in the corresponding fields in the database.

[0479] Step 5: Server analyzes ingredient information

[0480] The server references the database to identify detailed information about the ingredients (such as expiration date and storage method).

[0481] Input: Saved ingredient information

[0482] Output:Detailed information about ingredients

[0483] What happens: The server queries the database to identify the expiration date and storage method for the stored item. For example, searching for "milk" returns "Expiration date: 10 / 15 / 2023, Storage method: Refrigerate."

[0484] Step 6: Server evaluation of preservation status

[0485] The server uses an algorithm to assess the food's storage status and generate appropriate warnings.

[0486] Input:Detailed information about ingredients

[0487] Output: Preservation status evaluation results and warning messages

[0488] Specific operation: The server runs a storage condition assessment algorithm to check whether the food is stored properly. For example, if a food item that needs to be refrigerated is in the freezer, it generates a warning. A warning message is generated and sent to the user device.

[0489] Step 7: Generate the recipe

[0490] The server uses a generative AI model to generate an optimal recipe based on the identified ingredients.

[0491] Input: Identified ingredient information

[0492] Output: The generated recipe

[0493] Specific operation: The server generates a prompt based on the ingredient information and inputs it into the generative AI model. For example, the server generates a prompt such as "Please tell me a recipe using the following ingredients: milk, eggs, and broccoli." The generative AI model then suggests recipes such as "omelette" and "cream broccoli soup."

[0494] Step 8: Provide information to user devices

[0495] The server sends the generated recipe and ingredient information to the user's device, and the user checks the information in the app.

[0496] Input: Generated recipe and ingredient information

[0497] Output: Recipe and ingredient information displayed on the user's device

[0498] How it works: The server sends the generated recipe information through the API, and the user device receives it. The user can open the app and check the list of ingredients in the refrigerator, their expiration dates, and suggested recipes.

[0499] The above are the specific processing steps of the program of this system.

[0500] (Application example 1)

[0501] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0502] Managing ingredients in refrigerated storage devices requires a lot of time and effort, making it difficult for users to use food efficiently. There is also a high possibility of food waste occurring when ingredients that have passed their expiration date are discarded. Furthermore, it is difficult to link information about ingredients within the home with an external recipe database, and there is no system that can easily suggest optimal recipes. This creates the problem of not making effective use of ingredients and making it difficult to save money on household expenses.

[0503] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0504] In this invention, the server includes a recipe suggestion unit that suggests recipes using a generative AI model that generates optimal recipes based on ingredient information, and an input unit for inputting a prompt statement to the recipe suggestion unit. This allows users to easily obtain optimal cooking methods that utilize ingredients within their expiration dates based on the ingredient information in the refrigerated storage device. It also reduces food waste and promotes efficient and economical use of ingredients.

[0505] A "refrigerated storage device" is a device for storing items such as food and beverages at a constant low temperature.

[0506] "Information acquisition means" refers to a device for acquiring information about ingredients in the refrigerated storage device, such as a camera or scanner.

[0507] The "data processing means" is a device or software for analyzing the data obtained by the information acquisition means and identifying the state of the ingredients.

[0508] The "identification means" is a device or program for identifying the expiration date and appropriate storage method based on the condition of the identified food material.

[0509] The "information presentation means" is an interface for providing the identified information to the user, and includes devices such as smartphones and tablets.

[0510] The "recipe generation means" is a device or program for creating an optimal cooking recipe based on the identified ingredient information.

[0511] A "generative AI model" is an artificial intelligence model used to generate optimal recipes based on ingredient information.

[0512] A "recipe suggestion means" is a device or program that uses a generative AI model to suggest optimal recipes to users.

[0513] The "input means" is an interface for inputting a prompt sentence to the recipe suggestion means, and includes a keyboard and a touch screen.

[0514] A "prompt" is a text statement that provides specific instructions to the generative AI model, describing specific conditions or requirements for the desired recipe.

[0515] This system efficiently manages information about ingredients stored in refrigerated storage devices, provides information about expiration dates and storage methods, and even suggests optimal recipes using those ingredients. The specific configuration and processing flow of the system are described below.

[0516] 1. System Configuration

[0517] 1.1 Terminal

[0518] The terminal functions as an information acquisition means placed inside the refrigerated storage device, and acquires information about the ingredients in the refrigerated storage device using a camera or scanner.

[0519] 1.2 Server

[0520] The server is a central device that includes multiple means, specifically, a recipe suggestion means that uses a generative AI model to generate optimal recipes based on ingredient information, and an input means for inputting a prompt sentence to the recipe suggestion means.

[0521] 1.3 User Devices

[0522] The user device acts as an information presentation means, such as a smartphone or tablet, and provides an interface for displaying the identified information and suggested recipes.

[0523] 2. Program Processing

[0524] 2.1 Information acquisition

[0525] A camera installed in the terminal periodically scans and captures images of the food items in the refrigerated storage unit, and image recognition technology is used to identify these items and determine their condition through data processing means.

[0526] 2.2 Data Processing

[0527] The server compares the ingredient information sent from the terminal with the database to identify the expiration date and storage method of the specific ingredient, and provides this identified information to the user through a user interface.

[0528] 2.3 Recipe Generation

[0529] The server generates an optimal recipe using a generative AI model based on the identified ingredient information. The generated recipe can be made more specific by the user inputting a prompt sentence.

[0530] 2.4 Information presentation

[0531] Using a user device such as a smartphone or tablet, the user can check the list of ingredients currently in the refrigerator, their expiration date, storage instructions, and the generated recipe. The information displayed on the user device allows the user to use ingredients efficiently.

[0532] 3. Specific Examples

[0533] For example, suppose you have the following ingredients in your refrigerator:

[0534] Milk (Expiration date: 2023 / 10 / 15, Quantity: 1L)

[0535] Eggs (Best before: 2023 / 10 / 10, Quantity: 6)

[0536] Broccoli (Best before date: 2023 / 10 / 12, Quantity: 1 bunch)

[0537] To generate the best recipe using these ingredients, the user can enter the following prompt:

[0538] Create a recipe that meets the following conditions: You have the following ingredients in your refrigerator.

[0539] Milk (Expiration date: 2023 / 10 / 15, Quantity: 1L)

[0540] Eggs (Best before: 2023 / 10 / 10, Quantity: 6)

[0541] Broccoli (Best before date: 2023 / 10 / 12, Quantity: 1 bunch)

[0542] Prioritize ingredients that are close to their expiration date and suggest recipes that are easy to prepare.

[0543] By inputting such a prompt, the generative AI model generates recipes that make effective use of ingredients and displays them on the user's device.

[0544] Hardware and software used

[0545] Examples of the main hardware and software used in this system include:

[0546] Camera: Used to obtain information about ingredients in the refrigerated storage device.

[0547] Smartphone / Tablet: Provides the user interface.

[0548] Information acquisition method: Ingredient recognition using computer vision technology (e.g., OpenCV).

[0549] Data processing method: Processing and analyzing ingredient information (e.g., TensorFlow, Python).

[0550] Recipe suggestion method: Recipes are suggested using a generative AI model (e.g., generative AI model).

[0551] Means of information presentation: Building a user interface (e.g., React Native).

[0552] This configuration allows users to more efficiently manage ingredients in their refrigerators and easily retrieve and execute optimal recipes, thereby reducing food waste and saving money.

[0553] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0554] Step 1:

[0555] The terminal scans the inside of the refrigerated storage unit. A camera installed on the terminal periodically photographs the food inside the refrigerated storage unit and captures images. The captured images are sent to a server for data processing. The input is a video of the food inside the refrigerated storage unit, and the output is the image data required for analysis.

[0556] Step 2:

[0557] The server receives the image data. Based on the received image data, it uses computer vision technology to identify ingredients. Specific data processing involves analyzing barcodes and shapes using image recognition algorithms, and identifying ingredient information by matching it with a database. The input is the acquired image data, and the output is the analyzed ingredient information.

[0558] Step 3:

[0559] The server identifies the expiration date and storage method based on the ingredient information. It references the database to obtain information about the ingredient's expiration date and storage method. The input is the identified ingredient information, and the output is the identified expiration date and storage method data.

[0560] Step 4:

[0561] The server manages the identified ingredient information and sends it to the user device. A display interface on the user device allows the user to visually check the current ingredient status, expiration date, and storage method. The input is the identified expiration date and storage method data, and the output is the display data on the user device.

[0562] Step 5:

[0563] The server generates optimal recipes based on ingredient information. Using a generative AI model, it suggests recipes based on prompts entered by the user. The input is the identified ingredient information and prompts, and the output is the generated recipe information.

[0564] Step 6:

[0565] Users can receive more specific recipe suggestions by entering a prompt using the input means of a smartphone or tablet. For example, they can enter, "Please suggest a simple recipe using milk, eggs, and broccoli." The input is the user's prompt, and the output is a recipe suggestion from the generative AI model.

[0566] Step 7:

[0567] The server provides the generated recipe information to the user device, and the user confirms and selects it. The optimal recipe based on the expiration date is displayed on the user device. The input is the generated recipe information, and the output is the display data on the user device.

[0568] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0569] The system of this invention not only efficiently manages food ingredients in a refrigerated storage device and provides information on expiration dates and storage methods, but also has the function of recognizing the user's emotions and adjusting the way information is presented and recipe suggestions based on those emotions. The system is mainly composed of the following elements, which operate in an integrated manner:

[0570] System configuration

[0571] 1. Terminal

[0572] The terminal is a device installed inside the refrigerated storage unit and has the following functions:

[0573] Information acquisition method: Use cameras and scanners to acquire information about ingredients in the refrigerated storage unit. Use barcode scanning and image recognition technology.

[0574] Data processing means: Identify the type, quantity, expiration date, etc. of ingredients from the acquired ingredient information.

[0575] Emotion engine: Analyzes the user's facial images and voice data to recognize emotions.

[0576] 2. Server

[0577] The server receives and processes information sent from the terminal, manages the necessary data, and provides it to the user.

[0578] Identification method: Refer to the database and identify the expiration date and storage method from the ingredient information.

[0579] Recipe generation means: Generates optimal recipes based on ingredient information and suggests them to the user.

[0580] Emotion-based adjustment: The recipe suggestions and information presentation method are adjusted based on the user's emotional information.

[0581] 3. User Devices

[0582] The user device is a user interface such as a smartphone or tablet, which displays and operates the information provided by the server.

[0583] Information presentation means: Visually presents information about ingredients, expiration dates, storage methods, recipes, etc. to the user.

[0584] Example of a system

[0585] 1. The device scans the refrigerator

[0586] The device uses a camera to take pictures of the inside of the refrigerator and recognizes barcodes, labels, and shapes. For example, milk, eggs, and broccoli are detected. The device analyzes this information and sends it to a server.

[0587] 2. The server processes the information

[0588] The server receives the food ingredient information sent from the device and compares it with a database to identify information such as "Milk (best before date: 2023 / 10 / 15, quantity: 1L)," "Eggs (best before date: 2023 / 10 / 10, quantity: 6 eggs)," and "Broccoli (best before date: 2023 / 10 / 12, quantity: 1 bunch)," and also confirms the storage method.

[0589] The server also calculates and suggests optimal recipes based on the ingredient information, such as "Omelette," "Cream Broccoli Soup," and "Frittata."

[0590] 3. User Emotion Recognition and Information Provision

[0591] When a user opens the smartphone app, the device captures the user's facial image and voice via the camera and microphone, and identifies their emotion using an emotion engine. For example, if the server recognizes that the user is feeling stressed, it will use that information to present simple recipes for relaxation, along with soothing color schemes and audio.

[0592] 4. Customizing the user experience

[0593] The emotion engine continuously monitors the user's emotional state and provides the optimal recipes and information presentation methods depending on the situation. For example, it will suggest quick meals when the user is tired, and new and challenging recipes when the user has time, thereby enriching the user experience.

[0594] The above is a concrete example of how to implement the system of the present invention, which combines an emotion engine. This system further improves the efficiency of food management in refrigerated storage devices and provides personalized support according to the user's emotions.

[0595] The processing flow will be explained below.

[0596] Step 1:

[0597] The device scans the inside of the refrigerator. It uses its built-in camera to take detailed photos of the food inside the refrigerator. It then recognizes barcodes, labels, and shapes from the captured images.

[0598] Step 2:

[0599] The device uses image processing technology to extract ingredient information from the captured image. It scans barcodes to obtain information such as the ingredient name, quantity, and expiration date. Shape recognition is used to estimate the type and approximate quantity of ingredients without barcodes.

[0600] Step 3:

[0601] The device converts the extracted ingredient information into JSON format. The correctly formatted data includes fields such as ingredient name, quantity, and expiration date.

[0602] Step 4:

[0603] The device sends JSON-formatted ingredient information to the server via the network, using communication methods such as Wi-Fi and Bluetooth to send the data securely and quickly.

[0604] Step 5:

[0605] The server receives the JSON data sent from the device. The server analyzes the data and extracts information such as the ingredient name, quantity, and expiration date. The integrity of the data is then confirmed.

[0606] Step 6:

[0607] The server updates the database with the extracted information, adds new ingredient information or overwrites existing data, and checks for duplicates or incomplete data and corrects it as necessary.

[0608] Step 7:

[0609] The server identifies the expiration date and storage method of the ingredients, and checks the expiration date and optimal storage method by looking up information in the database about the storage period and appropriate storage temperature.

[0610] Step 8:

[0611] The server uses the ingredient information to generate the optimal recipe. It searches the recipe information in the database and suggests the optimal recipe based on the combination and quantity. When doing so, it gives priority to ingredients that are approaching their expiration date.

[0612] Step 9:

[0613] The server generates recipes and related information, converts them into JSON format, and sends them to the user's device. The recipes include recommended dishes, required ingredients, and cooking instructions.

[0614] Step 10:

[0615] When a user opens the smartphone app, the emotion engine activates via the camera and microphone. The device captures the user's facial image and voice, which the emotion engine analyzes to recognize the user's emotions.

[0616] Step 11:

[0617] The server receives the user's emotional state and adjusts the information presentation method and recipe suggestions accordingly. For example, if the user is feeling stressed, it will use simple recipes that are relaxing and a visually friendly color scheme.

[0618] Step 12:

[0619] The information presentation means displays the adjusted content to the user. The user browses the suggested recipes and information on ingredients and checks the details. The user selects a recipe that interests them, checks the details, and actually cooks the dish.

[0620] Through these processing steps, the system can efficiently manage ingredients in the refrigerator storage device and provide information and recipes that respond to the user's emotions.

[0621] Example 2

[0622] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0623] Conventional food management systems for refrigerated storage devices mainly focus on managing expiration dates and storage methods for ingredients, making it difficult to adjust the information content and recipe suggestions provided according to the user's emotional state. As a result, uniform information and recipes are provided without taking into account the user's current emotions and state, which creates a problem of not being able to improve the user experience.

[0624] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0625] In this invention, the server includes an information acquisition means, a data processing means for identifying the condition of ingredients based on the acquired information, an identification means for identifying expiration dates and storage methods based on the identified conditions, an information presentation means for providing the identified information to the user, a recipe generation means for generating suggested recipes using the identified ingredients, an emotion recognition means for recognizing the emotion of the user, and an emotion adjustment means for adjusting the recipe suggestions and information presentation method based on the recognized emotion. This not only enables efficient management of ingredients in the refrigerated storage device, but also enables personalized information and recipe suggestions according to the user's emotional state.

[0626] The "information acquisition means" refers to a device such as a camera or scanner for acquiring information about ingredients in the refrigerated storage device.

[0627] The "data processing means" is a combination of software and hardware for identifying the state of ingredients based on the acquired ingredient information.

[0628] The "identification means" is a device or algorithm for identifying the expiration date and storage method based on the condition of the identified food material.

[0629] "Information presentation means" refers to a user interface such as a display or smartphone app that provides the identified information to the user.

[0630] The "recipe generation means" refers to software or algorithms that generate and suggest optimal recipes using the identified ingredients.

[0631] "Emotion recognition means" refers to software or algorithms that analyze a user's facial image and voice data to recognize their emotions.

[0632] An "emotion adjustment means" is a combination of software and hardware for adjusting recipe suggestions and information presentation methods based on recognized emotions.

[0633] The system of this invention not only efficiently manages food ingredients in a refrigerated storage device and provides information on expiration dates and storage methods, but also has the function of recognizing the user's emotions and adjusting the way information is presented and recipe suggestions based on those emotions. This system is composed of the following elements, which operate in an integrated manner.

[0634] System configuration

[0635] 1. Terminal

[0636] The terminal is a device installed inside the refrigerated storage unit and has the following functions: The terminal uses a camera or scanner as an information acquisition means to acquire information about ingredients inside the refrigerated storage unit. For example, it uses barcode scanning and image recognition technology. The terminal processes the acquired ingredient information using a data processing means to identify the type, quantity, expiration date, etc. of the ingredients.

[0637] Furthermore, the device recognizes emotions by analyzing the user's facial images and voice data, using image recognition technology and voice analysis software.

[0638] 2. Server

[0639] The server receives and processes the information sent from the terminal and manages the necessary data. The server's identification means compares the received ingredient information with a database and identifies detailed ingredient information (e.g., expiration date and storage method). The identified information is provided to the user device using the information presentation means.

[0640] The server's recipe generation means generates optimal recipes based on the ingredient information and suggests them to the user. Furthermore, the emotion adjustment means adjusts the content of the recipe suggestions and the way in which the information is presented based on the user's emotion information. For example, if the user is tired, the server will suggest simple recipes that can be prepared in a short time.

[0641] 3. User Devices

[0642] The user device is a user interface such as a smartphone or tablet. The user device displays and operates information provided by the server. Information such as ingredient information, expiration dates, storage methods, and recipes are visually presented to the user through the information presentation means.

[0643] Specific examples

[0644] 1. The device scans the refrigerator

[0645] The device uses a camera to take pictures of the inside of the refrigerator and recognizes barcodes, labels, and shapes. For example, milk, eggs, and broccoli are detected. The device analyzes this information and sends it to a server.

[0646] 2. The server processes the information

[0647] The server receives the ingredient information sent from the device and compares it with the database to identify information such as "Milk (Expiration date: 2023 / 10 / 15, Amount: 1L)," "Eggs (Expiration date: 2023 / 10 / 10, Amount: 6 eggs)," and "Broccoli (Expiration date: 2023 / 10 / 12, Amount: 1 bunch)." The server then calculates and suggests the optimal recipe based on the ingredient information. For example, it might suggest "Omelette," "Cream Broccoli Soup," or "Frittata."

[0648] 3. User Emotion Recognition and Information Provision

[0649] When a user opens the smartphone app, the device captures the user's facial image and voice via the camera and microphone, and identifies their emotion using an emotion engine. For example, if the server recognizes that the user is feeling stressed, it will use that information to present simple recipes for relaxation, along with soothing color schemes and audio.

[0650] Prompt Sentence Examples

[0651] "Imagine a system that not only efficiently manages the ingredients in your refrigerator and tells you expiration dates and storage methods, but also recognizes your emotions and suggests the best recipes based on those emotions. For example, it could suggest quick meals when you're tired, and more challenging recipes when you have the time."

[0652] The system of the present invention makes food management in a refrigerated storage device more efficient and can provide personalized support that responds to the user's emotions.

[0653] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0654] Step 1:

[0655] The device obtains information about ingredients in the refrigerator. When the user opens the refrigerator door, the device's camera and scanner are automatically activated. For example, the camera takes a photo of the food in the refrigerator, and the scanner reads the barcode. The input data are images of the ingredients in the refrigerator and their barcode information. Based on this input data, the device uses image recognition technology to identify the ingredient information and format it as text data. The output is text information that includes the type, quantity, and expiration date of each ingredient.

[0656] Step 2:

[0657] The device sends the acquired ingredient information to the server. The input data is text-format ingredient information generated on the device. The device then sends this information to the server via the network. The data received by the server is text data containing detailed information about the ingredients. Specifically, the data is sent securely using a data transmission protocol (e.g., HTTP, HTTPS).

[0658] Step 3:

[0659] The server processes the ingredient information. The input data is the text-formatted ingredient information sent from the terminal. The server's identification means checks the information against a database to identify detailed information about the ingredient. For example, a database query is used to obtain detailed information such as "Milk (best before date: 2023 / 10 / 15, quantity: 1L)." As output, detailed information about the identified ingredient is generated in text format. Specifically, an SQL query is executed on the server to extract the required information from the database.

[0660] Step 4:

[0661] The server's recipe generation means generates an optimal recipe based on the ingredient information. The input data is detailed information about the identified ingredients. The server uses an algorithm to calculate an optimal recipe based on the current ingredients. For example, dish names such as "Omelette," "Cream Broccoli Soup," and "Frittata" are suggested. The output is a list of suggested recipes. Specifically, the server checks the recipe database programmatically and selects the optimal recipe.

[0662] Step 5:

[0663] A user opens a smartphone app and performs emotion recognition. The input data is the user's facial image and voice data. The device's camera and microphone capture this data and send it to the device's emotion recognition means. The device identifies the emotion using facial expression analysis software and voice analysis software. The output is analyzed emotional information. For example, it may determine that the user is "feeling stressed" based on facial expression or voice tone.

[0664] Step 6:

[0665] The server receives the emotional information and adjusts the information presentation method and recipe suggestions. The input data is the emotional information sent from the device. The server's emotion adjustment means uses this information to adjust the content and method of information provided to the user. For example, a user in a stressed state might be suggested "meals that can be made in a short time" or "easy recipes that will help them relax." The output generates a customized recipe list and visually and audibly adjusted information presentation. In concrete terms, the server compares the information with a recipe database and selects the recipe that best suits the user's emotional state, while also adjusting the color scheme of the information presentation screen and the audio guidance.

[0666] Step 7:

[0667] The user's experience is executed. Input data is a customized recipe list and adjusted information presentation sent from the server. The user receives this information through a smartphone app, selects a dish, and begins cooking. Specific operations include displaying the recipe on the user's smartphone, allowing them to cook according to their mood and situation that day.

[0668] (Application example 2)

[0669] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0670] Conventional refrigerated storage devices make it difficult to efficiently manage expiration dates and storage conditions of ingredients, leading to users often wasting expired ingredients. While recipe suggestions based on ingredient information were offered, they were unable to tailor suggestions to individual users' emotions or circumstances, resulting in low satisfaction. Furthermore, the time required to manually order missing ingredients meant that these systems lacked convenience.

[0671] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0672] In this invention, the server includes an emotion engine that recognizes emotions, an adjustment means that adjusts the content of suggestions and the method of information presentation based on the user's emotion information, and a delivery ordering means that automatically orders ingredients that are in short supply, thereby enabling personalized recipe suggestions and automatic delivery ordering according to the user's emotions.

[0673] The "information acquisition means" is a device that is placed inside the refrigerated storage device and acquires food ingredient information using barcode scanning or image recognition technology.

[0674] The "data processing means" is a device that analyzes the acquired ingredient information and identifies the condition of the ingredients, such as type, quantity, and expiration date.

[0675] The "identification means" is a device that identifies the expiration date and storage method based on the state of the identified food material.

[0676] The "information presentation means" is a device for providing the identified information to the user.

[0677] The "recipe generation means" is a device that generates and proposes optimal recipes based on the identified ingredient information.

[0678] An "emotion engine" is a device that analyzes a user's facial images and voice data to recognize their emotions.

[0679] The "adjustment means" is a device that adjusts the content of recipe suggestions and the method of presenting information based on the user's emotional information.

[0680] The "delivery ordering means" is a device that automatically places an additional order when ingredients are in short supply.

[0681] The system of this invention not only efficiently manages food ingredients in a refrigerated storage device and provides information on expiration dates and storage methods, but also has the function of recognizing the user's emotions and adjusting the way information is presented and recipe suggestions based on those emotions. The main elements of the system are a terminal installed in the refrigerated storage device, a server that processes data, and a user device.

[0682] Terminal

[0683] The terminal is located within the refrigerated storage unit and has the following functions:

[0684] 1. Information acquisition method: Using a smartphone camera, food items in the refrigerator are scanned. Image recognition technology and barcode scanning are used to acquire food information (name, quantity, expiration date).

[0685] 2. Data processing means: Analyze the acquired food ingredient information and identify the type, quantity, expiration date, etc.

[0686] server

[0687] The server receives the information sent by the terminal and performs the following functions:

[0688] 1. Identification method: Based on the acquired data, the expiration date and storage method are checked against a database to identify the product.

[0689] 2. Recipe generation means: Generates optimal recipes using the identified ingredients and suggests them to the user.

[0690] 3. Emotion Engine: Analyzes the user's facial image and voice data to recognize emotions. This is done using EmotionRecognizer (software for emotion recognition).

[0691] 4. Adjustment method: Based on the information obtained from the emotion engine, recipe suggestions and the way information is presented are adjusted.

[0692] 5. Delivery ordering method: Automatically order missing ingredients from a food delivery service.

[0693] User Device

[0694] The user device (smartphone or tablet) is used to display and operate the information provided by the server.

[0695] 1. Information presentation means: Visually present information about ingredients, expiration dates, storage methods, recipes, etc. to the user.

[0696] Specific examples

[0697] When a user scans ingredients in a refrigerator, information such as "Milk (Best before date: 2023 / 10 / 15, Quantity: 1L)," "Eggs (Best before date: 2023 / 10 / 10, Quantity: 6 eggs)," and "Broccoli (Best before date: 2023 / 10 / 12, Quantity: 1 bunch)" is obtained. This information is sent from the device to the server and registered in a database along with storage instructions. Based on this information, the server generates recipes such as "Omelette" and "Cream Broccoli Soup" and suggests them to the user.

[0698] When a user opens the smartphone app and captures their facial images and voice via the camera and microphone, the emotion engine identifies the user's emotions. For example, if the app detects that the user is feeling stressed, it will present information such as simple, relaxing recipes with soothing colors and sounds. If the suggested recipe is missing ingredients, the delivery ordering method will automatically order the missing ingredients.

[0699] Prompt Sentence Examples

[0700] Ingredients needed: 1L milk, 6 eggs, 1 bunch of broccoli, 200g cheese

[0701] Best before date: Milk 2023 / 10 / 15, Eggs 2023 / 10 / 10, Broccoli 2023 / 10 / 12

[0702] User Emotion: Tired

[0703] Generate suggested recipes.

[0704] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0705] Step 1:

[0706] The terminal scans ingredients in the refrigerator. Using the smartphone camera, it obtains ingredient information using barcode scanning and image recognition technology. The data obtained includes the ingredient's name, quantity, and expiration date. For example, the camera scans the barcode on a piece of milk and obtains the information "Milk (quantity: 1L, expiration date: 2023 / 10 / 15)."

[0707] Step 2:

[0708] The device analyzes the acquired ingredient information using a data processing means and sends the identified ingredient information to the server. The input is ingredient information (name, quantity, expiration date), and the output is a data packet containing this information. For example, a data packet such as "Eggs (quantity: 6, expiration date: 2023 / 10 / 10)" is sent to the server.

[0709] Step 3:

[0710] The server receives the data packet and uses an identification means to compare it with the database. The data calculation performed here is to identify the optimal storage method based on the ingredient name, quantity, and expiration date. For example, for "broccoli (quantity: 1 bunch, expiration date: 2023 / 10 / 12)", the storage method is identified as "refrigerated storage."

[0711] Step 4:

[0712] The server sends the identified storage information and expiration date information to the user device, and the information presentation means presents it to the user. The input is the identified storage method and expiration date information, and the output is the display on the user device. For example, the information "Broccoli (storage method: refrigerated, expiration date: 2023 / 10 / 12)" is displayed on a smartphone.

[0713] Step 5:

[0714] The server's recipe generation means generates an optimal recipe based on the identified ingredient information. The input is ingredient information and storage method, and the output is the generated recipe. For example, it generates recipes for "omelette" or "cream broccoli soup" based on "milk (amount: 1L), eggs (amount: 6), broccoli (amount: 1 bunch)".

[0715] Step 6:

[0716] A user opens a smartphone app and sends facial images and voice data to the emotion engine via the camera and microphone. The input is the user's facial image and voice data, and the output is the identified emotion. For example, the emotion engine may identify the user as "tired."

[0717] Step 7:

[0718] The server's adjustment means adjusts the recipe suggestions and information presentation method based on the identified emotional information. The input is the user's emotional information and ingredient information, and the output is the adjusted recipe suggestions and information presentation method. For example, for a tired user, information is presented that includes "easy recipes" and "relaxing color schemes."

[0719] Step 8:

[0720] The user checks the proposed recipe, and if there are missing ingredients, the delivery ordering method automatically orders the missing ingredients from a food delivery service. The input is a list of the missing ingredients, and the output is an order confirmation to the delivery service. For example, "200g of cheese" and "3 tomatoes" can be automatically ordered.

[0721] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0722] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0723] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0724] [Third embodiment]

[0725] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0726] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0727] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0728] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0729] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0730] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0731] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0732] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0733] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0734] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0735] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0736] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0737] The system of this invention efficiently manages ingredients stored in a refrigerated storage device, provides information on expiration dates and storage methods, and suggests optimal recipes using those ingredients. The system is mainly configured as follows, and each process is executed as follows.

[0738] System configuration

[0739] 1. Terminal

[0740] The terminal is a device installed inside the refrigerated storage device and has the following main functions:

[0741] Information acquisition method: Information on ingredients in the refrigerated storage device is acquired using a camera or scanner.

[0742] Data processing method: Based on the acquired ingredient information, AI technology is used to recognize ingredients and extract the necessary information.

[0743] 2. Server

[0744] The server receives and processes information sent from the terminal, manages the necessary data, and provides it to the user.

[0745] Identification method: Refer to the database and identify the expiration date and storage method from the ingredient information.

[0746] Recipe generation means: Based on parameterized ingredient information, an optimal recipe is generated and proposed to the user.

[0747] 3. User Devices

[0748] The user device is a user interface such as a smartphone or tablet, which displays and operates the information provided by the server.

[0749] Information presentation means: Visually presents information about ingredients, expiration dates, storage methods, recipes, etc. to the user.

[0750] Example of a system

[0751] 1. The device scans the refrigerator

[0752] The device uses a camera to photograph the inside of the refrigerator and recognize barcodes, labels, and shapes. During this process, ingredients such as milk, eggs, and broccoli are detected. The device analyzes and extracts information about these ingredients and sends it to the server.

[0753] 2. The server processes the information

[0754] The server receives the food ingredient information sent from the device and compares it with information in the database. For example, specific information such as "Milk (Best before date: 2023 / 10 / 15, quantity: 1L)," "Eggs (Best before date: 2023 / 10 / 10, quantity: 6 eggs)," and "Broccoli (Best before date: 2023 / 10 / 12, quantity: 1 bunch)" can be identified. It also checks whether these ingredients have been stored properly based on the expiration date.

[0755] The server then analyzes this information and calculates the optimal recipe, suggesting recipes such as "Omelette," "Cream Broccoli Soup," and "Frittata" using these ingredients.

[0756] 3. Provide information to users

[0757] The user opens a dedicated smartphone app and checks the information provided by the server. For example, a list of ingredients currently in the refrigerator and their expiration dates are displayed on the screen. The user can also check which ingredients are close to their expiration date and select a recipe that uses those ingredients. For example, the user can select a recipe for "cream broccoli soup" and proceed with the cooking process while checking the ingredients and cooking instructions in detail.

[0758] The above is a specific embodiment for implementing the system of the present invention. This system improves the efficiency of food management in a refrigerated storage device, reducing food waste and saving household money.

[0759] The processing flow will be explained below.

[0760] Step 1:

[0761] The device scans the inside of the refrigerator. It uses its built-in camera to take multiple images in maximum detail, capturing food labels, barcodes, shapes, and other features.

[0762] Step 2:

[0763] The device uses image processing algorithms to extract ingredient information from the captured image. It recognizes barcodes and labels to obtain the name, quantity, expiration date, and other information of the corresponding ingredients. Shape recognition is used to estimate the type and approximate quantity of ingredients without barcodes.

[0764] Step 3:

[0765] The device converts the extracted ingredient information into JSON format, including data fields such as ingredient name, quantity, and expiration date to ensure the information is formatted correctly.

[0766] Step 4:

[0767] The device sends the ingredient information in JSON format to the server via the network, using communication methods such as Wi-Fi or Bluetooth to send the data quickly and securely.

[0768] Step 5:

[0769] The server receives the JSON data sent from the device. The server parses the data and extracts each field (ingredients, quantity, expiration date, etc.). This analysis verifies the integrity of the data.

[0770] Step 6:

[0771] The server updates the database with the extracted information, adding new ingredient information or overwriting existing data, checking for duplicates or incomplete data and correcting it as necessary.

[0772] Step 7:

[0773] The server identifies the expiration date and storage method of the ingredients. It looks up information about each ingredient in the database (e.g., shelf life, appropriate storage temperature, etc.) to check the expiration date and best storage method of the ingredients.

[0774] Step 8:

[0775] The server uses the ingredient information to calculate and select the optimal recipe. It references recipe information in the database and suggests recipes that are best suited to the ingredients based on combinations and quantities. It also selects recipes that prioritize ingredients that are approaching their expiration date.

[0776] Step 9:

[0777] The server converts the selected recipes and related information into JSON format and sends it back to the user's device, including a list of recommended recipes, the ingredients needed, and cooking instructions.

[0778] Step 10:

[0779] The user opens the smartphone app and checks the information sent from the server. A list of ingredients in the refrigerator, their expiration dates, and storage methods are displayed on the screen, along with suggested recipes.

[0780] Step 11:

[0781] The user selects a suggested recipe and reviews its details. The user then checks the ingredients list and cooking instructions for each recipe and gets to cooking.

[0782] Through the above steps, the system of the present invention can efficiently manage food ingredients stored in a refrigerated storage device and provide optimal information and suggestions to the user.

[0783] Example 1

[0784] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0785] There is a need for a system that can efficiently manage food ingredients stored in refrigerated storage devices, provide information on expiration dates and storage methods, and even suggest optimal recipes using those ingredients. However, conventional systems have incomplete acquisition and processing of ingredient information, making it difficult to efficiently provide useful information to users. For this reason, a new system is needed to reduce food waste and save household budgets.

[0786] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0787] In this invention, the server includes the information acquisition means, data processing means for identifying the condition of ingredients based on the acquired information, identification means for identifying expiration dates and storage methods based on the identified conditions, information presentation means for providing the identified information to users, recipe generation means for generating suggested recipes using the identified ingredients, and means for transmitting the generated recipes to a user device. This makes it possible to accurately and efficiently acquire and process information about ingredients in the refrigerated storage device and provide users with useful information and optimal recipes.

[0788] The "information acquisition means" is a device that is placed inside the refrigerated storage device and acquires information about ingredients using a camera or scanner.

[0789] The "data processing means" is a device that identifies the state of ingredients based on the acquired ingredient information and extracts necessary information.

[0790] The "identification means" is a device that identifies the expiration date and storage method based on the state of the identified food material.

[0791] The "information presentation means" is a device that provides the user with information about identified ingredients, and is primarily displayed on the user's device.

[0792] A "recipe generator" is a device that generates suggested recipes using identified ingredients.

[0793] A "generative AI model" is an artificial intelligence model used to generate optimal recipes based on ingredient information.

[0794] A "prompt sentence" is a sentence used to input ingredient information into the generative AI model.

[0795] The "user device" is an interface device that displays information provided by the information presentation means and is operated by the user, and includes a smartphone, tablet, etc.

[0796] MODE FOR CARRYING OUT THE INVENTION

[0797] This invention is a system that efficiently manages ingredients stored in a refrigerated storage device, provides information on expiration dates and storage methods, and suggests optimal recipes using those ingredients. The system is primarily composed of a terminal, a server, and a user device.

[0798] Information acquisition by terminal

[0799] The terminal is a device placed inside the refrigerated storage unit, and its role is to obtain information about ingredients using a camera or scanner. Specifically, a camera installed inside the refrigerator periodically photographs ingredients and obtains image data. Barcode and label information is extracted from the image data to obtain basic information about the ingredients (product name, expiration date, quantity, etc.). For example, the terminal photographs a milk package and scans the barcode to obtain the information "Milk (Expiration date: 2023 / 10 / 15, quantity: 1L)."

[0800] Data processing and identification by the server

[0801] The ingredient information sent from the device is received by the server. The server analyzes this information and checks the database to identify detailed information about the ingredient (such as expiration date and storage method). For example, the server receives information about "eggs (expiration date: 2023 / 10 / 10, quantity: 6 pieces)" and refers to the database to identify the "refrigerated storage" method. The server then uses an algorithm to evaluate the ingredient's storage status and generate appropriate warnings.

[0802] Recipe Generation Method

[0803] The server also generates optimal recipes based on the identified ingredients. Specifically, it uses a generative AI model (e.g., an open-source generative AI model) to suggest recipes based on ingredient combinations. For example, a user might input a prompt such as "Please tell me a recipe that uses milk, eggs, and broccoli," and the server would suggest recipes such as "omelette," "cream broccoli soup," and "fritatta." The generated recipes are then sent from the server to the user device.

[0804] How to provide information

[0805] Users can check the provided information and recipes through their smartphones, tablets, or other user devices. For example, when a user opens the dedicated app, a list of ingredients currently in the refrigerator and expiration dates are displayed. Users can also check ingredients that are close to their expiration date and select recipes that use those ingredients.

[0806] Specific prompt examples:

[0807] "Can you tell me a recipe that uses the milk, eggs, and broccoli I have in my fridge?"

[0808] "Please suggest a dish that can be made using ingredients that are close to their expiration date."

[0809] The above is a specific embodiment for implementing the system of the present invention. This system allows users to efficiently manage food ingredients in a refrigerated storage device and reduce food waste.

[0810] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0811] Step 1: Obtaining ingredient information on the device

[0812] The terminal periodically photographs ingredients using a camera placed inside the refrigerated storage device.

[0813] Input: Image of the inside of the refrigerator

[0814] Output: Captured image data

[0815] Specific operation: The device takes a full image of the inside of the refrigerated storage device once in the morning and once in the afternoon. Specifically, the timer module in the device activates the camera at a set time to capture images inside the storage device.

[0816] Step 2: Extracting ingredient information on the device

[0817] The device analyzes the image data it acquires and extracts barcode and label information.

[0818] Input: Photographed image data

[0819] Output: Basic information about ingredients (product name, expiration date, quantity, etc.)

[0820] Specific operation: Using image recognition software (e.g., OpenCV) within the device, barcodes and labels are detected from the image. The barcode reader scans the barcode information and extracts data such as the product name, expiration date, and quantity.

[0821] Step 3: Send data from the device to the server

[0822] The terminal transmits the extracted ingredient information to the server.

[0823] Input: Basic information about ingredients (product name, expiration date, quantity, etc.)

[0824] Output: Sending ingredient information to the server

[0825] Specific operation: The communication module in the device formats the extracted data into JSON format and sends it to the server using the HTTP protocol. For example, it contains information such as "Milk (expiration date: 2023 / 10 / 15, quantity: 1L)."

[0826] Step 4: Server receives and stores information

[0827] The server receives the ingredient information sent from the terminal and stores it in a database.

[0828] Input: JSON formatted ingredient information

[0829] Output: Saved ingredient information

[0830] What happens: The server receives the HTTP request, parses the JSON data, and saves it in the database. Specifically, the server's API decodes the ingredient information and saves it in the corresponding fields in the database.

[0831] Step 5: Server analyzes ingredient information

[0832] The server references the database to identify detailed information about the ingredients (such as expiration date and storage method).

[0833] Input: Saved ingredient information

[0834] Output:Detailed information about ingredients

[0835] What happens: The server queries the database to identify the expiration date and storage method for the stored item. For example, searching for "milk" returns "Expiration date: 10 / 15 / 2023, Storage method: Refrigerate."

[0836] Step 6: Server evaluation of preservation status

[0837] The server uses an algorithm to assess the food's storage status and generate appropriate warnings.

[0838] Input:Detailed information about ingredients

[0839] Output: Preservation status evaluation results and warning messages

[0840] Specific operation: The server runs a storage condition assessment algorithm to check whether the food is stored properly. For example, if a food item that needs to be refrigerated is in the freezer, it generates a warning. A warning message is generated and sent to the user device.

[0841] Step 7: Generate the recipe

[0842] The server uses a generative AI model to generate an optimal recipe based on the identified ingredients.

[0843] Input: Identified ingredient information

[0844] Output: The generated recipe

[0845] Specific operation: The server generates a prompt based on the ingredient information and inputs it into the generative AI model. For example, the server generates a prompt such as "Please tell me a recipe using the following ingredients: milk, eggs, and broccoli." The generative AI model then suggests recipes such as "omelette" and "cream broccoli soup."

[0846] Step 8: Provide information to user devices

[0847] The server sends the generated recipe and ingredient information to the user's device, and the user checks the information in the app.

[0848] Input: Generated recipe and ingredient information

[0849] Output: Recipe and ingredient information displayed on the user's device

[0850] How it works: The server sends the generated recipe information through the API, and the user device receives it. The user can open the app and check the list of ingredients in the refrigerator, their expiration dates, and suggested recipes.

[0851] The above are the specific processing steps of the program of this system.

[0852] (Application example 1)

[0853] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0854] Managing ingredients in refrigerated storage devices requires a lot of time and effort, making it difficult for users to use food efficiently. There is also a high possibility of food waste occurring when ingredients that have passed their expiration date are discarded. Furthermore, it is difficult to link information about ingredients within the home with an external recipe database, and there is no system that can easily suggest optimal recipes. This creates the problem of not making effective use of ingredients and making it difficult to save money on household expenses.

[0855] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0856] In this invention, the server includes a recipe suggestion unit that suggests recipes using a generative AI model that generates optimal recipes based on ingredient information, and an input unit for inputting a prompt statement to the recipe suggestion unit. This allows users to easily obtain optimal cooking methods that utilize ingredients within their expiration dates based on the ingredient information in the refrigerated storage device. It also reduces food waste and promotes efficient and economical use of ingredients.

[0857] A "refrigerated storage device" is a device for storing items such as food and beverages at a constant low temperature.

[0858] "Information acquisition means" refers to a device for acquiring information about ingredients in the refrigerated storage device, such as a camera or scanner.

[0859] The "data processing means" is a device or software for analyzing the data obtained by the information acquisition means and identifying the state of the ingredients.

[0860] The "identification means" is a device or program for identifying the expiration date and appropriate storage method based on the condition of the identified food material.

[0861] The "information presentation means" is an interface for providing the identified information to the user, and includes devices such as smartphones and tablets.

[0862] The "recipe generation means" is a device or program for creating an optimal cooking recipe based on the identified ingredient information.

[0863] A "generative AI model" is an artificial intelligence model used to generate optimal recipes based on ingredient information.

[0864] A "recipe suggestion means" is a device or program that uses a generative AI model to suggest optimal recipes to users.

[0865] The "input means" is an interface for inputting a prompt sentence to the recipe suggestion means, and includes a keyboard and a touch screen.

[0866] A "prompt" is a text statement that provides specific instructions to the generative AI model, describing specific conditions or requirements for the desired recipe.

[0867] This system efficiently manages information about ingredients stored in refrigerated storage devices, provides information about expiration dates and storage methods, and even suggests optimal recipes using those ingredients. The specific configuration and processing flow of the system are described below.

[0868] 1. System Configuration

[0869] 1.1 Terminal

[0870] The terminal functions as an information acquisition means placed inside the refrigerated storage device, and acquires information about the ingredients in the refrigerated storage device using a camera or scanner.

[0871] 1.2 Server

[0872] The server is a central device that includes multiple means, specifically, a recipe suggestion means that uses a generative AI model to generate optimal recipes based on ingredient information, and an input means for inputting a prompt sentence to the recipe suggestion means.

[0873] 1.3 User Devices

[0874] The user device acts as an information presentation means, such as a smartphone or tablet, and provides an interface for displaying the identified information and suggested recipes.

[0875] 2. Program Processing

[0876] 2.1 Information acquisition

[0877] A camera installed in the terminal periodically scans and captures images of the food items in the refrigerated storage unit, and image recognition technology is used to identify these items and determine their condition through data processing means.

[0878] 2.2 Data Processing

[0879] The server compares the ingredient information sent from the terminal with the database to identify the expiration date and storage method of the specific ingredient, and provides this identified information to the user through a user interface.

[0880] 2.3 Recipe Generation

[0881] The server generates an optimal recipe using a generative AI model based on the identified ingredient information. The generated recipe can be made more specific by the user inputting a prompt sentence.

[0882] 2.4 Information presentation

[0883] Using a user device such as a smartphone or tablet, the user can check the list of ingredients currently in the refrigerator, their expiration date, storage instructions, and the generated recipe. The information displayed on the user device allows the user to use ingredients efficiently.

[0884] 3. Specific Examples

[0885] For example, suppose you have the following ingredients in your refrigerator:

[0886] Milk (Expiration date: 2023 / 10 / 15, Quantity: 1L)

[0887] Eggs (Best before: 2023 / 10 / 10, Quantity: 6)

[0888] Broccoli (Best before date: 2023 / 10 / 12, Quantity: 1 bunch)

[0889] To generate the best recipe using these ingredients, the user can enter the following prompt:

[0890] Create a recipe that meets the following conditions: You have the following ingredients in your refrigerator.

[0891] Milk (Expiration date: 2023 / 10 / 15, Quantity: 1L)

[0892] Eggs (Best before: 2023 / 10 / 10, Quantity: 6)

[0893] Broccoli (Best before date: 2023 / 10 / 12, Quantity: 1 bunch)

[0894] Prioritize ingredients that are close to their expiration date and suggest recipes that are easy to prepare.

[0895] By inputting such a prompt, the generative AI model generates recipes that make effective use of ingredients and displays them on the user's device.

[0896] Hardware and software used

[0897] Examples of the main hardware and software used in this system include:

[0898] Camera: Used to obtain information about ingredients in the refrigerated storage device.

[0899] Smartphone / Tablet: Provides the user interface.

[0900] Information acquisition method: Ingredient recognition using computer vision technology (e.g., OpenCV).

[0901] Data processing method: Processing and analyzing ingredient information (e.g., TensorFlow, Python).

[0902] Recipe suggestion method: Recipes are suggested using a generative AI model (e.g., generative AI model).

[0903] Means of information presentation: Building a user interface (e.g., React Native).

[0904] This configuration allows users to more efficiently manage ingredients in their refrigerators and easily retrieve and execute optimal recipes, thereby reducing food waste and saving money.

[0905] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0906] Step 1:

[0907] The terminal scans the inside of the refrigerated storage unit. A camera installed on the terminal periodically photographs the food inside the refrigerated storage unit and captures images. The captured images are sent to a server for data processing. The input is a video of the food inside the refrigerated storage unit, and the output is the image data required for analysis.

[0908] Step 2:

[0909] The server receives the image data. Based on the received image data, it uses computer vision technology to identify ingredients. Specific data processing involves analyzing barcodes and shapes using image recognition algorithms, and identifying ingredient information by matching it with a database. The input is the acquired image data, and the output is the analyzed ingredient information.

[0910] Step 3:

[0911] The server identifies the expiration date and storage method based on the ingredient information. It references the database to obtain information about the ingredient's expiration date and storage method. The input is the identified ingredient information, and the output is the identified expiration date and storage method data.

[0912] Step 4:

[0913] The server manages the identified ingredient information and sends it to the user device. A display interface on the user device allows the user to visually check the current ingredient status, expiration date, and storage method. The input is the identified expiration date and storage method data, and the output is the display data on the user device.

[0914] Step 5:

[0915] The server generates optimal recipes based on ingredient information. Using a generative AI model, it suggests recipes based on prompts entered by the user. The input is the identified ingredient information and prompts, and the output is the generated recipe information.

[0916] Step 6:

[0917] Users can receive more specific recipe suggestions by entering a prompt using the input means of a smartphone or tablet. For example, they can enter, "Please suggest a simple recipe using milk, eggs, and broccoli." The input is the user's prompt, and the output is a recipe suggestion from the generative AI model.

[0918] Step 7:

[0919] The server provides the generated recipe information to the user device, and the user confirms and selects it. The optimal recipe based on the expiration date is displayed on the user device. The input is the generated recipe information, and the output is the display data on the user device.

[0920] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0921] The system of this invention not only efficiently manages food ingredients in a refrigerated storage device and provides information on expiration dates and storage methods, but also has the function of recognizing the user's emotions and adjusting the way information is presented and recipe suggestions based on those emotions. The system is mainly composed of the following elements, which operate in an integrated manner:

[0922] System configuration

[0923] 1. Terminal

[0924] The terminal is a device installed inside the refrigerated storage unit and has the following functions:

[0925] Information acquisition method: Use cameras and scanners to acquire information about ingredients in the refrigerated storage unit. Use barcode scanning and image recognition technology.

[0926] Data processing means: Identify the type, quantity, expiration date, etc. of ingredients from the acquired ingredient information.

[0927] Emotion engine: Analyzes the user's facial images and voice data to recognize emotions.

[0928] 2. Server

[0929] The server receives and processes information sent from the terminal, manages the necessary data, and provides it to the user.

[0930] Identification method: Refer to the database and identify the expiration date and storage method from the ingredient information.

[0931] Recipe generation means: Generates optimal recipes based on ingredient information and suggests them to the user.

[0932] Emotion-based adjustment: The recipe suggestions and information presentation method are adjusted based on the user's emotional information.

[0933] 3. User Devices

[0934] The user device is a user interface such as a smartphone or tablet, which displays and operates the information provided by the server.

[0935] Information presentation means: Visually presents information about ingredients, expiration dates, storage methods, recipes, etc. to the user.

[0936] Example of a system

[0937] 1. The device scans the refrigerator

[0938] The device uses a camera to take pictures of the inside of the refrigerator and recognizes barcodes, labels, and shapes. For example, milk, eggs, and broccoli are detected. The device analyzes this information and sends it to a server.

[0939] 2. The server processes the information

[0940] The server receives the food ingredient information sent from the device and compares it with a database to identify information such as "Milk (best before date: 2023 / 10 / 15, quantity: 1L)," "Eggs (best before date: 2023 / 10 / 10, quantity: 6 eggs)," and "Broccoli (best before date: 2023 / 10 / 12, quantity: 1 bunch)," and also confirms the storage method.

[0941] The server also calculates and suggests optimal recipes based on the ingredient information, such as "Omelette," "Cream Broccoli Soup," and "Frittata."

[0942] 3. User Emotion Recognition and Information Provision

[0943] When a user opens the smartphone app, the device captures the user's facial image and voice via the camera and microphone, and identifies their emotion using an emotion engine. For example, if the server recognizes that the user is feeling stressed, it will use that information to present simple recipes for relaxation, along with soothing color schemes and audio.

[0944] 4. Customizing the user experience

[0945] The emotion engine continuously monitors the user's emotional state and provides the optimal recipes and information presentation methods depending on the situation. For example, it will suggest quick meals when the user is tired, and new and challenging recipes when the user has time, thereby enriching the user experience.

[0946] The above is a concrete example of how to implement the system of the present invention, which combines an emotion engine. This system further improves the efficiency of food management in refrigerated storage devices and provides personalized support according to the user's emotions.

[0947] The processing flow will be explained below.

[0948] Step 1:

[0949] The device scans the inside of the refrigerator. It uses its built-in camera to take detailed photos of the food inside the refrigerator. It then recognizes barcodes, labels, and shapes from the captured images.

[0950] Step 2:

[0951] The device uses image processing technology to extract ingredient information from the captured image. It scans barcodes to obtain information such as the ingredient name, quantity, and expiration date. Shape recognition is used to estimate the type and approximate quantity of ingredients without barcodes.

[0952] Step 3:

[0953] The device converts the extracted ingredient information into JSON format. The correctly formatted data includes fields such as ingredient name, quantity, and expiration date.

[0954] Step 4:

[0955] The device sends JSON-formatted ingredient information to the server via the network, using communication methods such as Wi-Fi and Bluetooth to send the data securely and quickly.

[0956] Step 5:

[0957] The server receives the JSON data sent from the device. The server analyzes the data and extracts information such as the ingredient name, quantity, and expiration date. The integrity of the data is then confirmed.

[0958] Step 6:

[0959] The server updates the database with the extracted information, adds new ingredient information or overwrites existing data, and checks for duplicates or incomplete data and corrects it as necessary.

[0960] Step 7:

[0961] The server identifies the expiration date and storage method of the ingredients, and checks the expiration date and optimal storage method by looking up information in the database about the storage period and appropriate storage temperature.

[0962] Step 8:

[0963] The server uses the ingredient information to generate the optimal recipe. It searches the recipe information in the database and suggests the optimal recipe based on the combination and quantity. When doing so, it gives priority to ingredients that are approaching their expiration date.

[0964] Step 9:

[0965] The server generates recipes and related information, converts them into JSON format, and sends them to the user's device. The recipes include recommended dishes, required ingredients, and cooking instructions.

[0966] Step 10:

[0967] When a user opens the smartphone app, the emotion engine activates via the camera and microphone. The device captures the user's facial image and voice, which the emotion engine analyzes to recognize the user's emotions.

[0968] Step 11:

[0969] The server receives the user's emotional state and adjusts the information presentation method and recipe suggestions accordingly. For example, if the user is feeling stressed, it will use simple recipes that are relaxing and a visually friendly color scheme.

[0970] Step 12:

[0971] The information presentation means displays the adjusted content to the user. The user browses the suggested recipes and information on ingredients and checks the details. The user selects a recipe that interests them, checks the details, and actually cooks the dish.

[0972] Through these processing steps, the system can efficiently manage ingredients in the refrigerator storage device and provide information and recipes that respond to the user's emotions.

[0973] Example 2

[0974] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0975] Conventional food management systems for refrigerated storage devices mainly focus on managing expiration dates and storage methods for ingredients, making it difficult to adjust the information content and recipe suggestions provided according to the user's emotional state. As a result, uniform information and recipes are provided without taking into account the user's current emotions and state, which creates a problem of not being able to improve the user experience.

[0976] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0977] In this invention, the server includes an information acquisition means, a data processing means for identifying the condition of ingredients based on the acquired information, an identification means for identifying expiration dates and storage methods based on the identified conditions, an information presentation means for providing the identified information to the user, a recipe generation means for generating suggested recipes using the identified ingredients, an emotion recognition means for recognizing the emotion of the user, and an emotion adjustment means for adjusting the recipe suggestions and information presentation method based on the recognized emotion. This not only enables efficient management of ingredients in the refrigerated storage device, but also enables personalized information and recipe suggestions according to the user's emotional state.

[0978] The "information acquisition means" refers to a device such as a camera or scanner for acquiring information about ingredients in the refrigerated storage device.

[0979] The "data processing means" is a combination of software and hardware for identifying the state of ingredients based on the acquired ingredient information.

[0980] The "identification means" is a device or algorithm for identifying the expiration date and storage method based on the condition of the identified food material.

[0981] "Information presentation means" refers to a user interface such as a display or smartphone app that provides the identified information to the user.

[0982] The "recipe generation means" refers to software or algorithms that generate and suggest optimal recipes using the identified ingredients.

[0983] "Emotion recognition means" refers to software or algorithms that analyze a user's facial image and voice data to recognize their emotions.

[0984] An "emotion adjustment means" is a combination of software and hardware for adjusting recipe suggestions and information presentation methods based on recognized emotions.

[0985] The system of this invention not only efficiently manages food ingredients in a refrigerated storage device and provides information on expiration dates and storage methods, but also has the function of recognizing the user's emotions and adjusting the way information is presented and recipe suggestions based on those emotions. This system is composed of the following elements, which operate in an integrated manner.

[0986] System configuration

[0987] 1. Terminal

[0988] The terminal is a device installed inside the refrigerated storage unit and has the following functions: The terminal uses a camera or scanner as an information acquisition means to acquire information about ingredients inside the refrigerated storage unit. For example, it uses barcode scanning and image recognition technology. The terminal processes the acquired ingredient information using a data processing means to identify the type, quantity, expiration date, etc. of the ingredients.

[0989] Furthermore, the device recognizes emotions by analyzing the user's facial images and voice data, using image recognition technology and voice analysis software.

[0990] 2. Server

[0991] The server receives and processes the information sent from the terminal and manages the necessary data. The server's identification means compares the received ingredient information with a database and identifies detailed ingredient information (e.g., expiration date and storage method). The identified information is provided to the user device using the information presentation means.

[0992] The server's recipe generation means generates optimal recipes based on the ingredient information and suggests them to the user. Furthermore, the emotion adjustment means adjusts the content of the recipe suggestions and the way in which the information is presented based on the user's emotion information. For example, if the user is tired, the server will suggest simple recipes that can be prepared in a short time.

[0993] 3. User Devices

[0994] The user device is a user interface such as a smartphone or tablet. The user device displays and operates information provided by the server. Information such as ingredient information, expiration dates, storage methods, and recipes are visually presented to the user through the information presentation means.

[0995] Specific examples

[0996] 1. The device scans the refrigerator

[0997] The device uses a camera to take pictures of the inside of the refrigerator and recognizes barcodes, labels, and shapes. For example, milk, eggs, and broccoli are detected. The device analyzes this information and sends it to a server.

[0998] 2. The server processes the information

[0999] The server receives the ingredient information sent from the device and compares it with the database to identify information such as "Milk (Expiration date: 2023 / 10 / 15, Amount: 1L)," "Eggs (Expiration date: 2023 / 10 / 10, Amount: 6 eggs)," and "Broccoli (Expiration date: 2023 / 10 / 12, Amount: 1 bunch)." The server then calculates and suggests the optimal recipe based on the ingredient information. For example, it might suggest "Omelette," "Cream Broccoli Soup," or "Frittata."

[1000] 3. User Emotion Recognition and Information Provision

[1001] When a user opens the smartphone app, the device captures the user's facial image and voice via the camera and microphone, and identifies their emotion using an emotion engine. For example, if the server recognizes that the user is feeling stressed, it will use that information to present simple recipes for relaxation, along with soothing color schemes and audio.

[1002] Prompt Sentence Examples

[1003] "Imagine a system that not only efficiently manages the ingredients in your refrigerator and tells you expiration dates and storage methods, but also recognizes your emotions and suggests the best recipes based on those emotions. For example, it could suggest quick meals when you're tired, and more challenging recipes when you have the time."

[1004] The system of the present invention makes food management in a refrigerated storage device more efficient and can provide personalized support that responds to the user's emotions.

[1005] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1006] Step 1:

[1007] The device obtains information about ingredients in the refrigerator. When the user opens the refrigerator door, the device's camera and scanner are automatically activated. For example, the camera takes a photo of the food in the refrigerator, and the scanner reads the barcode. The input data are images of the ingredients in the refrigerator and their barcode information. Based on this input data, the device uses image recognition technology to identify the ingredient information and format it as text data. The output is text information that includes the type, quantity, and expiration date of each ingredient.

[1008] Step 2:

[1009] The device sends the acquired ingredient information to the server. The input data is text-format ingredient information generated on the device. The device then sends this information to the server via the network. The data received by the server is text data containing detailed information about the ingredients. Specifically, the data is sent securely using a data transmission protocol (e.g., HTTP, HTTPS).

[1010] Step 3:

[1011] The server processes the ingredient information. The input data is the text-formatted ingredient information sent from the terminal. The server's identification means checks the information against a database to identify detailed information about the ingredient. For example, a database query is used to obtain detailed information such as "Milk (best before date: 2023 / 10 / 15, quantity: 1L)." As output, detailed information about the identified ingredient is generated in text format. Specifically, an SQL query is executed on the server to extract the required information from the database.

[1012] Step 4:

[1013] The server's recipe generation means generates an optimal recipe based on the ingredient information. The input data is detailed information about the identified ingredients. The server uses an algorithm to calculate an optimal recipe based on the current ingredients. For example, dish names such as "Omelette," "Cream Broccoli Soup," and "Frittata" are suggested. The output is a list of suggested recipes. Specifically, the server checks the recipe database programmatically and selects the optimal recipe.

[1014] Step 5:

[1015] A user opens a smartphone app and performs emotion recognition. The input data is the user's facial image and voice data. The device's camera and microphone capture this data and send it to the device's emotion recognition means. The device identifies the emotion using facial expression analysis software and voice analysis software. The output is analyzed emotional information. For example, it may determine that the user is "feeling stressed" based on facial expression or voice tone.

[1016] Step 6:

[1017] The server receives the emotional information and adjusts the information presentation method and recipe suggestions. The input data is the emotional information sent from the device. The server's emotion adjustment means uses this information to adjust the content and method of information provided to the user. For example, a user in a stressed state might be suggested "meals that can be made in a short time" or "easy recipes that will help them relax." The output generates a customized recipe list and visually and audibly adjusted information presentation. In concrete terms, the server compares the information with a recipe database and selects the recipe that best suits the user's emotional state, while also adjusting the color scheme of the information presentation screen and the audio guidance.

[1018] Step 7:

[1019] The user's experience is executed. Input data is a customized recipe list and adjusted information presentation sent from the server. The user receives this information through a smartphone app, selects a dish, and begins cooking. Specific operations include displaying the recipe on the user's smartphone, allowing them to cook according to their mood and situation that day.

[1020] (Application example 2)

[1021] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1022] Conventional refrigerated storage devices make it difficult to efficiently manage expiration dates and storage conditions of ingredients, leading to users often wasting expired ingredients. While recipe suggestions based on ingredient information were offered, they were unable to tailor suggestions to individual users' emotions or circumstances, resulting in low satisfaction. Furthermore, the time required to manually order missing ingredients meant that these systems lacked convenience.

[1023] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1024] In this invention, the server includes an emotion engine that recognizes emotions, an adjustment means that adjusts the content of suggestions and the method of information presentation based on the user's emotion information, and a delivery ordering means that automatically orders ingredients that are in short supply, thereby enabling personalized recipe suggestions and automatic delivery ordering according to the user's emotions.

[1025] The "information acquisition means" is a device that is placed inside the refrigerated storage device and acquires food ingredient information using barcode scanning or image recognition technology.

[1026] The "data processing means" is a device that analyzes the acquired ingredient information and identifies the condition of the ingredients, such as type, quantity, and expiration date.

[1027] The "identification means" is a device that identifies the expiration date and storage method based on the state of the identified food material.

[1028] The "information presentation means" is a device for providing the identified information to the user.

[1029] The "recipe generation means" is a device that generates and proposes optimal recipes based on the identified ingredient information.

[1030] An "emotion engine" is a device that analyzes a user's facial images and voice data to recognize their emotions.

[1031] The "adjustment means" is a device that adjusts the content of recipe suggestions and the method of presenting information based on the user's emotional information.

[1032] The "delivery ordering means" is a device that automatically places an additional order when ingredients are in short supply.

[1033] The system of this invention not only efficiently manages food ingredients in a refrigerated storage device and provides information on expiration dates and storage methods, but also has the function of recognizing the user's emotions and adjusting the way information is presented and recipe suggestions based on those emotions. The main elements of the system are a terminal installed in the refrigerated storage device, a server that processes data, and a user device.

[1034] Terminal

[1035] The terminal is located within the refrigerated storage unit and has the following functions:

[1036] 1. Information acquisition method: Using a smartphone camera, food items in the refrigerator are scanned. Image recognition technology and barcode scanning are used to acquire food information (name, quantity, expiration date).

[1037] 2. Data processing means: Analyze the acquired food ingredient information and identify the type, quantity, expiration date, etc.

[1038] server

[1039] The server receives the information sent by the terminal and performs the following functions:

[1040] 1. Identification method: Based on the acquired data, the expiration date and storage method are checked against a database to identify the product.

[1041] 2. Recipe generation means: Generates optimal recipes using the identified ingredients and suggests them to the user.

[1042] 3. Emotion Engine: Analyzes the user's facial image and voice data to recognize emotions. This is done using EmotionRecognizer (software for emotion recognition).

[1043] 4. Adjustment method: Based on the information obtained from the emotion engine, recipe suggestions and the way information is presented are adjusted.

[1044] 5. Delivery ordering method: Automatically order missing ingredients from a food delivery service.

[1045] User Device

[1046] The user device (smartphone or tablet) is used to display and operate the information provided by the server.

[1047] 1. Information presentation means: Visually present information about ingredients, expiration dates, storage methods, recipes, etc. to the user.

[1048] Specific examples

[1049] When a user scans ingredients in a refrigerator, information such as "Milk (Best before date: 2023 / 10 / 15, Quantity: 1L)," "Eggs (Best before date: 2023 / 10 / 10, Quantity: 6 eggs)," and "Broccoli (Best before date: 2023 / 10 / 12, Quantity: 1 bunch)" is obtained. This information is sent from the device to the server and registered in a database along with storage instructions. Based on this information, the server generates recipes such as "Omelette" and "Cream Broccoli Soup" and suggests them to the user.

[1050] When a user opens the smartphone app and captures their facial images and voice via the camera and microphone, the emotion engine identifies the user's emotions. For example, if the app detects that the user is feeling stressed, it will present information such as simple, relaxing recipes with soothing colors and sounds. If the suggested recipe is missing ingredients, the delivery ordering method will automatically order the missing ingredients.

[1051] Prompt Sentence Examples

[1052] Ingredients needed: 1L milk, 6 eggs, 1 bunch of broccoli, 200g cheese

[1053] Best before date: Milk 2023 / 10 / 15, Eggs 2023 / 10 / 10, Broccoli 2023 / 10 / 12

[1054] User Emotion: Tired

[1055] Generate suggested recipes.

[1056] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1057] Step 1:

[1058] The terminal scans ingredients in the refrigerator. Using the smartphone camera, it obtains ingredient information using barcode scanning and image recognition technology. The data obtained includes the ingredient's name, quantity, and expiration date. For example, the camera scans the barcode on a piece of milk and obtains the information "Milk (quantity: 1L, expiration date: 2023 / 10 / 15)."

[1059] Step 2:

[1060] The device analyzes the acquired ingredient information using a data processing means and sends the identified ingredient information to the server. The input is ingredient information (name, quantity, expiration date), and the output is a data packet containing this information. For example, a data packet such as "Eggs (quantity: 6, expiration date: 2023 / 10 / 10)" is sent to the server.

[1061] Step 3:

[1062] The server receives the data packet and uses an identification means to compare it with the database. The data calculation performed here is to identify the optimal storage method based on the ingredient name, quantity, and expiration date. For example, for "broccoli (quantity: 1 bunch, expiration date: 2023 / 10 / 12)", the storage method is identified as "refrigerated storage."

[1063] Step 4:

[1064] The server sends the identified storage information and expiration date information to the user device, and the information presentation means presents it to the user. The input is the identified storage method and expiration date information, and the output is the display on the user device. For example, the information "Broccoli (storage method: refrigerated, expiration date: 2023 / 10 / 12)" is displayed on a smartphone.

[1065] Step 5:

[1066] The server's recipe generation means generates an optimal recipe based on the identified ingredient information. The input is ingredient information and storage method, and the output is the generated recipe. For example, it generates recipes for "omelette" or "cream broccoli soup" based on "milk (amount: 1L), eggs (amount: 6), broccoli (amount: 1 bunch)".

[1067] Step 6:

[1068] A user opens a smartphone app and sends facial images and voice data to the emotion engine via the camera and microphone. The input is the user's facial image and voice data, and the output is the identified emotion. For example, the emotion engine may identify the user as "tired."

[1069] Step 7:

[1070] The server's adjustment means adjusts the recipe suggestions and information presentation method based on the identified emotional information. The input is the user's emotional information and ingredient information, and the output is the adjusted recipe suggestions and information presentation method. For example, for a tired user, information is presented that includes "easy recipes" and "relaxing color schemes."

[1071] Step 8:

[1072] The user checks the proposed recipe, and if there are missing ingredients, the delivery ordering method automatically orders the missing ingredients from a food delivery service. The input is a list of the missing ingredients, and the output is an order confirmation to the delivery service. For example, "200g of cheese" and "3 tomatoes" can be automatically ordered.

[1073] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1074] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1075] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1076] [Fourth embodiment]

[1077] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1078] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1079] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1080] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1081] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1082] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1083] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1084] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1085] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1086] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1087] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1088] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1089] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1090] The system of this invention efficiently manages ingredients stored in a refrigerated storage device, provides information on expiration dates and storage methods, and suggests optimal recipes using those ingredients. The system is mainly configured as follows, and each process is executed as follows.

[1091] System configuration

[1092] 1. Terminal

[1093] The terminal is a device installed inside the refrigerated storage device and has the following main functions:

[1094] Information acquisition method: Information on ingredients in the refrigerated storage device is acquired using a camera or scanner.

[1095] Data processing method: Based on the acquired ingredient information, AI technology is used to recognize ingredients and extract the necessary information.

[1096] 2. Server

[1097] The server receives and processes information sent from the terminal, manages the necessary data, and provides it to the user.

[1098] Identification method: Refer to the database and identify the expiration date and storage method from the ingredient information.

[1099] Recipe generation means: Based on parameterized ingredient information, an optimal recipe is generated and proposed to the user.

[1100] 3. User Devices

[1101] The user device is a user interface such as a smartphone or tablet, which displays and operates the information provided by the server.

[1102] Information presentation means: Visually presents information about ingredients, expiration dates, storage methods, recipes, etc. to the user.

[1103] Example of a system

[1104] 1. The device scans the refrigerator

[1105] The device uses a camera to photograph the inside of the refrigerator and recognize barcodes, labels, and shapes. During this process, ingredients such as milk, eggs, and broccoli are detected. The device analyzes and extracts information about these ingredients and sends it to the server.

[1106] 2. The server processes the information

[1107] The server receives the food ingredient information sent from the device and compares it with information in the database. For example, specific information such as "Milk (Best before date: 2023 / 10 / 15, quantity: 1L)," "Eggs (Best before date: 2023 / 10 / 10, quantity: 6 eggs)," and "Broccoli (Best before date: 2023 / 10 / 12, quantity: 1 bunch)" can be identified. It also checks whether these ingredients have been stored properly based on the expiration date.

[1108] The server then analyzes this information and calculates the optimal recipe, suggesting recipes such as "Omelette," "Cream Broccoli Soup," and "Frittata" using these ingredients.

[1109] 3. Provide information to users

[1110] The user opens a dedicated smartphone app and checks the information provided by the server. For example, a list of ingredients currently in the refrigerator and their expiration dates are displayed on the screen. The user can also check which ingredients are close to their expiration date and select a recipe that uses those ingredients. For example, the user can select a recipe for "cream broccoli soup" and proceed with the cooking process while checking the ingredients and cooking instructions in detail.

[1111] The above is a specific embodiment for implementing the system of the present invention. This system improves the efficiency of food management in a refrigerated storage device, reducing food waste and saving household money.

[1112] The processing flow will be explained below.

[1113] Step 1:

[1114] The device scans the inside of the refrigerator. It uses its built-in camera to take multiple images in maximum detail, capturing food labels, barcodes, shapes, and other features.

[1115] Step 2:

[1116] The device uses image processing algorithms to extract ingredient information from the captured image. It recognizes barcodes and labels to obtain the name, quantity, expiration date, and other information of the corresponding ingredients. Shape recognition is used to estimate the type and approximate quantity of ingredients without barcodes.

[1117] Step 3:

[1118] The device converts the extracted ingredient information into JSON format, including data fields such as ingredient name, quantity, and expiration date to ensure the information is formatted correctly.

[1119] Step 4:

[1120] The device sends the ingredient information in JSON format to the server via the network, using communication methods such as Wi-Fi or Bluetooth to send the data quickly and securely.

[1121] Step 5:

[1122] The server receives the JSON data sent from the device. The server parses the data and extracts each field (ingredients, quantity, expiration date, etc.). This analysis verifies the integrity of the data.

[1123] Step 6:

[1124] The server updates the database with the extracted information, adding new ingredient information or overwriting existing data, checking for duplicates or incomplete data and correcting it as necessary.

[1125] Step 7:

[1126] The server identifies the expiration date and storage method of the ingredients. It looks up information about each ingredient in the database (e.g., shelf life, appropriate storage temperature, etc.) to check the expiration date and best storage method of the ingredients.

[1127] Step 8:

[1128] The server uses the ingredient information to calculate and select the optimal recipe. It references recipe information in the database and suggests recipes that are best suited to the ingredients based on combinations and quantities. It also selects recipes that prioritize ingredients that are approaching their expiration date.

[1129] Step 9:

[1130] The server converts the selected recipes and related information into JSON format and sends it back to the user's device, including a list of recommended recipes, the ingredients needed, and cooking instructions.

[1131] Step 10:

[1132] The user opens the smartphone app and checks the information sent from the server. A list of ingredients in the refrigerator, their expiration dates, and storage methods are displayed on the screen, along with suggested recipes.

[1133] Step 11:

[1134] The user selects a suggested recipe and reviews its details. The user then checks the ingredients list and cooking instructions for each recipe and gets to cooking.

[1135] Through the above steps, the system of the present invention can efficiently manage food ingredients stored in a refrigerated storage device and provide optimal information and suggestions to the user.

[1136] Example 1

[1137] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1138] There is a need for a system that can efficiently manage food ingredients stored in refrigerated storage devices, provide information on expiration dates and storage methods, and even suggest optimal recipes using those ingredients. However, conventional systems have incomplete acquisition and processing of ingredient information, making it difficult to efficiently provide useful information to users. For this reason, a new system is needed to reduce food waste and save household budgets.

[1139] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1140] In this invention, the server includes the information acquisition means, data processing means for identifying the condition of ingredients based on the acquired information, identification means for identifying expiration dates and storage methods based on the identified conditions, information presentation means for providing the identified information to users, recipe generation means for generating suggested recipes using the identified ingredients, and means for transmitting the generated recipes to a user device. This makes it possible to accurately and efficiently acquire and process information about ingredients in the refrigerated storage device and provide users with useful information and optimal recipes.

[1141] The "information acquisition means" is a device that is placed inside the refrigerated storage device and acquires information about ingredients using a camera or scanner.

[1142] The "data processing means" is a device that identifies the state of ingredients based on the acquired ingredient information and extracts necessary information.

[1143] The "identification means" is a device that identifies the expiration date and storage method based on the state of the identified food material.

[1144] The "information presentation means" is a device that provides the user with information about identified ingredients, and is primarily displayed on the user's device.

[1145] A "recipe generator" is a device that generates suggested recipes using identified ingredients.

[1146] A "generative AI model" is an artificial intelligence model used to generate optimal recipes based on ingredient information.

[1147] A "prompt sentence" is a sentence used to input ingredient information into the generative AI model.

[1148] The "user device" is an interface device that displays information provided by the information presentation means and is operated by the user, and includes a smartphone, tablet, etc.

[1149] MODE FOR CARRYING OUT THE INVENTION

[1150] This invention is a system that efficiently manages ingredients stored in a refrigerated storage device, provides information on expiration dates and storage methods, and suggests optimal recipes using those ingredients. The system is primarily composed of a terminal, a server, and a user device.

[1151] Information acquisition by terminal

[1152] The terminal is a device placed inside the refrigerated storage unit, and its role is to obtain information about ingredients using a camera or scanner. Specifically, a camera installed inside the refrigerator periodically photographs ingredients and obtains image data. Barcode and label information is extracted from the image data to obtain basic information about the ingredients (product name, expiration date, quantity, etc.). For example, the terminal photographs a milk package and scans the barcode to obtain the information "Milk (Expiration date: 2023 / 10 / 15, quantity: 1L)."

[1153] Data processing and identification by the server

[1154] The ingredient information sent from the device is received by the server. The server analyzes this information and checks the database to identify detailed information about the ingredient (such as expiration date and storage method). For example, the server receives information about "eggs (expiration date: 2023 / 10 / 10, quantity: 6 pieces)" and refers to the database to identify the "refrigerated storage" method. The server then uses an algorithm to evaluate the ingredient's storage status and generate appropriate warnings.

[1155] Recipe Generation Method

[1156] The server also generates optimal recipes based on the identified ingredients. Specifically, it uses a generative AI model (e.g., an open-source generative AI model) to suggest recipes based on ingredient combinations. For example, a user might input a prompt such as "Please tell me a recipe that uses milk, eggs, and broccoli," and the server would suggest recipes such as "omelette," "cream broccoli soup," and "fritatta." The generated recipes are then sent from the server to the user device.

[1157] How to provide information

[1158] Users can check the provided information and recipes through their smartphones, tablets, or other user devices. For example, when a user opens the dedicated app, a list of ingredients currently in the refrigerator and expiration dates are displayed. Users can also check ingredients that are close to their expiration date and select recipes that use those ingredients.

[1159] Specific prompt examples:

[1160] "Can you tell me a recipe that uses the milk, eggs, and broccoli I have in my fridge?"

[1161] "Please suggest a dish that can be made using ingredients that are close to their expiration date."

[1162] The above is a specific embodiment for implementing the system of the present invention. This system allows users to efficiently manage food ingredients in a refrigerated storage device and reduce food waste.

[1163] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1164] Step 1: Obtaining ingredient information on the device

[1165] The terminal periodically photographs ingredients using a camera placed inside the refrigerated storage device.

[1166] Input: Image of the inside of the refrigerator

[1167] Output: Captured image data

[1168] Specific operation: The device takes a full image of the inside of the refrigerated storage device once in the morning and once in the afternoon. Specifically, the timer module in the device activates the camera at a set time to capture images inside the storage device.

[1169] Step 2: Extracting ingredient information on the device

[1170] The device analyzes the image data it acquires and extracts barcode and label information.

[1171] Input: Photographed image data

[1172] Output: Basic information about ingredients (product name, expiration date, quantity, etc.)

[1173] Specific operation: Using image recognition software (e.g., OpenCV) within the device, barcodes and labels are detected from the image. The barcode reader scans the barcode information and extracts data such as the product name, expiration date, and quantity.

[1174] Step 3: Send data from the device to the server

[1175] The terminal transmits the extracted ingredient information to the server.

[1176] Input: Basic information about ingredients (product name, expiration date, quantity, etc.)

[1177] Output: Sending ingredient information to the server

[1178] Specific operation: The communication module in the device formats the extracted data into JSON format and sends it to the server using the HTTP protocol. For example, it contains information such as "Milk (expiration date: 2023 / 10 / 15, quantity: 1L)."

[1179] Step 4: Server receives and stores information

[1180] The server receives the ingredient information sent from the terminal and stores it in a database.

[1181] Input: JSON formatted ingredient information

[1182] Output: Saved ingredient information

[1183] What happens: The server receives the HTTP request, parses the JSON data, and saves it in the database. Specifically, the server's API decodes the ingredient information and saves it in the corresponding fields in the database.

[1184] Step 5: Server analyzes ingredient information

[1185] The server references the database to identify detailed information about the ingredients (such as expiration date and storage method).

[1186] Input: Saved ingredient information

[1187] Output:Detailed information about ingredients

[1188] What happens: The server queries the database to identify the expiration date and storage method for the stored item. For example, searching for "milk" returns "Expiration date: 10 / 15 / 2023, Storage method: Refrigerate."

[1189] Step 6: Server evaluation of preservation status

[1190] The server uses an algorithm to assess the food's storage status and generate appropriate warnings.

[1191] Input:Detailed information about ingredients

[1192] Output: Preservation status evaluation results and warning messages

[1193] Specific operation: The server runs a storage condition assessment algorithm to check whether the food is stored properly. For example, if a food item that needs to be refrigerated is in the freezer, it generates a warning. A warning message is generated and sent to the user device.

[1194] Step 7: Generate the recipe

[1195] The server uses a generative AI model to generate an optimal recipe based on the identified ingredients.

[1196] Input: Identified ingredient information

[1197] Output: The generated recipe

[1198] Specific operation: The server generates a prompt based on the ingredient information and inputs it into the generative AI model. For example, the server generates a prompt such as "Please tell me a recipe using the following ingredients: milk, eggs, and broccoli." The generative AI model then suggests recipes such as "omelette" and "cream broccoli soup."

[1199] Step 8: Provide information to user devices

[1200] The server sends the generated recipe and ingredient information to the user's device, and the user checks the information in the app.

[1201] Input: Generated recipe and ingredient information

[1202] Output: Recipe and ingredient information displayed on the user's device

[1203] How it works: The server sends the generated recipe information through the API, and the user device receives it. The user can open the app and check the list of ingredients in the refrigerator, their expiration dates, and suggested recipes.

[1204] The above are the specific processing steps of the program of this system.

[1205] (Application example 1)

[1206] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1207] Managing ingredients in refrigerated storage devices requires a lot of time and effort, making it difficult for users to use food efficiently. There is also a high possibility of food waste occurring when ingredients that have passed their expiration date are discarded. Furthermore, it is difficult to link information about ingredients within the home with an external recipe database, and there is no system that can easily suggest optimal recipes. This creates the problem of not making effective use of ingredients and making it difficult to save money on household expenses.

[1208] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1209] In this invention, the server includes a recipe suggestion unit that suggests recipes using a generative AI model that generates optimal recipes based on ingredient information, and an input unit for inputting a prompt statement to the recipe suggestion unit. This allows users to easily obtain optimal cooking methods that utilize ingredients within their expiration dates based on the ingredient information in the refrigerated storage device. It also reduces food waste and promotes efficient and economical use of ingredients.

[1210] A "refrigerated storage device" is a device for storing items such as food and beverages at a constant low temperature.

[1211] "Information acquisition means" refers to a device for acquiring information about ingredients in the refrigerated storage device, such as a camera or scanner.

[1212] The "data processing means" is a device or software for analyzing the data obtained by the information acquisition means and identifying the state of the ingredients.

[1213] The "identification means" is a device or program for identifying the expiration date and appropriate storage method based on the condition of the identified food material.

[1214] The "information presentation means" is an interface for providing the identified information to the user, and includes devices such as smartphones and tablets.

[1215] The "recipe generation means" is a device or program for creating an optimal cooking recipe based on the identified ingredient information.

[1216] A "generative AI model" is an artificial intelligence model used to generate optimal recipes based on ingredient information.

[1217] A "recipe suggestion means" is a device or program that uses a generative AI model to suggest optimal recipes to users.

[1218] The "input means" is an interface for inputting a prompt sentence to the recipe suggestion means, and includes a keyboard and a touch screen.

[1219] A "prompt" is a text statement that provides specific instructions to the generative AI model, describing specific conditions or requirements for the desired recipe.

[1220] This system efficiently manages information about ingredients stored in refrigerated storage devices, provides information about expiration dates and storage methods, and even suggests optimal recipes using those ingredients. The specific configuration and processing flow of the system are described below.

[1221] 1. System Configuration

[1222] 1.1 Terminal

[1223] The terminal functions as an information acquisition means placed inside the refrigerated storage device, and acquires information about the ingredients in the refrigerated storage device using a camera or scanner.

[1224] 1.2 Server

[1225] The server is a central device that includes multiple means, specifically, a recipe suggestion means that uses a generative AI model to generate optimal recipes based on ingredient information, and an input means for inputting a prompt sentence to the recipe suggestion means.

[1226] 1.3 User Devices

[1227] The user device acts as an information presentation means, such as a smartphone or tablet, and provides an interface for displaying the identified information and suggested recipes.

[1228] 2. Program Processing

[1229] 2.1 Information acquisition

[1230] A camera installed in the terminal periodically scans and captures images of the food items in the refrigerated storage unit, and image recognition technology is used to identify these items and determine their condition through data processing means.

[1231] 2.2 Data Processing

[1232] The server compares the ingredient information sent from the terminal with the database to identify the expiration date and storage method of the specific ingredient, and provides this identified information to the user through a user interface.

[1233] 2.3 Recipe Generation

[1234] The server generates an optimal recipe using a generative AI model based on the identified ingredient information. The generated recipe can be made more specific by the user inputting a prompt sentence.

[1235] 2.4 Information presentation

[1236] Using a user device such as a smartphone or tablet, the user can check the list of ingredients currently in the refrigerator, their expiration date, storage instructions, and the generated recipe. The information displayed on the user device allows the user to use ingredients efficiently.

[1237] 3. Specific Examples

[1238] For example, suppose you have the following ingredients in your refrigerator:

[1239] Milk (Expiration date: 2023 / 10 / 15, Quantity: 1L)

[1240] Eggs (Best before: 2023 / 10 / 10, Quantity: 6)

[1241] Broccoli (Best before date: 2023 / 10 / 12, Quantity: 1 bunch)

[1242] To generate the best recipe using these ingredients, the user can enter the following prompt:

[1243] Create a recipe that meets the following conditions: You have the following ingredients in your refrigerator.

[1244] Milk (Expiration date: 2023 / 10 / 15, Quantity: 1L)

[1245] Eggs (Best before: 2023 / 10 / 10, Quantity: 6)

[1246] Broccoli (Best before date: 2023 / 10 / 12, Quantity: 1 bunch)

[1247] Prioritize ingredients that are close to their expiration date and suggest recipes that are easy to prepare.

[1248] By inputting such a prompt, the generative AI model generates recipes that make effective use of ingredients and displays them on the user's device.

[1249] Hardware and software used

[1250] Examples of the main hardware and software used in this system include:

[1251] Camera: Used to obtain information about ingredients in the refrigerated storage device.

[1252] Smartphone / Tablet: Provides the user interface.

[1253] Information acquisition method: Ingredient recognition using computer vision technology (e.g., OpenCV).

[1254] Data processing method: Processing and analyzing ingredient information (e.g., TensorFlow, Python).

[1255] Recipe suggestion method: Recipes are suggested using a generative AI model (e.g., generative AI model).

[1256] Means of information presentation: Building a user interface (e.g., React Native).

[1257] This configuration allows users to more efficiently manage ingredients in their refrigerators and easily retrieve and execute optimal recipes, thereby reducing food waste and saving money.

[1258] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1259] Step 1:

[1260] The terminal scans the inside of the refrigerated storage unit. A camera installed on the terminal periodically photographs the food inside the refrigerated storage unit and captures images. The captured images are sent to a server for data processing. The input is a video of the food inside the refrigerated storage unit, and the output is the image data required for analysis.

[1261] Step 2:

[1262] The server receives the image data. Based on the received image data, it uses computer vision technology to identify ingredients. Specific data processing involves analyzing barcodes and shapes using image recognition algorithms, and identifying ingredient information by matching it with a database. The input is the acquired image data, and the output is the analyzed ingredient information.

[1263] Step 3:

[1264] The server identifies the expiration date and storage method based on the ingredient information. It references the database to obtain information about the ingredient's expiration date and storage method. The input is the identified ingredient information, and the output is the identified expiration date and storage method data.

[1265] Step 4:

[1266] The server manages the identified ingredient information and sends it to the user device. A display interface on the user device allows the user to visually check the current ingredient status, expiration date, and storage method. The input is the identified expiration date and storage method data, and the output is the display data on the user device.

[1267] Step 5:

[1268] The server generates optimal recipes based on ingredient information. Using a generative AI model, it suggests recipes based on prompts entered by the user. The input is the identified ingredient information and prompts, and the output is the generated recipe information.

[1269] Step 6:

[1270] Users can receive more specific recipe suggestions by entering a prompt using the input means of a smartphone or tablet. For example, they can enter, "Please suggest a simple recipe using milk, eggs, and broccoli." The input is the user's prompt, and the output is a recipe suggestion from the generative AI model.

[1271] Step 7:

[1272] The server provides the generated recipe information to the user device, and the user confirms and selects it. The optimal recipe based on the expiration date is displayed on the user device. The input is the generated recipe information, and the output is the display data on the user device.

[1273] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1274] The system of this invention not only efficiently manages food ingredients in a refrigerated storage device and provides information on expiration dates and storage methods, but also has the function of recognizing the user's emotions and adjusting the way information is presented and recipe suggestions based on those emotions. The system is mainly composed of the following elements, which operate in an integrated manner:

[1275] System configuration

[1276] 1. Terminal

[1277] The terminal is a device installed inside the refrigerated storage unit and has the following functions:

[1278] Information acquisition method: Use cameras and scanners to acquire information about ingredients in the refrigerated storage unit. Use barcode scanning and image recognition technology.

[1279] Data processing means: Identify the type, quantity, expiration date, etc. of ingredients from the acquired ingredient information.

[1280] Emotion engine: Analyzes the user's facial images and voice data to recognize emotions.

[1281] 2. Server

[1282] The server receives and processes information sent from the terminal, manages the necessary data, and provides it to the user.

[1283] Identification method: Refer to the database and identify the expiration date and storage method from the ingredient information.

[1284] Recipe generation means: Generates optimal recipes based on ingredient information and suggests them to the user.

[1285] Emotion-based adjustment: The recipe suggestions and information presentation method are adjusted based on the user's emotional information.

[1286] 3. User Devices

[1287] The user device is a user interface such as a smartphone or tablet, which displays and operates the information provided by the server.

[1288] Information presentation means: Visually presents information about ingredients, expiration dates, storage methods, recipes, etc. to the user.

[1289] Example of a system

[1290] 1. The device scans the refrigerator

[1291] The device uses a camera to take pictures of the inside of the refrigerator and recognizes barcodes, labels, and shapes. For example, milk, eggs, and broccoli are detected. The device analyzes this information and sends it to a server.

[1292] 2. The server processes the information

[1293] The server receives the food ingredient information sent from the device and compares it with a database to identify information such as "Milk (best before date: 2023 / 10 / 15, quantity: 1L)," "Eggs (best before date: 2023 / 10 / 10, quantity: 6 eggs)," and "Broccoli (best before date: 2023 / 10 / 12, quantity: 1 bunch)," and also confirms the storage method.

[1294] The server also calculates and suggests optimal recipes based on the ingredient information, such as "Omelette," "Cream Broccoli Soup," and "Frittata."

[1295] 3. User Emotion Recognition and Information Provision

[1296] When a user opens the smartphone app, the device captures the user's facial image and voice via the camera and microphone, and identifies their emotion using an emotion engine. For example, if the server recognizes that the user is feeling stressed, it will use that information to present simple recipes for relaxation, along with soothing color schemes and audio.

[1297] 4. Customizing the user experience

[1298] The emotion engine continuously monitors the user's emotional state and provides the optimal recipes and information presentation methods depending on the situation. For example, it will suggest quick meals when the user is tired, and new and challenging recipes when the user has time, thereby enriching the user experience.

[1299] The above is a concrete example of how to implement the system of the present invention, which combines an emotion engine. This system further improves the efficiency of food management in refrigerated storage devices and provides personalized support according to the user's emotions.

[1300] The processing flow will be explained below.

[1301] Step 1:

[1302] The device scans the inside of the refrigerator. It uses its built-in camera to take detailed photos of the food inside the refrigerator. It then recognizes barcodes, labels, and shapes from the captured images.

[1303] Step 2:

[1304] The device uses image processing technology to extract ingredient information from the captured image. It scans barcodes to obtain information such as the ingredient name, quantity, and expiration date. Shape recognition is used to estimate the type and approximate quantity of ingredients without barcodes.

[1305] Step 3:

[1306] The device converts the extracted ingredient information into JSON format. The correctly formatted data includes fields such as ingredient name, quantity, and expiration date.

[1307] Step 4:

[1308] The device sends JSON-formatted ingredient information to the server via the network, using communication methods such as Wi-Fi and Bluetooth to send the data securely and quickly.

[1309] Step 5:

[1310] The server receives the JSON data sent from the device. The server analyzes the data and extracts information such as the ingredient name, quantity, and expiration date. The integrity of the data is then confirmed.

[1311] Step 6:

[1312] The server updates the database with the extracted information, adds new ingredient information or overwrites existing data, and checks for duplicates or incomplete data and corrects it as necessary.

[1313] Step 7:

[1314] The server identifies the expiration date and storage method of the ingredients, and checks the expiration date and optimal storage method by looking up information in the database about the storage period and appropriate storage temperature.

[1315] Step 8:

[1316] The server uses the ingredient information to generate the optimal recipe. It searches the recipe information in the database and suggests the optimal recipe based on the combination and quantity. When doing so, it gives priority to ingredients that are approaching their expiration date.

[1317] Step 9:

[1318] The server generates recipes and related information, converts them into JSON format, and sends them to the user's device. The recipes include recommended dishes, required ingredients, and cooking instructions.

[1319] Step 10:

[1320] When a user opens the smartphone app, the emotion engine activates via the camera and microphone. The device captures the user's facial image and voice, which the emotion engine analyzes to recognize the user's emotions.

[1321] Step 11:

[1322] The server receives the user's emotional state and adjusts the information presentation method and recipe suggestions accordingly. For example, if the user is feeling stressed, it will use simple recipes that are relaxing and a visually friendly color scheme.

[1323] Step 12:

[1324] The information presentation means displays the adjusted content to the user. The user browses the suggested recipes and information on ingredients and checks the details. The user selects a recipe that interests them, checks the details, and actually cooks the dish.

[1325] Through these processing steps, the system can efficiently manage ingredients in the refrigerator storage device and provide information and recipes that respond to the user's emotions.

[1326] Example 2

[1327] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1328] Conventional food management systems for refrigerated storage devices mainly focus on managing expiration dates and storage methods for ingredients, making it difficult to adjust the information content and recipe suggestions provided according to the user's emotional state. As a result, uniform information and recipes are provided without taking into account the user's current emotions and state, which creates a problem of not being able to improve the user experience.

[1329] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1330] In this invention, the server includes an information acquisition means, a data processing means for identifying the condition of ingredients based on the acquired information, an identification means for identifying expiration dates and storage methods based on the identified conditions, an information presentation means for providing the identified information to the user, a recipe generation means for generating suggested recipes using the identified ingredients, an emotion recognition means for recognizing the emotion of the user, and an emotion adjustment means for adjusting the recipe suggestions and information presentation method based on the recognized emotion. This not only enables efficient management of ingredients in the refrigerated storage device, but also enables personalized information and recipe suggestions according to the user's emotional state.

[1331] The "information acquisition means" refers to a device such as a camera or scanner for acquiring information about ingredients in the refrigerated storage device.

[1332] The "data processing means" is a combination of software and hardware for identifying the state of ingredients based on the acquired ingredient information.

[1333] The "identification means" is a device or algorithm for identifying the expiration date and storage method based on the condition of the identified food material.

[1334] "Information presentation means" refers to a user interface such as a display or smartphone app that provides the identified information to the user.

[1335] The "recipe generation means" refers to software or algorithms that generate and suggest optimal recipes using the identified ingredients.

[1336] "Emotion recognition means" refers to software or algorithms that analyze a user's facial image and voice data to recognize their emotions.

[1337] An "emotion adjustment means" is a combination of software and hardware for adjusting recipe suggestions and information presentation methods based on recognized emotions.

[1338] The system of this invention not only efficiently manages food ingredients in a refrigerated storage device and provides information on expiration dates and storage methods, but also has the function of recognizing the user's emotions and adjusting the way information is presented and recipe suggestions based on those emotions. This system is composed of the following elements, which operate in an integrated manner.

[1339] System configuration

[1340] 1. Terminal

[1341] The terminal is a device installed inside the refrigerated storage unit and has the following functions: The terminal uses a camera or scanner as an information acquisition means to acquire information about ingredients inside the refrigerated storage unit. For example, it uses barcode scanning and image recognition technology. The terminal processes the acquired ingredient information using a data processing means to identify the type, quantity, expiration date, etc. of the ingredients.

[1342] Furthermore, the device recognizes emotions by analyzing the user's facial images and voice data, using image recognition technology and voice analysis software.

[1343] 2. Server

[1344] The server receives and processes the information sent from the terminal and manages the necessary data. The server's identification means compares the received ingredient information with a database and identifies detailed ingredient information (e.g., expiration date and storage method). The identified information is provided to the user device using the information presentation means.

[1345] The server's recipe generation means generates optimal recipes based on the ingredient information and suggests them to the user. Furthermore, the emotion adjustment means adjusts the content of the recipe suggestions and the way in which the information is presented based on the user's emotion information. For example, if the user is tired, the server will suggest simple recipes that can be prepared in a short time.

[1346] 3. User Devices

[1347] The user device is a user interface such as a smartphone or tablet. The user device displays and operates information provided by the server. Information such as ingredient information, expiration dates, storage methods, and recipes are visually presented to the user through the information presentation means.

[1348] Specific examples

[1349] 1. The device scans the refrigerator

[1350] The device uses a camera to take pictures of the inside of the refrigerator and recognizes barcodes, labels, and shapes. For example, milk, eggs, and broccoli are detected. The device analyzes this information and sends it to a server.

[1351] 2. The server processes the information

[1352] The server receives the ingredient information sent from the device and compares it with the database to identify information such as "Milk (Expiration date: 2023 / 10 / 15, Amount: 1L)," "Eggs (Expiration date: 2023 / 10 / 10, Amount: 6 eggs)," and "Broccoli (Expiration date: 2023 / 10 / 12, Amount: 1 bunch)." The server then calculates and suggests the optimal recipe based on the ingredient information. For example, it might suggest "Omelette," "Cream Broccoli Soup," or "Frittata."

[1353] 3. User Emotion Recognition and Information Provision

[1354] When a user opens the smartphone app, the device captures the user's facial image and voice via the camera and microphone, and identifies their emotion using an emotion engine. For example, if the server recognizes that the user is feeling stressed, it will use that information to present simple recipes for relaxation, along with soothing color schemes and audio.

[1355] Prompt Sentence Examples

[1356] "Imagine a system that not only efficiently manages the ingredients in your refrigerator and tells you expiration dates and storage methods, but also recognizes your emotions and suggests the best recipes based on those emotions. For example, it could suggest quick meals when you're tired, and more challenging recipes when you have the time."

[1357] The system of the present invention makes food management in a refrigerated storage device more efficient and can provide personalized support that responds to the user's emotions.

[1358] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1359] Step 1:

[1360] The device obtains information about ingredients in the refrigerator. When the user opens the refrigerator door, the device's camera and scanner are automatically activated. For example, the camera takes a photo of the food in the refrigerator, and the scanner reads the barcode. The input data are images of the ingredients in the refrigerator and their barcode information. Based on this input data, the device uses image recognition technology to identify the ingredient information and format it as text data. The output is text information that includes the type, quantity, and expiration date of each ingredient.

[1361] Step 2:

[1362] The device sends the acquired ingredient information to the server. The input data is text-format ingredient information generated on the device. The device then sends this information to the server via the network. The data received by the server is text data containing detailed information about the ingredients. Specifically, the data is sent securely using a data transmission protocol (e.g., HTTP, HTTPS).

[1363] Step 3:

[1364] The server processes the ingredient information. The input data is the text-formatted ingredient information sent from the terminal. The server's identification means checks the information against a database to identify detailed information about the ingredient. For example, a database query is used to obtain detailed information such as "Milk (best before date: 2023 / 10 / 15, quantity: 1L)." As output, detailed information about the identified ingredient is generated in text format. Specifically, an SQL query is executed on the server to extract the required information from the database.

[1365] Step 4:

[1366] The server's recipe generation means generates an optimal recipe based on the ingredient information. The input data is detailed information about the identified ingredients. The server uses an algorithm to calculate an optimal recipe based on the current ingredients. For example, dish names such as "Omelette," "Cream Broccoli Soup," and "Frittata" are suggested. The output is a list of suggested recipes. Specifically, the server checks the recipe database programmatically and selects the optimal recipe.

[1367] Step 5:

[1368] A user opens a smartphone app and performs emotion recognition. The input data is the user's facial image and voice data. The device's camera and microphone capture this data and send it to the device's emotion recognition means. The device identifies the emotion using facial expression analysis software and voice analysis software. The output is analyzed emotional information. For example, it may determine that the user is "feeling stressed" based on facial expression or voice tone.

[1369] Step 6:

[1370] The server receives the emotional information and adjusts the information presentation method and recipe suggestions. The input data is the emotional information sent from the device. The server's emotion adjustment means uses this information to adjust the content and method of information provided to the user. For example, a user in a stressed state might be suggested "meals that can be made in a short time" or "easy recipes that will help them relax." The output generates a customized recipe list and visually and audibly adjusted information presentation. In concrete terms, the server compares the information with a recipe database and selects the recipe that best suits the user's emotional state, while also adjusting the color scheme of the information presentation screen and the audio guidance.

[1371] Step 7:

[1372] The user's experience is executed. Input data is a customized recipe list and adjusted information presentation sent from the server. The user receives this information through a smartphone app, selects a dish, and begins cooking. Specific operations include displaying the recipe on the user's smartphone, allowing them to cook according to their mood and situation that day.

[1373] (Application example 2)

[1374] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1375] Conventional refrigerated storage devices make it difficult to efficiently manage expiration dates and storage conditions of ingredients, leading to users often wasting expired ingredients. While recipe suggestions based on ingredient information were offered, they were unable to tailor suggestions to individual users' emotions or circumstances, resulting in low satisfaction. Furthermore, the time required to manually order missing ingredients meant that these systems lacked convenience.

[1376] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1377] In this invention, the server includes an emotion engine that recognizes emotions, an adjustment means that adjusts the content of suggestions and the method of information presentation based on the user's emotion information, and a delivery ordering means that automatically orders ingredients that are in short supply, thereby enabling personalized recipe suggestions and automatic delivery ordering according to the user's emotions.

[1378] The "information acquisition means" is a device that is placed inside the refrigerated storage device and acquires food ingredient information using barcode scanning or image recognition technology.

[1379] The "data processing means" is a device that analyzes the acquired ingredient information and identifies the condition of the ingredients, such as type, quantity, and expiration date.

[1380] The "identification means" is a device that identifies the expiration date and storage method based on the state of the identified food material.

[1381] The "information presentation means" is a device for providing the identified information to the user.

[1382] The "recipe generation means" is a device that generates and proposes optimal recipes based on the identified ingredient information.

[1383] An "emotion engine" is a device that analyzes a user's facial images and voice data to recognize their emotions.

[1384] The "adjustment means" is a device that adjusts the content of recipe suggestions and the method of presenting information based on the user's emotional information.

[1385] The "delivery ordering means" is a device that automatically places an additional order when ingredients are in short supply.

[1386] The system of this invention not only efficiently manages food ingredients in a refrigerated storage device and provides information on expiration dates and storage methods, but also has the function of recognizing the user's emotions and adjusting the way information is presented and recipe suggestions based on those emotions. The main elements of the system are a terminal installed in the refrigerated storage device, a server that processes data, and a user device.

[1387] Terminal

[1388] The terminal is located within the refrigerated storage unit and has the following functions:

[1389] 1. Information acquisition method: Using a smartphone camera, food items in the refrigerator are scanned. Image recognition technology and barcode scanning are used to acquire food information (name, quantity, expiration date).

[1390] 2. Data processing means: Analyze the acquired food ingredient information and identify the type, quantity, expiration date, etc.

[1391] server

[1392] The server receives the information sent by the terminal and performs the following functions:

[1393] 1. Identification method: Based on the acquired data, the expiration date and storage method are checked against a database to identify the product.

[1394] 2. Recipe generation means: Generates optimal recipes using the identified ingredients and suggests them to the user.

[1395] 3. Emotion Engine: Analyzes the user's facial image and voice data to recognize emotions. This is done using EmotionRecognizer (software for emotion recognition).

[1396] 4. Adjustment method: Based on the information obtained from the emotion engine, recipe suggestions and the way information is presented are adjusted.

[1397] 5. Delivery ordering method: Automatically order missing ingredients from a food delivery service.

[1398] User Device

[1399] The user device (smartphone or tablet) is used to display and operate the information provided by the server.

[1400] 1. Information presentation means: Visually present information about ingredients, expiration dates, storage methods, recipes, etc. to the user.

[1401] Specific examples

[1402] When a user scans ingredients in a refrigerator, information such as "Milk (Best before date: 2023 / 10 / 15, Quantity: 1L)," "Eggs (Best before date: 2023 / 10 / 10, Quantity: 6 eggs)," and "Broccoli (Best before date: 2023 / 10 / 12, Quantity: 1 bunch)" is obtained. This information is sent from the device to the server and registered in a database along with storage instructions. Based on this information, the server generates recipes such as "Omelette" and "Cream Broccoli Soup" and suggests them to the user.

[1403] When a user opens the smartphone app and captures their facial images and voice via the camera and microphone, the emotion engine identifies the user's emotions. For example, if the app detects that the user is feeling stressed, it will present information such as simple, relaxing recipes with soothing colors and sounds. If the suggested recipe is missing ingredients, the delivery ordering method will automatically order the missing ingredients.

[1404] Prompt Sentence Examples

[1405] Ingredients needed: 1L milk, 6 eggs, 1 bunch of broccoli, 200g cheese

[1406] Best before date: Milk 2023 / 10 / 15, Eggs 2023 / 10 / 10, Broccoli 2023 / 10 / 12

[1407] User Emotion: Tired

[1408] Generate suggested recipes.

[1409] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1410] Step 1:

[1411] The terminal scans ingredients in the refrigerator. Using the smartphone camera, it obtains ingredient information using barcode scanning and image recognition technology. The data obtained includes the ingredient's name, quantity, and expiration date. For example, the camera scans the barcode on a piece of milk and obtains the information "Milk (quantity: 1L, expiration date: 2023 / 10 / 15)."

[1412] Step 2:

[1413] The device analyzes the acquired ingredient information using a data processing means and sends the identified ingredient information to the server. The input is ingredient information (name, quantity, expiration date), and the output is a data packet containing this information. For example, a data packet such as "Eggs (quantity: 6, expiration date: 2023 / 10 / 10)" is sent to the server.

[1414] Step 3:

[1415] The server receives the data packet and uses an identification means to compare it with the database. The data calculation performed here is to identify the optimal storage method based on the ingredient name, quantity, and expiration date. For example, for "broccoli (quantity: 1 bunch, expiration date: 2023 / 10 / 12)", the storage method is identified as "refrigerated storage."

[1416] Step 4:

[1417] The server sends the identified storage information and expiration date information to the user device, and the information presentation means presents it to the user. The input is the identified storage method and expiration date information, and the output is the display on the user device. For example, the information "Broccoli (storage method: refrigerated, expiration date: 2023 / 10 / 12)" is displayed on a smartphone.

[1418] Step 5:

[1419] The server's recipe generation means generates an optimal recipe based on the identified ingredient information. The input is ingredient information and storage method, and the output is the generated recipe. For example, it generates recipes for "omelette" or "cream broccoli soup" based on "milk (amount: 1L), eggs (amount: 6), broccoli (amount: 1 bunch)".

[1420] Step 6:

[1421] A user opens a smartphone app and sends facial images and voice data to the emotion engine via the camera and microphone. The input is the user's facial image and voice data, and the output is the identified emotion. For example, the emotion engine may identify the user as "tired."

[1422] Step 7:

[1423] The server's adjustment means adjusts the recipe suggestions and information presentation method based on the identified emotional information. The input is the user's emotional information and ingredient information, and the output is the adjusted recipe suggestions and information presentation method. For example, for a tired user, information is presented that includes "easy recipes" and "relaxing color schemes."

[1424] Step 8:

[1425] The user checks the proposed recipe, and if there are missing ingredients, the delivery ordering method automatically orders the missing ingredients from a food delivery service. The input is a list of the missing ingredients, and the output is an order confirmation to the delivery service. For example, "200g of cheese" and "3 tomatoes" can be automatically ordered.

[1426] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1427] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1428] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1429] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1430] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1431] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1432] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1433] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1434] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1435] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1436] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1437] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1438] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1439] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1440] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1441] The hardware resource for executing a specific process can be any of the following types of processors: A processor, for example, is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1442] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1443] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1444] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1445] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1446] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1447] The following is further disclosed regarding the above embodiment.

[1448] (Claim 1)

[1449] an information acquisition means disposed within the refrigerated storage device;

[1450] a data processing means for identifying the state of ingredients based on the acquired information;

[1451] an identification means for identifying a best-before date and a storage method based on the identified condition;

[1452] an information presentation means for providing the identified information to a user;

[1453] a recipe generator for generating suggested recipes using the identified ingredients;

[1454] A system including:

[1455] (Claim 2)

[1456] 2. The system according to claim 1, wherein the information acquisition means is a means for scanning a barcode and extracting ingredient information.

[1457] (Claim 3)

[1458] 10. The system of claim 1, wherein the data processing means includes means for identifying ingredients using image recognition technology.

[1459] "Example 1"

[1460] (Claim 1)

[1461] an information acquisition means disposed within the refrigerated storage device;

[1462] a data processing means for identifying the state of ingredients based on the acquired information;

[1463] an identification means for identifying a best-before date and a storage method based on the identified condition;

[1464] an information presentation means for providing the identified information to a user;

[1465] a recipe generator for generating suggested recipes using the identified ingredients;

[1466] means for transmitting the generated recipe to a user device;

[1467] A system including:

[1468] (Claim 2)

[1469] 2. The system according to claim 1, wherein the information acquisition means is a means for scanning a barcode and extracting ingredient information.

[1470] (Claim 3)

[1471] 10. The system of claim 1, wherein the data processing means includes means for identifying ingredients using image recognition technology.

[1472] (Claim 4)

[1473] 10. The system of claim 1, wherein the recipe generation means includes means for generating the recipe using a generative AI model.

[1474] (Claim 5)

[1475] The system of claim 4, further comprising means for constructing a prompt sentence to input ingredient information to the generative AI model.

[1476] "Application Example 1"

[1477] (Claim 1)

[1478] an information acquisition means disposed within the refrigerated storage device;

[1479] a data processing means for identifying the state of ingredients based on the acquired information;

[1480] an identification means for identifying a best-before date and a storage method based on the identified condition;

[1481] an information presentation means for providing the identified information to a user;

[1482] a recipe generator for generating suggested recipes using the identified ingredients;

[1483] a recipe suggestion means for suggesting recipes using a generative AI model that generates optimal recipes based on ingredient information;

[1484] an input means for inputting a prompt sentence to the recipe suggestion means;

[1485] A system including:

[1486] (Claim 2)

[1487] 2. The system according to claim 1, wherein the information acquisition means is a means for scanning a barcode and extracting ingredient information.

[1488] (Claim 3)

[1489] 10. The system of claim 1, wherein the data processing means includes means for identifying ingredients using image recognition technology.

[1490] "Example 2: Combining Emotion Engines"

[1491] (Claim 1)

[1492] An information acquisition means;

[1493] a data processing means for identifying the state of ingredients based on the acquired information;

[1494] an identification means for identifying a best-before date and a storage method based on the identified condition;

[1495] an information presentation means for providing the identified information to a user;

[1496] a recipe generator for generating suggested recipes using the identified ingredients;

[1497] An emotion recognition means for recognizing an emotion of a user;

[1498] an emotion adjustment means for adjusting recipe suggestions and information presentation methods based on the recognized emotions;

[1499] A system including:

[1500] (Claim 2)

[1501] 10. The system of claim 1, further comprising means for scanning barcodes to extract ingredient information.

[1502] (Claim 3)

[1503] 10. The system of claim 1, further comprising means for identifying ingredients using image recognition technology.

[1504] "Application example 2 when combining emotion engines"

[1505] (Claim 1)

[1506] an information acquisition means disposed within the refrigerated storage device;

[1507] a data processing means for identifying the state of ingredients based on the acquired information;

[1508] an identification means for identifying a best-before date and a storage method based on the identified condition;

[1509] an information presentation means for providing the identified information to a user;

[1510] a recipe generator for generating suggested recipes using the identified ingredients;

[1511] Includes an emotion engine that recognizes emotions,

[1512] an adjustment means for adjusting the content of suggestions and the method of presenting information based on the emotional information of the user;

[1513] A delivery ordering method that automatically orders missing ingredients,

[1514] A system including:

[1515] (Claim 2)

[1516] 2. The system according to claim 1, wherein the information acquisition means is a means for scanning a barcode and extracting ingredient information.

[1517] (Claim 3)

[1518] 10. The system of claim 1, wherein the data processing means includes means for identifying ingredients using image recognition technology. [Explanation of symbols]

[1519] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. an information acquisition means disposed within the refrigerated storage device; a data processing means for identifying the state of ingredients based on the acquired information; an identification means for identifying a best-before date and a storage method based on the identified condition; an information presentation means for providing the identified information to a user; a recipe generator for generating suggested recipes using the identified ingredients; A system including:

2. 2. The system according to claim 1, wherein the information acquisition means is means for scanning a barcode and extracting ingredient information.

3. 10. The system of claim 1, wherein the data processing means includes means for identifying ingredients using image recognition technology.

Images