System

Abstract

A system is provided.SOLUTION: A system comprising: a detection unit configured to detect an object in a refrigeration apparatus; a transmission unit configured to transmit object information detected by the detection unit to a processing apparatus; a generation unit configured to generate a cooking method based on the object information; an image generation unit configured to generate an image of the generated cooking method; and a transmission unit configured to transmit the generated cooking method and image to a user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's busy lifestyles, managing ingredients at home and devising cooking methods can be a tedious task. It is particularly difficult to plan nutritionally balanced meals while maintaining the freshness of ingredients in the refrigerator and making effective use of them. Another important issue is how to purchase the necessary ingredients efficiently while preventing food waste. [Means for solving the problem]

[0005] The present invention provides a system including a detection means for detecting objects in a refrigeration unit, a transmission means for transmitting object information detected by the detection means to a processing device, a generation means for the processing device to generate a cooking method based on the object information, an image generation means for generating an image of the generated cooking method, and a means for transmitting the generated cooking method and image to a user terminal. This system allows users to automatically manage ingredients in a refrigerator and provides recipe suggestions and images, thereby reducing the hassle of cooking at home and preventing food waste. The present invention also includes a means for the processing device to periodically check object information in the refrigeration unit, generate a cooking plan for a certain period of time, and generate a list of missing objects. The system also includes a means for purchasing missing objects from an online sales device based on the list, allowing users to efficiently purchase necessary ingredients. Generating a list of missing ingredients and automatically purchasing them through the online sales device also saves the user the trouble of going shopping.

[0006] A "refrigeration device" is a device used to store food, beverages, etc. at low temperatures.

[0007] "Object" refers to any item, such as food or beverage, stored within the refrigeration unit.

[0008] "Detection Means" means any device or technology used to recognize and detect objects within the refrigeration unit, including, for example, RFID tags and image recognition technology.

[0009] The "transmitting means" refers to a device or technology for transmitting the object information obtained by the detecting means to the processing device.

[0010] The "processing device" refers to a computer or software that processes the object information received from the transmitting means and generates a cooking method.

[0011] The "generation means" refers to a program or algorithm that the processing device uses to generate a cooking method based on object information.

[0012] "Image generation means" refers to the technology or software used to generate an image of the generated recipe.

[0013] A "user terminal" refers to a device such as a computer or smartphone that receives and displays the generated recipes and images.

[0014] An "online sales device" is an online system for automatically purchasing missing items. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] This invention is a system that automates the management of food and beverages stored in a refrigerator and suggests effective cooking methods. The system detects objects in the refrigerator and generates cooking instructions and images based on that information, which are then provided to the user. It also has a function that periodically checks the information about the objects in the refrigerator and automatically purchases any missing items online.

[0037] (Program overview and processing explanation)

[0038] Ingredient detection and data generation

[0039] 1. A sensor scans the object inside the refrigerator.

[0040] The server installs multiple sensors inside the refrigeration unit and uses image recognition technology and RFID tags to detect food and beverages.

[0041] For example, the sensor detects two tomatoes, 50g of cheese, and a small amount of basil.

[0042] 2. The server receives object information from the sensor.

[0043] The server receives object information detected by the sensor in real time.

[0044] Examples of received data include {"Object": [{"Name": "Tomato", "Quantity": "2"}, {"Name": "Cheese", "Quantity": "50g"}, {"Name": "Basil", "Quantity": "Small"}]}.

[0045] 3. The server stores the received object information in a database.

[0046] The server stores the received object information in a database and manages it as the latest inventory information.

[0047] Recipe generation and image creation

[0048] 4. The server sends a recipe generation request to the generative AI based on the object information.

[0049] The server sends a request to the generative AI to generate a recipe based on information about the objects inside the refrigerator.

[0050] For example, the server generates a request containing tomato, cheese, basil, and bread.

[0051] 5. A generative AI receives the request and generates a recipe based on the object.

[0052] The generative AI generates the optimal recipe based on the requested object.

[0053] For example, the generated recipe is "Tomato and Basil Bruschetta."

[0054] 6. The server sends the generated recipe to DALL-E and generates an image of the finished dish.

[0055] The server sends an image generation request to DALL-E based on the recipe information received from the generative AI.

[0056] For example, send a request to "generate an image of bruschetta."

[0057] 7. The server sends the generated recipe and image to the user device.

[0058] The server transmits the generated recipe text and image file to the user terminal.

[0059] The content provided includes a "Tomato and Basil Bruschetta Recipe" and an image of it.

[0060] Regular menu planning and online purchasing

[0061] 8. The server periodically checks the information of objects in the refrigerator.

[0062] The server retrieves and checks information about objects in the refrigerator from the database at a specific time each week.

[0063] 9. The server requests a week's worth of menus from the generative AI.

[0064] To generate a week's worth of menus, the server requests information about all objects and the number of dishes required from the generative AI.

[0065] 10. Generative AI generates a week's worth of menus and creates a list of missing items.

[0066] The generative AI creates a weekly menu and identifies missing items and generates a list.

[0067] 11. The server sends the list of missing objects to the online sales device and purchases the required objects.

[0068] The server sends a list of missing objects to an online sales device, which automatically purchases the objects.

[0069] User Examples

[0070] For example, if tomatoes, cheese, basil, and bread are stored in a refrigerator, the user uses the system in the following steps:

[0071] 1. The user starts using the refrigeration unit.

[0072] A user places an object in the refrigerator and a sensor detects this.

[0073] 2. The server receives the object information and stores it in a database.

[0074] The server receives data from the sensors and registers the object information in a database.

[0075] 3. The server requests the generative AI to generate a recipe.

[0076] The server requests the generative AI to generate a recipe based on the object information, and the AI ​​generates a recipe for "Tomato and Basil Bruschetta."

[0077] 4. The server requests image generation from DALL-E.

[0078] The server sends the generated recipe to DALL-E, which generates an image of bruschetta.

[0079] 5. The server sends the generated recipe and image to the user device.

[0080] The recipe and images are displayed on the user's terminal, and the user cooks while looking at them.

[0081] 6. The server periodically checks the object information and creates a menu for the week.

[0082] The server periodically checks the object information and requests a week's worth of menus from the generative AI.

[0083] 7. Automatically purchase absent objects online.

[0084] The server transmits a list of missing items to the online sales device, which automatically carries out the purchase procedure.

[0085] In this way, the present invention automates ingredient management, recipe suggestions, menu creation, and online purchasing, supporting users' busy lives.

[0086] The processing flow will be explained below.

[0087] Step 1:

[0088] The server activates sensors to detect objects inside the refrigerator, which use image recognition technology and RFID tags to detect food and beverages inside the refrigerator.

[0089] Step 2:

[0090] The sensor detects the objects in the refrigerator and sends them to the server. For example, the sensor detects two tomatoes, 50g of cheese, and a small amount of basil.

[0091] Step 3:

[0092] The server stores the object information received from the sensor in a database, including the object name and quantity.

[0093] Step 4:

[0094] The server retrieves the latest object information from the database and sends a recipe generation request to the generative AI, which includes information on all ingredients in the refrigerator.

[0095] Step 5:

[0096] The generative AI receives requests from the server and generates recipes based on the object information. For example, it generates a recipe for "Tomato and Basil Bruschetta."

[0097] Step 6:

[0098] The server sends the generated recipe information to DALL-E, which is an image generation means, and sends a request to generate an image of the finished dish.

[0099] Step 7:

[0100] DALL-E generates an image based on the recipe and sends it back to the server. For example, it generates an "image of bruschetta."

[0101] Step 8:

[0102] The server sends the generated recipe and image to the user's device, where the user can view the proposed recipe and image.

[0103] Step 9:

[0104] The server periodically checks the information of objects in the refrigerator once a week. This check is performed automatically by the scheduler.

[0105] Step 10:

[0106] The server retrieves all object information from the database and sends a request to the generative AI to generate a week's worth of menus.

[0107] Step 11:

[0108] A generative AI receives the request and generates a week's worth of cooking plans and a list of ingredients needed, including any missing ingredients.

[0109] Step 12:

[0110] The server transmits the generated list of absent objects to an online sales device, which automatically purchases the missing objects.

[0111] Step 13:

[0112] Users can pick up ingredients purchased online at a designated location. By receiving any missing ingredients, users can prepare a week's worth of meals as planned.

[0113] Example 1

[0114] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] In today's busy world, users spend a lot of time managing ingredients and planning their cooking. In addition, improper management of food and beverages stored in refrigerators can lead to food waste. Furthermore, it can be cumbersome for users to regularly purchase ingredients needed for cooking, and poor management can make cooking difficult.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0117] In this invention, the server includes a detection means for detecting objects in the refrigeration unit, a transmission means for transmitting object information detected by the detection means to a processing device, and a generation means for the processing device to generate a cooking method based on the object information. This enables automatic ingredient management and optimal cooking method suggestions. The server also includes an image generation means for generating an image of the generated cooking method, a means for transmitting the generated cooking method and the image to a user terminal, a means for storing the object information received by the processing device in a database, a means for the processing device to request recipe generation from a generative artificial intelligence model, a means for the generative artificial intelligence model to receive the request and generate a recipe based on the object, and a means for the processing device to transmit the generated recipe to an image generation model and generate an image of the finished dish. This allows users to consistently automate everything from daily ingredient management to cooking method suggestions and food image confirmation. It also reduces food waste and realizes efficient ingredient purchasing.

[0118] The "detection means" is a device installed to detect objects inside the refrigeration device, and may use image recognition technology or radio frequency identification technology.

[0119] The "transmitting means" is a device or system for transmitting object information detected by the detecting means to the processing device.

[0120] The "generation means" is a device or system that allows the processing device to generate a cooking method based on object information.

[0121] "Image generation means" refers to a device or system for generating an image of the generated recipe.

[0122] The "means for transmitting to the user terminal" refers to a device or system for transmitting the generated recipes and images to the terminal used by the user.

[0123] A "processing device" is a device or system for processing object information received from a sensor and storing it in a database.

[0124] A "generative artificial intelligence model" is an artificial intelligence model that generates recipes in natural language based on input prompts.

[0125] An "image generation model" is an artificial intelligence model for generating images based on input text information.

[0126] A "database" is a data management system for storing and managing received object information.

[0127] A "refrigeration device" is a device used to store food and beverages at a constant low temperature.

[0128] "Object information" is data about the names and quantities of food and beverages stored in the refrigerator.

[0129] A "recipe" is information that describes the steps and ingredients for cooking a particular dish.

[0130] This invention is a system that automates the management of food and beverages stored in a refrigerator and suggests effective cooking methods. The system detects objects in the refrigerator and generates cooking instructions and images based on that information, which are then provided to the user. It also has a function that periodically checks the information about the objects in the refrigerator and automatically purchases any missing items online.

[0131] This system mainly uses the following hardware and software:

[0132] 1. Sensors: Image recognition sensors and RFID readers are used to detect objects inside the refrigerator.

[0133] 2. Server: Receives object information from sensors, stores it in a database, and requests recipe and image generation from generative artificial intelligence models and image generation models.

[0134] 3. Generative AI models: For example, using natural language generation models such as GPT-3 to generate recipes based on ingredients.

[0135] 4. Image generation model: For example, a text-to-image generation model such as DALL-E is used to generate images of the generated recipe.

[0136] 5. User device: For example, the generated recipes and images are provided to the user using a smartphone or tablet.

[0137] As a concrete example, consider a case where a user stores two tomatoes, 50g of cheese, a little basil, and bread in a refrigerator. The system operates as follows:

[0138] First, sensors inside the refrigerator detect these ingredients and send object information to a server, which stores this information in a database and manages the overall inventory.

[0139] The server then sends a recipe generation request to the generative artificial intelligence model, such as a prompt like "Generate a recipe using tomatoes, cheese, basil, and bread."

[0140] The generative AI model receives this request and generates a recipe called "Tomato and Basil Bruschetta." The server then sends this generated recipe to the image generation model, requesting it to "generate an image of the bruschetta."

[0141] The image generation model receives the request and generates an image of the bruschetta. The server then sends the generated recipe and image to the user's device. The user then cooks the dish while viewing the recipe and image on the device.

[0142] Furthermore, the server can periodically check the information about the objects in the refrigerator and ask the generative AI model to create a week's worth of menus. Once a list of missing objects is created, the server sends the list to the online sales device, which then automatically carries out the purchasing process.

[0143] In this way, the present invention reduces the burden on the user and enables efficient and effective food ingredient management and cooking.

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1:

[0146] Sensors scan objects inside the refrigerator

[0147] Input: Current state of ingredients in the refrigerator

[0148] How it works: Image recognition sensors and RFID readers installed inside the refrigerator detect ingredients like tomatoes, cheese, and basil. Each sensor captures image data and RFID tag information for each ingredient.

[0149] Output: A list of ingredients and their amounts (e.g., "2 tomatoes, 50g cheese, a little basil")

[0150] Step 2:

[0151] The server receives object information from the sensor

[0152] Input: A list of ingredient names and quantities sent from the sensor

[0153] Specific operation: The server receives object information sent from the sensor in real time. The received data format is, for example, JSON: {"Object": [{"Name": "Tomato", "Quantity": "2"}, {"Name": "Cheese", "Quantity": "50g"}, {"Name": "Basil", "Quantity": "Small"}]}.

[0154] Output: Received object information data

[0155] Step 3:

[0156] The server stores the received object information in a database.

[0157] Input: Received object information data

[0158] Specific operation: The server analyzes the object information and stores it in a database. The database keeps up-to-date information on the types and quantities of ingredients. In this example, the database stores "2 tomatoes, 50g of cheese, and a little basil."

[0159] Output: Latest object information stored in the database

[0160] Step 4:

[0161] The server sends a recipe generation request to the generative AI based on the object information.

[0162] Input: The latest object information stored in the database

[0163] Specific operation: The server retrieves object information from the database and sends it to the generative AI (e.g., GPT-3) as a recipe generation request. The request is sent in the form of a prompt: "Please generate a recipe using tomatoes, cheese, basil, and bread."

[0164] Output: The prompt sent to the generative AI

[0165] Step 5:

[0166] A generative AI receives the request and generates a recipe based on the object.

[0167] Input: Prompt sent from the server

[0168] How it works: A generative AI (e.g., GPT-3) analyzes the prompt and generates an optimal recipe based on the input object information. For example, the generated recipe might be "Tomato and Basil Bruschetta."

[0169] Output: Generated recipe (text format)

[0170] Step 6:

[0171] The server sends the generated recipe to the image generation model, which generates an image of the finished dish.

[0172] Input: Recipe generated by generative AI

[0173] Specific operation: The server obtains the generated recipe information and sends an image generation request to the image generation model (e.g., DALL-E). The request is sent in the form of, for example, "Please generate an image of bruschetta."

[0174] Output: Generated food images

[0175] Step 7:

[0176] The server sends the generated recipe and image to the user's device.

[0177] Input: Generated recipes and food images

[0178] Specific operation: The server sends the generated recipe text and image file to the user's device so that the user can check it. For example, the user's smartphone will display "Tomato and Basil Bruschetta Recipe" and its image.

[0179] Output: Recipe and image displayed on the user's device

[0180] Step 8:

[0181] The server periodically checks the information about objects in the refrigerator.

[0182] Input: Object information stored in the database

[0183] Specific operation: The server periodically (e.g., every Monday at 9:00 AM) accesses the database and checks the information about objects in the refrigerator.

[0184] Output: Information about the most recent object in the refrigerator that was checked

[0185] Step 9:

[0186] The server requests a week's worth of menus from the generative AI

[0187] Input: Information about the object in the most recent refrigerator that was checked

[0188] Specific operation: To generate a week's worth of menus, the server requests all object information in the database and the number of dishes required from the generative AI. The request is sent in the form of a prompt, such as "Please generate a seven-day menu."

[0189] Output: The prompt sent to the generative AI

[0190] Step 10:

[0191] Generative AI generates a week's worth of meals and creates a list of missing items

[0192] Input: Prompt sent from the server

[0193] Specific operation: A generative AI (e.g., GPT-3) analyzes the prompts and generates a weekly menu, identifying and listing any missing items. For example, a generated menu and a "list of missing items" are created.

[0194] Output: Generated weekly menu and list of missing items

[0195] Step 11:

[0196] The server sends the list of missing objects to the online sales device and purchases the required objects.

[0197] Input: Generated missing object list

[0198] Specific operation: The server sends a list of missing items to the online sales device, which then automatically processes the purchase. For example, a list of "2 tomatoes, 100g of cheese" is sent, and an online order is placed.

[0199] Output: Required objects purchased

[0200] In this way, the entire system flow is completed, allowing users to smoothly carry out a series of processes, from managing ingredients and suggesting cooking methods to automatically purchasing the necessary ingredients.

[0201] (Application example 1)

[0202] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0203] Managing food and beverages in a refrigerator requires manual checking and replenishing, which is a significant time burden for users. Furthermore, since no cooking method suggestions are provided, it is difficult to effectively utilize the ingredients in the refrigerator. Furthermore, there is no system that automatically replenishes ingredients when they run out, so users must manually order them. To solve these issues, there is a need for a system that can automatically manage food and beverages, suggest effective cooking methods, and automatically replenish ingredients when they run out.

[0204] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0205] In this invention, the server includes a detection means for detecting objects in the refrigeration unit, a transmission means for transmitting object information detected by the detection means to a processing device, a generation means for the processing device to generate a cooking method based on the object information, an image generation means for generating an image of the generated cooking method, a means for transmitting the generated cooking method and the image to a user terminal, and a replenishment means for the processing device to generate order information for replenishing ingredients based on the object information and automatically order ingredients through an online service. This enables automated management of food and beverages in the refrigeration unit, suggestions for effective cooking methods, and automatic replenishment of ingredients when they run out.

[0206] A "refrigeration unit" is a device designed to maintain a low internal temperature and is used to preserve food, beverages, etc.

[0207] A "system" is a whole made up of multiple elements that are interrelated and combined to achieve a certain purpose.

[0208] "Detection means" means a device or function, including sensor technology, for accurately identifying and detecting the presence and type of object within the refrigeration unit.

[0209] "Transmitting means" refers to a device or function that includes communication technology for transmitting object information collected by the detecting means to the processing device.

[0210] The "generation means" is a device or function that includes an algorithm or artificial intelligence for automatically creating a cooking method based on object information.

[0211] "Image generation means" refers to a device or function including an artificial intelligence or image generation engine for creating visual food images based on the generated cooking instructions.

[0212] A "user terminal" is a device such as a smartphone or computer that a user uses to receive, check, or operate information provided by the system.

[0213] The "replenishment means" is a function or system for automatically ordering and replenishing missing food or beverages based on object information.

[0214] The "processing device" is a computer or control unit that receives and analyzes the object information transmitted from the detection means and controls the entire system.

[0215] The system to realize this application example utilizes various sensors, processing devices, and communication means built into the refrigeration equipment to automatically manage food and beverages, suggest effective cooking methods, and even automatically replenish ingredients when they run out.

[0216] The server includes a detection means that uses image recognition technology or radio frequency identification technology to detect objects in the refrigeration device. The detection means collects object information using, for example, a camera or an RFID reader. The collected object information is transmitted to the server via a transmission means.

[0217] The server receives the transmitted object information with a processing device and generates an optimal cooking method based on the object information using a generation means, such as an OpenAI generative AI model. Based on the generated cooking method, an image generation engine such as DALL-E is used to generate an image of the cooking method.

[0218] The image of the cooking method and the recipe generated by the image generating means are transmitted to a user terminal via a transmitting means, which may be a smartphone or tablet, and the user can check the cooking method and the image through the user terminal.

[0219] Furthermore, the server periodically checks the information on the items in the refrigerator and generates a cooking plan for the week. Based on the generated cooking plan, it creates a list of items that are in short supply and automatically orders them online. This eliminates the need for users to manually check and order.

[0220] For example, if tomatoes, cheese, and basil are stored in the refrigerator, the server will request a recipe from the generative AI model based on this information. The generative AI model will suggest "Tomato and Basil Bruschetta," and DALL-E will generate an image of it. Furthermore, if the bread needed for the bruschetta is in short supply, it will automatically be ordered from an online service.

[0221] Here are some examples of prompts for generative AI models:

[0222] If you have the following ingredients in your fridge, please suggest some possible dishes.

[0223] object:

[0224] Name: Tomato

[0225] Quantity: 2 pieces

[0226] Name: Cheese

[0227] Amount: 50g

[0228] Name: Basil

[0229] Quantity: small quantity

[0230] This system automates the management of food and beverages in refrigeration units, allowing users to cook efficiently while saving time and effort.

[0231] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0232] Step 1:

[0233] Detect objects inside a refrigeration unit.

[0234] How it works: The server collects data on food and beverages using cameras and RFID readers installed inside the refrigeration units.

[0235] Input: Image data or RFID tag data of objects in the refrigeration unit.

[0236] Data processing: Identifying the type and quantity of objects using image recognition and RFID technology.

[0237] Output: Identified object information (e.g., "2 tomatoes", "50g cheese", "small amount of basil").

[0238] Step 2:

[0239] The detected object information is transmitted to a processing device.

[0240] Specific operations: The server transmits the object information obtained by the detection means to the processing device.

[0241] Input: Identified object information.

[0242] Data calculation: Formatting object information according to the protocol.

[0243] Output: Formatted object information data.

[0244] Step 3:

[0245] Cooking methods are generated based on object information.

[0246] Specific operation: The server sends a cooking method generation request to the generative AI model based on the object information.

[0247] Input: Formatted object information data.

[0248] Data calculation: The generative AI model performs natural language processing and database search to generate optimal recipes based on object information.

[0249] Output: The generated recipe (e.g., "Tomato and Basil Bruschetta").

[0250] Step 4:

[0251] Generate an image of the cooking method.

[0252] Specific operation: The server sends the generated text information of the cooking method to the image generation engine to generate a visual image of the dish.

[0253] Input: The generated text information of the cooking instructions.

[0254] Data calculation: An AI model processes and generates images based on text information.

[0255] Output: The generated food image (e.g. "Image of bruschetta").

[0256] Step 5:

[0257] The generated recipe and image are sent to the user terminal.

[0258] Specific operation: The server sends cooking instructions and images to the user's smartphone or tablet.

[0259] Input: Generated recipes and images.

[0260] Data calculation: Processes data in a format suitable for the user terminal and sends it.

[0261] Output: Cooking instructions and images displayed on the user's device.

[0262] Step 6:

[0263] Regularly check the contents of the refrigerator.

[0264] Specific operation: The server rechecks the information about objects in the refrigerator at set intervals.

[0265] Input: Current object information data.

[0266] Data calculation: Compare with existing information in the database and update new object information.

[0267] Output: Updated object information data.

[0268] Step 7:

[0269] Generate cooking plans for a certain period of time.

[0270] Specific operation: The server sends a request to the generative AI model to generate a cooking plan for a certain period (e.g., one week) based on the collected object information.

[0271] Input: Updated object information data.

[0272] Data calculation: A generative AI model generates a cooking plan that combines multiple recipes based on object information.

[0273] Output: Generated cooking plan (e.g., recipes for a week).

[0274] Step 8:

[0275] Automatically order missing items online.

[0276] Specific operation: The server creates a list of ingredients that are missing based on the generated cooking plan and places an order with the online sales service.

[0277] Input: Generated cooking plan, missing object list.

[0278] Data processing: Generates order information and processes orders through the API of online services.

[0279] Output: Online order confirmation information.

[0280] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0281] This invention relates to a system that manages objects stored in a refrigerator and suggests optimal cooking methods based on the user's emotions. The system detects objects stored in the refrigerator, generates optimal recipes based on that information, and customizes cooking methods by taking the user's emotions into account using an emotion engine. It also has the functionality to automate object management and periodic online purchasing of items that are in short supply.

[0282] (Program overview and processing explanation)

[0283] Ingredient detection and data generation

[0284] 1. A sensor scans the object inside the refrigerator.

[0285] The server activates sensors installed inside the refrigerator that use image recognition technology and RFID tags to detect objects, such as two tomatoes, 50g of cheese, and a small amount of basil.

[0286] 2. The server receives object information from the sensor.

[0287] The sensor sends the detected object information to the server, which stores it in a database. The stored information includes the object name and quantity.

[0288] Recipe Generation and Emotion Recognition

[0289] 3. The server launches an emotion engine that recognizes the user's emotions.

[0290] The emotion engine analyzes user input, voice, and facial expressions to recognize the user's emotional state. For example, if the user is feeling stressed, it will suggest recipes to help them relax.

[0291] 4. The server sends the object information and the user's emotion information to the generative AI.

[0292] The server sends the object information obtained from the database and the user's emotion information recognized by the emotion engine to the generative AI.

[0293] 5. Generative AI generates recipes based on object information and emotional information.

[0294] The generative AI generates optimal recipes based on object information and the user's emotional information. For example, if the user is feeling stressed, it generates a relaxing recipe such as "Tomato and Basil Bruschetta."

[0295] Image generation and information transmission

[0296] 6. The server sends the generated recipe to DALL-E, which is the image generation means.

[0297] The server sends the recipe information received from the generative AI to DALL-E, which then generates an image of the dish based on that recipe.

[0298] 7. DALL-E generates an image based on the recipe and sends it back to the server.

[0299] For example, an "image of bruschetta" is generated and sent back to the server.

[0300] 8. The server sends the generated recipe and image to the user device.

[0301] The server sends the generated recipe text and image files to the user's terminal, where the user can view the proposed recipe and images.

[0302] Regular menu planning and online purchasing

[0303] 9. The server periodically checks the information of objects in the refrigerator once a week.

[0304] The server checks the database for information about objects in the refrigerator at a specific time each week according to a scheduler.

[0305] 10. The server requests a week's worth of menus from the generative AI.

[0306] To generate a week's worth of menus, the server requests information about all objects and the number of dishes required from the generative AI.

[0307] 11. Generative AI generates a week's worth of menus and creates a list of missing items.

[0308] The generative AI creates a weekly menu and identifies missing items and generates a list.

[0309] 12. The server sends the list of missing objects to the online sales device and purchases the required objects.

[0310] The server sends a list of missing objects to an online sales device, which automatically purchases the objects.

[0311] User Examples

[0312] For example, if tomatoes, cheese, basil, and bread are stored in a refrigerator, the user uses the system in the following steps:

[0313] 1. The user starts using the refrigeration unit.

[0314] A user places an object in the refrigerator and a sensor detects this.

[0315] 2. The server receives the object information and stores it in a database.

[0316] The server receives data from the sensors and registers the object information in a database.

[0317] 3. The emotion engine detects the user's emotions.

[0318] The emotion engine analyzes the user's voice and facial expressions and recognizes that the user is seeking relaxation.

[0319] 4. The server requests the generative AI to generate a recipe.

[0320] The server requests a generative AI to generate a recipe based on object information and emotional information, and the AI ​​generates a recipe for "Tomato and Basil Bruschetta."

[0321] 5. The server requests DALL-E to generate an image.

[0322] The server sends the generated recipe to DALL-E, which generates an image of bruschetta.

[0323] 6. The server sends the generated recipe and image to the user device.

[0324] The recipe and images are displayed on the user's terminal, and the user cooks while looking at them.

[0325] 7. The server periodically checks the object information and creates a menu for the week.

[0326] The server periodically checks the object information and requests a week's worth of menus from the generative AI.

[0327] 8. Automatically purchase absent objects online.

[0328] The server transmits a list of missing items to the online sales device, which automatically carries out the purchase procedure.

[0329] In this way, the present invention automates ingredient management, recipe suggestions, menu planning, emotion-based recipe optimization, and online purchasing to support users' busy lives.

[0330] The processing flow will be explained below.

[0331] Step 1:

[0332] The server activates the sensor to detect the object inside the refrigerator. The sensor uses image recognition technology and RFID tags to detect the object (food or drink) inside the refrigerator.

[0333] Step 2:

[0334] The sensor detects information about objects in the refrigerator and sends it to the server. For example, the sensor detects "2 tomatoes," "50g of cheese," and "a little basil."

[0335] Step 3:

[0336] The server stores the object information received from the sensor in a database, including the object name and quantity.

[0337] Step 4:

[0338] The server activates an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's input, voice, and facial expressions to recognize the user's emotional state.

[0339] Step 5:

[0340] The emotion engine analyzes the user's emotions and generates emotion information, for example, recognizing that the user is feeling stressed.

[0341] Step 6:

[0342] The server sends a recipe generation request to the generative AI based on the object information obtained from the database and the emotion information generated by the emotion engine.

[0343] Step 7:

[0344] Generative AI generates optimal recipes based on object and emotional information. For example, it generates a recipe for "Tomato and Basil Bruschetta" to help users relax.

[0345] Step 8:

[0346] The server sends the generated recipe to DALL-E, an image generation means, which generates an image of the dish based on the recipe.

[0347] Step 9:

[0348] DALL-E generates an image of the dish based on the recipe and sends it back to the server. For example, it generates an image of bruschetta.

[0349] Step 10:

[0350] The server sends the generated recipe and image to the user's terminal, where the user can check the proposed recipe and image.

[0351] Step 11:

[0352] The server sets a scheduler to periodically check the object information in the refrigerator, retrieving and checking the object information from the database at a specific time every week.

[0353] Step 12:

[0354] The server retrieves all object information from the database and sends a request to the generative AI to generate a week's worth of menus.

[0355] Step 13:

[0356] Generative AI generates a weekly menu and creates a list of missing items. For example, it creates a menu that includes menus for Monday through Sunday.

[0357] Step 14:

[0358] The server transmits a list of missing objects to an online sales device, which automatically purchases the missing objects.

[0359] Step 15:

[0360] Users can pick up items they have purchased online at a specified location. By having any missing ingredients delivered to them, users can plan their meals for the week.

[0361] Example 2

[0362] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0363] In today's busy lifestyles, managing ingredients and suggesting effective cooking methods requires a great deal of effort. It is also difficult to find appropriate cooking methods that reduce stress and fatigue. Therefore, there is a need for a system that can effectively manage items in a refrigerator, suggest optimal cooking methods based on the user's emotions, and automatically purchase ingredients when they are in short supply.

[0364] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0365] In this invention, the server includes a detection means for detecting objects in the refrigeration unit, a transmission means for transmitting object information detected by the detection means to the processing device, a generation means for generating a cooking method based on the object information, an emotion recognition means for recognizing a user's emotion, a generation means for generating a cooking method based on the object information and emotion information, an image generation means for generating an image of the generated cooking method, and a means for transmitting the generated cooking method and image to a user terminal. This makes it possible to efficiently manage objects in the refrigeration unit and suggest appropriate cooking methods based on the user's emotion, not only facilitating meal preparation but also improving the user's psychological satisfaction. It also enables automatic purchasing of ingredients that are in short supply, further improving convenience.

[0366] "Detection means" refers to a device or system for detecting objects within the refrigeration unit.

[0367] The "transmitting means" refers to a device or system for transmitting object information detected by the detecting means to the processing device.

[0368] The term "processing device" refers to a device or system that generates and manages recipes based on object information received from a transmitting means.

[0369] The "generation means" refers to a device or system for generating a cooking method based on object information and user emotion information.

[0370] "Emotion recognition means" refers to a device or system for recognizing the emotional state of a user.

[0371] "Image generation means" refers to a device or system for generating an image based on the generated recipe.

[0372] The term "user terminal" refers to an apparatus or device for displaying generated recipes and images to a user.

[0373] "Database" refers to a system for storing object information, user emotion information, and other related data.

[0374] "Online sales device" refers to a system that processes orders online to purchase missing objects.

[0375] "Scheduler" refers to a system that manages a schedule for periodically checking information about objects in a refrigeration device.

[0376] The present invention relates to a system that manages items stored in a refrigerator and suggests optimal cooking methods based on the user's emotions. This system utilizes multiple hardware and software components to efficiently and automatically manage ingredients and suggest cooking methods.

[0377] Hardware and Software Configuration

[0378] 1. Sensors: Sensors with image recognition technology and RFID tag readers are placed inside the refrigerator, allowing for accurate detection of objects inside the refrigerator.

[0379] 2. Server: The central part of the system, it centrally manages and processes data from various devices and software, such as sensors, emotion recognition engines, generative AI, and image generation methods.

[0380] 3. Emotion Recognition Engine: Uses software to analyze user input (voice, facial expressions, etc.) and recognize emotions. For example, determining stress or happiness from the user's tone of voice and facial expressions.

[0381] 4. Generative AI: Uses AI models to generate optimal recipes based on object and emotion information. For example, this applies to recipe suggestion systems that use natural language processing.

[0382] 5. Image generation means: Software for generating food images based on the generated recipe. This includes DALL-E and similar image generation models.

[0383] 6. User device: A device (smartphone, tablet, etc.) on which the user checks the recipe and generated images. It displays the data sent from the server.

[0384] 7. Database: Use a system to store object information, user emotion information, recipe information, etc.

[0385] 8. Online Sales Device: A system for automatically purchasing missing items. The purchasing process is carried out using the online store API.

[0386] Specific actions

[0387] 1. Ingredient detection:

[0388] When a user places new ingredients in the refrigerator, the sensor scans them and sends the detection results to the server. For example, if a tomato and cheese are newly added to the refrigerator, image recognition technology and an RFID tag reader will detect this.

[0389] 2. Data storage:

[0390] The server analyzes the received data and stores it in a database. For example, it might register "2 tomatoes, 50g of cheese."

[0391] 3. Emotion Recognition:

[0392] When a user speaks into the smartphone or faces the camera, the emotion recognition engine analyzes this and sends the user's emotional information to the server. For example, it may determine that the user is feeling stressed.

[0393] 4. Generate the recipe:

[0394] The server retrieves object and emotion information from the database and sends it to the generative AI, which then generates the optimal recipe based on the user's emotions. For example, it suggests "relaxing tomato and basil bruschetta."

[0395] 5. Image Generation:

[0396] The server transmits the generated recipe to the image generating means, which generates a related food image, for example, an image of "bruschetta."

[0397] 6. Submit your recipe and images:

[0398] The server sends the generated recipe and images to the user terminal, where the user can check the proposed recipe and images.

[0399] 7. Regular Checks and Purchases:

[0400] The server periodically checks the information of the items in the refrigerator and lists the items that are missing, and automatically purchases the missing items through the online sales device based on the list.

[0401] Specific examples

[0402] Example prompt sentence:

[0403] "What are some relaxing dishes that you would suggest to users when they are feeling stressed?"

[0404] "If I have tomatoes, cheese, and basil in my refrigerator, can you suggest a recipe that works best?"

[0405] In this way, the present invention provides a system that allows users to efficiently manage ingredients and enjoy optimal recipes that suit their own emotional state.

[0406] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0407] Step 1:

[0408] Ingredient detection

[0409] Input: User stores new ingredients in the refrigerator.

[0410] Processing: The server activates sensors (image recognition technology and RFID tag readers) inside the refrigeration unit to scan the object.

[0411] Output: Data of detected ingredients.

[0412] What it does: When a user adds 2 tomatoes, 50g of cheese, and a little basil to the refrigerator, the sensor detects this and sends the data (2 tomatoes, 50g of cheese, a little basil) to the server.

[0413] Step 2:

[0414] Data storage

[0415] Input: Ingredient data sent from the sensor.

[0416] Processing: The server receives the data from the sensors and stores it in a database.

[0417] Output: Ingredient information stored in a database.

[0418] Specific operation: The server analyzes the data received from the sensor (2 tomatoes, 50g of cheese, and a small amount of basil) and registers it in a database.

[0419] Step 3:

[0420] Emotion recognition

[0421] Input: User voice and facial expression data.

[0422] Processing: The server uses an emotion recognition engine to analyze the user's emotions.

[0423] Output: The perceived emotional state of the user.

[0424] Specific operation: When a user speaks into the smartphone or faces the camera, the emotion recognition engine analyzes this and determines that the user is feeling stressed.

[0425] Step 4:

[0426] Recipe Generation

[0427] Input: Ingredient information obtained from the database and emotion information from the emotion recognition engine.

[0428] Processing: The server sends the ingredient information and emotion information to the generative AI and requests it to generate an appropriate recipe.

[0429] Output: Generated recipe information.

[0430] Specific operation: The server sends information about tomatoes, cheese, and basil, as well as the user's stress level, to the generative AI, which then generates a recipe for "Tomato and Basil Bruschetta."

[0431] Step 5:

[0432] Image generation

[0433] Input: Recipe information from a generative AI.

[0434] Processing: The server sends the recipe information to the image generation means (such as DALL-E), which generates an image based on the recipe.

[0435] Output: The generated food image.

[0436] Specific operation: The server sends the recipe information for "Tomato and Basil Bruschetta" received from the generative AI to DALL-E, and generates an image of the bruschetta.

[0437] Step 6:

[0438] Submit recipes and images

[0439] Input: Generated recipe information and images.

[0440] Processing: The server sends the generated recipe and image to the user device.

[0441] Output: Recipe and image displayed on user's device.

[0442] Specific operation: The server sends the generated recipe and image of "Tomato and Basil Bruschetta" to the user's smartphone or tablet, and the user confirms it.

[0443] Step 7:

[0444] Regular checks and purchases

[0445] Input: Ingredient information stored in the database.

[0446] Processing: The server periodically checks the ingredient information in the database using a scheduler and generates a list of missing items.

[0447] Output: A list of missing objects.

[0448] Specific operation: At a set time each week, the server checks the information about the objects in the refrigerator, requests a week's worth of menus from the generative AI, and creates a list of missing objects.

[0449] Step 8:

[0450] Buy Online

[0451] Input: A list of missing objects.

[0452] Processing: The server sends the list of missing objects to the online sales device, which automatically purchases the required objects.

[0453] Output: Purchased object information.

[0454] Specific operation: The server uses an online sales API to automatically order the missing items (e.g., 1 liter of milk, 6 eggs) and complete the purchase.

[0455] (Application example 2)

[0456] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0457] The present invention aims to provide a system that reduces the effort required for managing items in a refrigerator and provides optimal cooking methods according to the user's emotional state. Another objective is to make users' lives more convenient by automatically detecting shortages of ingredients and providing a mechanism for purchasing necessary items online.

[0458] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a detection means for detecting objects in the refrigeration device, a transmission means for transmitting object information detected by the detection means to the processing device, a generation means for the processing device to generate a cooking method based on the object information, a means for the generation means to customize the cooking method based on user emotion information, an image generation means for generating an image of the generated cooking method, and a means for transmitting the generated cooking method and image to a user terminal. This automates the management of objects in the refrigeration device, makes it possible to suggest optimal cooking methods based on the user's emotions, and further enables efficient use of the refrigeration device by automatically purchasing ingredients that are running low, without the user having to perform any cumbersome operations.

[0459] A "refrigeration unit" is a box-shaped electrical device used to keep food and beverages at low temperatures.

[0460] "Object" refers to the specific food or beverage stored within the refrigeration unit.

[0461] "Detection means" refers to devices and techniques for identifying and obtaining information about objects within the refrigeration unit.

[0462] "Transmitting means" refers to a device or technology for transmitting object information acquired by the detecting means to a processing device.

[0463] The "processing device" is a computer device that processes the received object information and generates cooking methods and recipes.

[0464] "Generation means" refers to technology or devices for generating optimal cooking methods and recipes based on object information and user emotional information.

[0465] "Emotion information" is information that represents the emotional state of the user, obtained by analyzing the user's facial expression or voice.

[0466] "Means for customizing" refers to a technology or device in which the generating means changes the cooking method based on the user's emotional information.

[0467] "Image generation means" refers to technology or equipment for generating an image of a dish based on the generated cooking method.

[0468] A "user terminal" is a device for displaying the generated recipes and images, and includes a smartphone, a computer, etc.

[0469] A "generative AI model" is an artificial intelligence model used to generate cooking instructions and images from input data.

[0470] A "prompt sentence" is an input text provided to a generative AI model, and is a sentence that indicates the direction of the content to be generated.

[0471] "Online sales device" refers to a system or device for purchasing missing items via the Internet.

[0472] The present invention is a system that manages objects in a refrigeration device and suggests optimal cooking methods based on the user's emotional information. This system includes multiple steps, such as object detection, information processing, emotion recognition, recipe generation, and image generation. Specific system embodiments are described below.

[0473] Hardware and software used

[0474] Refrigeration unit: An electrical device used to preserve food and beverages.

[0475] Detection methods: Image recognition technology (e.g., OpenCV) and radio frequency identification technology (e.g., RFID tags).

[0476] Transmission method: Data communication using Wi-Fi module.

[0477] Processing unit: A high-performance computer or cloud server.

[0478] Generation method: Cooking instructions are generated using AI technology (e.g., OpenAI GPT-3).

[0479] Emotion Recognition Engine: Facial expression recognition technology (e.g., facial_emotion_recognition) and voice analysis technology.

[0480] Image generation method: AI image generation technology (e.g., DALL-E).

[0481] User devices: smartphones and computers.

[0482] System Operation

[0483] 1. Object detection:

[0484] Objects inside the refrigerator are detected using image recognition and radio frequency identification technology.

[0485] The detected object information is transmitted from the sensor to a processing unit.

[0486] 2. Processing of information:

[0487] The processing device (server) stores the received object information in a database.

[0488] The system periodically checks the information about items in the refrigerator, generates a cooking plan for a certain period of time, and creates a list of items that are missing.

[0489] 3. User emotion recognition:

[0490] The emotion recognition engine analyzes the user's facial expressions and voice to recognize the user's emotional information.

[0491] For example, if the user is feeling stressed, that information is sent to the server.

[0492] 4. Generate the recipe:

[0493] The server generates recipes using a generative AI model based on emotional and object information.

[0494] An example prompt is, "The refrigerator contains the following ingredients: tomatoes, cheese, and basil. The user is feeling stressed. Please suggest a recipe."

[0495] 5. Image Generation:

[0496] Based on the generated recipe information, AI image generation technology is used to generate images of the dish. For example, DALL-E is used to generate an image of "Tomato and Basil Bruschetta."

[0497] 6. Transmission and Display of Information:

[0498] The server transmits the generated recipe and image to the user terminal.

[0499] Suggested recipes and images are displayed on the user's device (smartphone or computer), which can be used as a reference for cooking.

[0500] Specific examples

[0501] If the refrigerator contains tomatoes, cheese, and basil, it is recognized that the user is seeking relaxation.

[0502] The server sends prompts to the generative AI model based on object information and emotion information, generating the optimal recipe: "Tomato and Basil Bruschetta."

[0503] Based on the recipe generated by the generative AI model, DALL-E generates an image of "Tomato and Basil Bruschetta."

[0504] This information is sent to the user's terminal, and the user can cook while looking at the displayed recipe and images.

[0505] In this way, the present invention significantly improves user convenience through the management of objects in a refrigerator, the suggestion of cooking methods based on the user's emotional information, and the automatic purchase of ingredients that are in short supply.

[0506] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0507] Step 1:

[0508] The server detects objects in the refrigerator. Sensors in the refrigerator use image recognition technology or radio frequency identification technology (RFID) to detect objects (e.g., tomatoes, cheese, basil, etc.). The detected object information is sent to the server. The input is the image or data from the RFID tag, and the output is the object information.

[0509] Step 2:

[0510] The server stores the received object information in a database. The object information is sent to the server, which records the information in the database. The database contains the object type, quantity, and detection date and time. The input is the object information, and the output is the updated database.

[0511] Step 3:

[0512] The server starts an emotion recognition engine to recognize the user's emotional information. The emotion recognition engine analyzes facial expression images and voice data from the user's device to identify the user's emotional state (e.g., stress, relaxation, etc.). The input is facial expression images and voice data, and the output is emotional information.

[0513] Step 4:

[0514] The server generates a recipe using a generative AI model based on object information and emotional information. The server generates a prompt sentence and inputs it into the generative AI model to suggest the optimal cooking method. For example, a prompt sentence could be "The following ingredients are in the refrigerator: tomatoes, cheese, and basil. The user is feeling stressed. Please suggest a dish." The input is object information and emotional information, and the output is the generated recipe.

[0515] Step 5:

[0516] The server sends the generated recipe information to an image generation engine to generate a food image. Based on the generated recipe, an AI image generation technology such as DALL-E is used to generate a corresponding food image (e.g., "Tomato and Basil Bruschetta"). The input is the recipe information, and the output is the food image.

[0517] Step 6:

[0518] The server sends the generated recipe and image to the user's device. The generated cooking method and its image are then sent to the user's device, where the user can check the suggested recipe and image on their smartphone or computer. The input is the recipe information and image, and the output is the display on the user's device.

[0519] Step 7:

[0520] The server periodically checks the object information in the refrigeration unit. For example, daily or weekly, the server updates the object information in the database and identifies any missing objects. The input is the existing object information, and the output is a list of missing objects.

[0521] Step 8:

[0522] The server generates a cooking plan for one week and sends a list of missing items to the online sales device. A generative AI model is used to generate recipes for one week, list missing items, and automatically send order information to the online sales device. The input is ingredient information and the number of recipes required, and the output is a list of missing items and order information.

[0523] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0524] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0525] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0526] [Second embodiment]

[0527] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0528] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0529] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0530] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0531] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0532] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0533] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0534] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0535] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0536] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0537] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0538] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0539] This invention is a system that automates the management of food and beverages stored in a refrigerator and suggests effective cooking methods. The system detects objects in the refrigerator and generates cooking instructions and images based on that information, which are then provided to the user. It also has a function that periodically checks the information about the objects in the refrigerator and automatically purchases any missing items online.

[0540] (Program overview and processing explanation)

[0541] Ingredient detection and data generation

[0542] 1. A sensor scans the object inside the refrigerator.

[0543] The server installs multiple sensors inside the refrigeration unit and uses image recognition technology and RFID tags to detect food and beverages.

[0544] For example, the sensor detects two tomatoes, 50g of cheese, and a small amount of basil.

[0545] 2. The server receives object information from the sensor.

[0546] The server receives object information detected by the sensor in real time.

[0547] Examples of received data include {"Object": [{"Name": "Tomato", "Quantity": "2"}, {"Name": "Cheese", "Quantity": "50g"}, {"Name": "Basil", "Quantity": "Small"}]}.

[0548] 3. The server stores the received object information in a database.

[0549] The server stores the received object information in a database and manages it as the latest inventory information.

[0550] Recipe generation and image creation

[0551] 4. The server sends a recipe generation request to the generative AI based on the object information.

[0552] The server sends a request to the generative AI to generate a recipe based on information about the objects inside the refrigerator.

[0553] For example, the server generates a request containing tomato, cheese, basil, and bread.

[0554] 5. A generative AI receives the request and generates a recipe based on the object.

[0555] The generative AI generates the optimal recipe based on the requested object.

[0556] For example, the generated recipe is "Tomato and Basil Bruschetta."

[0557] 6. The server sends the generated recipe to DALL-E and generates an image of the finished dish.

[0558] The server sends an image generation request to DALL-E based on the recipe information received from the generative AI.

[0559] For example, send a request to "generate an image of bruschetta."

[0560] 7. The server sends the generated recipe and image to the user device.

[0561] The server transmits the generated recipe text and image file to the user terminal.

[0562] The content provided includes a "Tomato and Basil Bruschetta Recipe" and an image of it.

[0563] Regular menu planning and online purchasing

[0564] 8. The server periodically checks the information of objects in the refrigerator.

[0565] The server retrieves and checks information about objects in the refrigerator from the database at a specific time each week.

[0566] 9. The server requests a week's worth of menus from the generative AI.

[0567] To generate a week's worth of menus, the server requests information about all objects and the number of dishes required from the generative AI.

[0568] 10. Generative AI generates a week's worth of menus and creates a list of missing items.

[0569] The generative AI creates a weekly menu and identifies missing items and generates a list.

[0570] 11. The server sends the list of missing objects to the online sales device and purchases the required objects.

[0571] The server sends a list of missing objects to an online sales device, which automatically purchases the objects.

[0572] User Examples

[0573] For example, if tomatoes, cheese, basil, and bread are stored in a refrigerator, the user uses the system in the following steps:

[0574] 1. The user starts using the refrigeration unit.

[0575] A user places an object in the refrigerator and a sensor detects this.

[0576] 2. The server receives the object information and stores it in a database.

[0577] The server receives data from the sensors and registers the object information in a database.

[0578] 3. The server requests the generative AI to generate a recipe.

[0579] The server requests the generative AI to generate a recipe based on the object information, and the AI ​​generates a recipe for "Tomato and Basil Bruschetta."

[0580] 4. The server requests image generation from DALL-E.

[0581] The server sends the generated recipe to DALL-E, which generates an image of bruschetta.

[0582] 5. The server sends the generated recipe and image to the user device.

[0583] The recipe and images are displayed on the user's terminal, and the user cooks while looking at them.

[0584] 6. The server periodically checks the object information and creates a menu for the week.

[0585] The server periodically checks the object information and requests a week's worth of menus from the generative AI.

[0586] 7. Automatically purchase absent objects online.

[0587] The server transmits a list of missing items to the online sales device, which automatically carries out the purchase procedure.

[0588] In this way, the present invention automates ingredient management, recipe suggestions, menu creation, and online purchasing, supporting users' busy lives.

[0589] The processing flow will be explained below.

[0590] Step 1:

[0591] The server activates sensors to detect objects inside the refrigerator, which use image recognition technology and RFID tags to detect food and beverages inside the refrigerator.

[0592] Step 2:

[0593] The sensor detects the objects in the refrigerator and sends them to the server. For example, the sensor detects two tomatoes, 50g of cheese, and a small amount of basil.

[0594] Step 3:

[0595] The server stores the object information received from the sensor in a database, including the object name and quantity.

[0596] Step 4:

[0597] The server retrieves the latest object information from the database and sends a recipe generation request to the generative AI, which includes information on all ingredients in the refrigerator.

[0598] Step 5:

[0599] The generative AI receives requests from the server and generates recipes based on the object information. For example, it generates a recipe for "Tomato and Basil Bruschetta."

[0600] Step 6:

[0601] The server sends the generated recipe information to DALL-E, which is an image generation means, and sends a request to generate an image of the finished dish.

[0602] Step 7:

[0603] DALL-E generates an image based on the recipe and sends it back to the server. For example, it generates an "image of bruschetta."

[0604] Step 8:

[0605] The server sends the generated recipe and image to the user's device, where the user can view the proposed recipe and image.

[0606] Step 9:

[0607] The server periodically checks the information of objects in the refrigerator once a week. This check is performed automatically by the scheduler.

[0608] Step 10:

[0609] The server retrieves all object information from the database and sends a request to the generative AI to generate a week's worth of menus.

[0610] Step 11:

[0611] A generative AI receives the request and generates a week's worth of cooking plans and a list of ingredients needed, including any missing ingredients.

[0612] Step 12:

[0613] The server transmits the generated list of absent objects to an online sales device, which automatically purchases the missing objects.

[0614] Step 13:

[0615] Users can pick up ingredients purchased online at a designated location. By receiving any missing ingredients, users can prepare a week's worth of meals as planned.

[0616] Example 1

[0617] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0618] In today's busy world, users spend a lot of time managing ingredients and planning their cooking. In addition, improper management of food and beverages stored in refrigerators can lead to food waste. Furthermore, it can be cumbersome for users to regularly purchase ingredients needed for cooking, and poor management can make cooking difficult.

[0619] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0620] In this invention, the server includes a detection means for detecting objects in the refrigeration unit, a transmission means for transmitting object information detected by the detection means to a processing device, and a generation means for the processing device to generate a cooking method based on the object information. This enables automatic ingredient management and optimal cooking method suggestions. The server also includes an image generation means for generating an image of the generated cooking method, a means for transmitting the generated cooking method and the image to a user terminal, a means for storing the object information received by the processing device in a database, a means for the processing device to request recipe generation from a generative artificial intelligence model, a means for the generative artificial intelligence model to receive the request and generate a recipe based on the object, and a means for the processing device to transmit the generated recipe to an image generation model and generate an image of the finished dish. This allows users to consistently automate everything from daily ingredient management to cooking method suggestions and food image confirmation. It also reduces food waste and realizes efficient ingredient purchasing.

[0621] The "detection means" is a device installed to detect objects inside the refrigeration device, and may use image recognition technology or radio frequency identification technology.

[0622] The "transmitting means" is a device or system for transmitting object information detected by the detecting means to the processing device.

[0623] The "generation means" is a device or system that allows the processing device to generate a cooking method based on object information.

[0624] "Image generation means" refers to a device or system for generating an image of the generated recipe.

[0625] The "means for transmitting to the user terminal" refers to a device or system for transmitting the generated recipes and images to the terminal used by the user.

[0626] A "processing device" is a device or system for processing object information received from a sensor and storing it in a database.

[0627] A "generative artificial intelligence model" is an artificial intelligence model that generates recipes in natural language based on input prompts.

[0628] An "image generation model" is an artificial intelligence model for generating images based on input text information.

[0629] A "database" is a data management system for storing and managing received object information.

[0630] A "refrigeration device" is a device used to store food and beverages at a constant low temperature.

[0631] "Object information" is data about the names and quantities of food and beverages stored in the refrigerator.

[0632] A "recipe" is information that describes the steps and ingredients for cooking a particular dish.

[0633] This invention is a system that automates the management of food and beverages stored in a refrigerator and suggests effective cooking methods. The system detects objects in the refrigerator and generates cooking instructions and images based on that information, which are then provided to the user. It also has a function that periodically checks the information about the objects in the refrigerator and automatically purchases any missing items online.

[0634] This system mainly uses the following hardware and software:

[0635] 1. Sensors: Image recognition sensors and RFID readers are used to detect objects inside the refrigerator.

[0636] 2. Server: Receives object information from sensors, stores it in a database, and requests recipe and image generation from generative artificial intelligence models and image generation models.

[0637] 3. Generative AI models: For example, using natural language generation models such as GPT-3 to generate recipes based on ingredients.

[0638] 4. Image generation model: For example, a text-to-image generation model such as DALL-E is used to generate images of the generated recipe.

[0639] 5. User device: For example, the generated recipes and images are provided to the user using a smartphone or tablet.

[0640] As a concrete example, consider a case where a user stores two tomatoes, 50g of cheese, a little basil, and bread in a refrigerator. The system operates as follows:

[0641] First, sensors inside the refrigerator detect these ingredients and send object information to a server, which stores this information in a database and manages the overall inventory.

[0642] The server then sends a recipe generation request to the generative artificial intelligence model, such as a prompt like "Generate a recipe using tomatoes, cheese, basil, and bread."

[0643] The generative AI model receives this request and generates a recipe called "Tomato and Basil Bruschetta." The server then sends this generated recipe to the image generation model, requesting it to "generate an image of the bruschetta."

[0644] The image generation model receives the request and generates an image of the bruschetta. The server then sends the generated recipe and image to the user's device. The user then cooks the dish while viewing the recipe and image on the device.

[0645] Furthermore, the server can periodically check the information about the objects in the refrigerator and ask the generative AI model to create a week's worth of menus. Once a list of missing objects is created, the server sends the list to the online sales device, which then automatically carries out the purchasing process.

[0646] In this way, the present invention reduces the burden on the user and enables efficient and effective food ingredient management and cooking.

[0647] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0648] Step 1:

[0649] Sensors scan objects inside the refrigerator

[0650] Input: Current state of ingredients in the refrigerator

[0651] How it works: Image recognition sensors and RFID readers installed inside the refrigerator detect ingredients like tomatoes, cheese, and basil. Each sensor captures image data and RFID tag information for each ingredient.

[0652] Output: A list of ingredients and their amounts (e.g., "2 tomatoes, 50g cheese, a little basil")

[0653] Step 2:

[0654] The server receives object information from the sensor

[0655] Input: A list of ingredient names and quantities sent from the sensor

[0656] Specific operation: The server receives object information sent from the sensor in real time. The received data format is, for example, JSON: {"Object": [{"Name": "Tomato", "Quantity": "2"}, {"Name": "Cheese", "Quantity": "50g"}, {"Name": "Basil", "Quantity": "Small"}]}.

[0657] Output: Received object information data

[0658] Step 3:

[0659] The server stores the received object information in a database.

[0660] Input: Received object information data

[0661] Specific operation: The server analyzes the object information and stores it in a database. The database keeps up-to-date information on the types and quantities of ingredients. In this example, the database stores "2 tomatoes, 50g of cheese, and a little basil."

[0662] Output: Latest object information stored in the database

[0663] Step 4:

[0664] The server sends a recipe generation request to the generative AI based on the object information.

[0665] Input: The latest object information stored in the database

[0666] Specific operation: The server retrieves object information from the database and sends it to the generative AI (e.g., GPT-3) as a recipe generation request. The request is sent in the form of a prompt: "Please generate a recipe using tomatoes, cheese, basil, and bread."

[0667] Output: The prompt sent to the generative AI

[0668] Step 5:

[0669] A generative AI receives the request and generates a recipe based on the object.

[0670] Input: Prompt sent from the server

[0671] How it works: A generative AI (e.g., GPT-3) analyzes the prompt and generates an optimal recipe based on the input object information. For example, the generated recipe might be "Tomato and Basil Bruschetta."

[0672] Output: Generated recipe (text format)

[0673] Step 6:

[0674] The server sends the generated recipe to the image generation model, which generates an image of the finished dish.

[0675] Input: Recipe generated by generative AI

[0676] Specific operation: The server obtains the generated recipe information and sends an image generation request to the image generation model (e.g., DALL-E). The request is sent in the form of, for example, "Please generate an image of bruschetta."

[0677] Output: Generated food images

[0678] Step 7:

[0679] The server sends the generated recipe and image to the user's device.

[0680] Input: Generated recipes and food images

[0681] Specific operation: The server sends the generated recipe text and image file to the user's device so that the user can check it. For example, the user's smartphone will display "Tomato and Basil Bruschetta Recipe" and its image.

[0682] Output: Recipe and image displayed on the user's device

[0683] Step 8:

[0684] The server periodically checks the information about objects in the refrigerator.

[0685] Input: Object information stored in the database

[0686] Specific operation: The server periodically (e.g., every Monday at 9:00 AM) accesses the database and checks the information about objects in the refrigerator.

[0687] Output: Information about the most recent object in the refrigerator that was checked

[0688] Step 9:

[0689] The server requests a week's worth of menus from the generative AI

[0690] Input: Information about the object in the most recent refrigerator that was checked

[0691] Specific operation: To generate a week's worth of menus, the server requests all object information in the database and the number of dishes required from the generative AI. The request is sent in the form of a prompt, such as "Please generate a seven-day menu."

[0692] Output: The prompt sent to the generative AI

[0693] Step 10:

[0694] Generative AI generates a week's worth of meals and creates a list of missing items

[0695] Input: Prompt sent from the server

[0696] Specific operation: A generative AI (e.g., GPT-3) analyzes the prompts and generates a weekly menu, identifying and listing any missing items. For example, a generated menu and a "list of missing items" are created.

[0697] Output: Generated weekly menu and list of missing items

[0698] Step 11:

[0699] The server sends the list of missing objects to the online sales device and purchases the required objects.

[0700] Input: Generated missing object list

[0701] Specific operation: The server sends a list of missing items to the online sales device, which then automatically processes the purchase. For example, a list of "2 tomatoes, 100g of cheese" is sent, and an online order is placed.

[0702] Output: Required objects purchased

[0703] In this way, the entire system flow is completed, allowing users to smoothly carry out a series of processes, from managing ingredients and suggesting cooking methods to automatically purchasing the necessary ingredients.

[0704] (Application example 1)

[0705] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0706] Managing food and beverages in a refrigerator requires manual checking and replenishing, which is a significant time burden for users. Furthermore, since no cooking method suggestions are provided, it is difficult to effectively utilize the ingredients in the refrigerator. Furthermore, there is no system that automatically replenishes ingredients when they run out, so users must manually order them. To solve these issues, there is a need for a system that can automatically manage food and beverages, suggest effective cooking methods, and automatically replenish ingredients when they run out.

[0707] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0708] In this invention, the server includes a detection means for detecting objects in the refrigeration unit, a transmission means for transmitting object information detected by the detection means to a processing device, a generation means for the processing device to generate a cooking method based on the object information, an image generation means for generating an image of the generated cooking method, a means for transmitting the generated cooking method and the image to a user terminal, and a replenishment means for the processing device to generate order information for replenishing ingredients based on the object information and automatically order ingredients through an online service. This enables automated management of food and beverages in the refrigeration unit, suggestions for effective cooking methods, and automatic replenishment of ingredients when they run out.

[0709] A "refrigeration unit" is a device designed to maintain a low internal temperature and is used to preserve food, beverages, etc.

[0710] A "system" is a whole made up of multiple elements that are interrelated and combined to achieve a certain purpose.

[0711] "Detection means" means a device or function, including sensor technology, for accurately identifying and detecting the presence and type of object within the refrigeration unit.

[0712] "Transmitting means" refers to a device or function that includes communication technology for transmitting object information collected by the detecting means to the processing device.

[0713] The "generation means" is a device or function that includes an algorithm or artificial intelligence for automatically creating a cooking method based on object information.

[0714] "Image generation means" refers to a device or function including an artificial intelligence or image generation engine for creating visual food images based on the generated cooking instructions.

[0715] A "user terminal" is a device such as a smartphone or computer that a user uses to receive, check, or operate information provided by the system.

[0716] The "replenishment means" is a function or system for automatically ordering and replenishing missing food or beverages based on object information.

[0717] The "processing device" is a computer or control unit that receives and analyzes the object information transmitted from the detection means and controls the entire system.

[0718] The system to realize this application example utilizes various sensors, processing devices, and communication means built into the refrigeration equipment to automatically manage food and beverages, suggest effective cooking methods, and even automatically replenish ingredients when they run out.

[0719] The server includes a detection means that uses image recognition technology or radio frequency identification technology to detect objects in the refrigeration device. The detection means collects object information using, for example, a camera or an RFID reader. The collected object information is transmitted to the server via a transmission means.

[0720] The server receives the transmitted object information with a processing device and generates an optimal cooking method based on the object information using a generation means, such as an OpenAI generative AI model. Based on the generated cooking method, an image generation engine such as DALL-E is used to generate an image of the cooking method.

[0721] The image of the cooking method and the recipe generated by the image generating means are transmitted to a user terminal via a transmitting means, which may be a smartphone or tablet, and the user can check the cooking method and the image through the user terminal.

[0722] Furthermore, the server periodically checks the information on the items in the refrigerator and generates a cooking plan for the week. Based on the generated cooking plan, it creates a list of items that are in short supply and automatically orders them online. This eliminates the need for users to manually check and order.

[0723] For example, if tomatoes, cheese, and basil are stored in the refrigerator, the server will request a recipe from the generative AI model based on this information. The generative AI model will suggest "Tomato and Basil Bruschetta," and DALL-E will generate an image of it. Furthermore, if the bread needed for the bruschetta is in short supply, it will automatically be ordered from an online service.

[0724] Here are some examples of prompts for generative AI models:

[0725] If you have the following ingredients in your fridge, please suggest some possible dishes.

[0726] object:

[0727] Name: Tomato

[0728] Quantity: 2 pieces

[0729] Name: Cheese

[0730] Amount: 50g

[0731] Name: Basil

[0732] Quantity: small quantity

[0733] This system automates the management of food and beverages in refrigeration units, allowing users to cook efficiently while saving time and effort.

[0734] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0735] Step 1:

[0736] Detect objects inside a refrigeration unit.

[0737] How it works: The server collects data on food and beverages using cameras and RFID readers installed inside the refrigeration units.

[0738] Input: Image data or RFID tag data of objects in the refrigeration unit.

[0739] Data processing: Identifying the type and quantity of objects using image recognition and RFID technology.

[0740] Output: Identified object information (e.g., "2 tomatoes", "50g cheese", "small amount of basil").

[0741] Step 2:

[0742] The detected object information is transmitted to a processing device.

[0743] Specific operations: The server transmits the object information obtained by the detection means to the processing device.

[0744] Input: Identified object information.

[0745] Data calculation: Formatting object information according to the protocol.

[0746] Output: Formatted object information data.

[0747] Step 3:

[0748] Cooking methods are generated based on object information.

[0749] Specific operation: The server sends a cooking method generation request to the generative AI model based on the object information.

[0750] Input: Formatted object information data.

[0751] Data calculation: The generative AI model performs natural language processing and database search to generate optimal recipes based on object information.

[0752] Output: The generated recipe (e.g., "Tomato and Basil Bruschetta").

[0753] Step 4:

[0754] Generate an image of the cooking method.

[0755] Specific operation: The server sends the generated text information of the cooking method to the image generation engine to generate a visual image of the dish.

[0756] Input: The generated text information of the cooking instructions.

[0757] Data calculation: An AI model processes and generates images based on text information.

[0758] Output: The generated food image (e.g. "Image of bruschetta").

[0759] Step 5:

[0760] The generated recipe and image are sent to the user terminal.

[0761] Specific operation: The server sends cooking instructions and images to the user's smartphone or tablet.

[0762] Input: Generated recipes and images.

[0763] Data calculation: Processes data in a format suitable for the user terminal and sends it.

[0764] Output: Cooking instructions and images displayed on the user's device.

[0765] Step 6:

[0766] Regularly check the contents of the refrigerator.

[0767] Specific operation: The server rechecks the information about objects in the refrigerator at set intervals.

[0768] Input: Current object information data.

[0769] Data calculation: Compare with existing information in the database and update new object information.

[0770] Output: Updated object information data.

[0771] Step 7:

[0772] Generate cooking plans for a certain period of time.

[0773] Specific operation: The server sends a request to the generative AI model to generate a cooking plan for a certain period (e.g., one week) based on the collected object information.

[0774] Input: Updated object information data.

[0775] Data calculation: A generative AI model generates a cooking plan that combines multiple recipes based on object information.

[0776] Output: Generated cooking plan (e.g., recipes for a week).

[0777] Step 8:

[0778] Automatically order missing items online.

[0779] Specific operation: The server creates a list of ingredients that are missing based on the generated cooking plan and places an order with the online sales service.

[0780] Input: Generated cooking plan, missing object list.

[0781] Data processing: Generates order information and processes orders through the API of online services.

[0782] Output: Online order confirmation information.

[0783] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0784] This invention relates to a system that manages objects stored in a refrigerator and suggests optimal cooking methods based on the user's emotions. The system detects objects stored in the refrigerator, generates optimal recipes based on that information, and customizes cooking methods by taking the user's emotions into account using an emotion engine. It also has the functionality to automate object management and periodic online purchasing of items that are in short supply.

[0785] (Program overview and processing explanation)

[0786] Ingredient detection and data generation

[0787] 1. A sensor scans the object inside the refrigerator.

[0788] The server activates sensors installed inside the refrigerator that use image recognition technology and RFID tags to detect objects, such as two tomatoes, 50g of cheese, and a small amount of basil.

[0789] 2. The server receives object information from the sensor.

[0790] The sensor sends the detected object information to the server, which stores it in a database. The stored information includes the object name and quantity.

[0791] Recipe Generation and Emotion Recognition

[0792] 3. The server launches an emotion engine that recognizes the user's emotions.

[0793] The emotion engine analyzes user input, voice, and facial expressions to recognize the user's emotional state. For example, if the user is feeling stressed, it will suggest recipes to help them relax.

[0794] 4. The server sends the object information and the user's emotion information to the generative AI.

[0795] The server sends the object information obtained from the database and the user's emotion information recognized by the emotion engine to the generative AI.

[0796] 5. Generative AI generates recipes based on object information and emotional information.

[0797] The generative AI generates optimal recipes based on object information and the user's emotional information. For example, if the user is feeling stressed, it generates a relaxing recipe such as "Tomato and Basil Bruschetta."

[0798] Image generation and information transmission

[0799] 6. The server sends the generated recipe to DALL-E, which is the image generation means.

[0800] The server sends the recipe information received from the generative AI to DALL-E, which then generates an image of the dish based on that recipe.

[0801] 7. DALL-E generates an image based on the recipe and sends it back to the server.

[0802] For example, an "image of bruschetta" is generated and sent back to the server.

[0803] 8. The server sends the generated recipe and image to the user device.

[0804] The server sends the generated recipe text and image files to the user's terminal, where the user can view the proposed recipe and images.

[0805] Regular menu planning and online purchasing

[0806] 9. The server periodically checks the information of objects in the refrigerator once a week.

[0807] The server checks the database for information about objects in the refrigerator at a specific time each week according to a scheduler.

[0808] 10. The server requests a week's worth of menus from the generative AI.

[0809] To generate a week's worth of menus, the server requests information about all objects and the number of dishes required from the generative AI.

[0810] 11. Generative AI generates a week's worth of menus and creates a list of missing items.

[0811] The generative AI creates a weekly menu and identifies missing items and generates a list.

[0812] 12. The server sends the list of missing objects to the online sales device and purchases the required objects.

[0813] The server sends a list of missing objects to an online sales device, which automatically purchases the objects.

[0814] User Examples

[0815] For example, if tomatoes, cheese, basil, and bread are stored in a refrigerator, the user uses the system in the following steps:

[0816] 1. The user starts using the refrigeration unit.

[0817] A user places an object in the refrigerator and a sensor detects this.

[0818] 2. The server receives the object information and stores it in a database.

[0819] The server receives data from the sensors and registers the object information in a database.

[0820] 3. The emotion engine detects the user's emotions.

[0821] The emotion engine analyzes the user's voice and facial expressions and recognizes that the user is seeking relaxation.

[0822] 4. The server requests the generative AI to generate a recipe.

[0823] The server requests a generative AI to generate a recipe based on object information and emotional information, and the AI ​​generates a recipe for "Tomato and Basil Bruschetta."

[0824] 5. The server requests DALL-E to generate an image.

[0825] The server sends the generated recipe to DALL-E, which generates an image of bruschetta.

[0826] 6. The server sends the generated recipe and image to the user device.

[0827] The recipe and images are displayed on the user's terminal, and the user cooks while looking at them.

[0828] 7. The server periodically checks the object information and creates a menu for the week.

[0829] The server periodically checks the object information and requests a week's worth of menus from the generative AI.

[0830] 8. Automatically purchase absent objects online.

[0831] The server transmits a list of missing items to the online sales device, which automatically carries out the purchase procedure.

[0832] In this way, the present invention automates ingredient management, recipe suggestions, menu planning, emotion-based recipe optimization, and online purchasing to support users' busy lives.

[0833] The processing flow will be explained below.

[0834] Step 1:

[0835] The server activates the sensor to detect the object inside the refrigerator. The sensor uses image recognition technology and RFID tags to detect the object (food or drink) inside the refrigerator.

[0836] Step 2:

[0837] The sensor detects information about objects in the refrigerator and sends it to the server. For example, the sensor detects "2 tomatoes," "50g of cheese," and "a little basil."

[0838] Step 3:

[0839] The server stores the object information received from the sensor in a database, including the object name and quantity.

[0840] Step 4:

[0841] The server activates an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's input, voice, and facial expressions to recognize the user's emotional state.

[0842] Step 5:

[0843] The emotion engine analyzes the user's emotions and generates emotion information, for example, recognizing that the user is feeling stressed.

[0844] Step 6:

[0845] The server sends a recipe generation request to the generative AI based on the object information obtained from the database and the emotion information generated by the emotion engine.

[0846] Step 7:

[0847] Generative AI generates optimal recipes based on object and emotional information. For example, it generates a recipe for "Tomato and Basil Bruschetta" to help users relax.

[0848] Step 8:

[0849] The server sends the generated recipe to DALL-E, an image generation means, which generates an image of the dish based on the recipe.

[0850] Step 9:

[0851] DALL-E generates an image of the dish based on the recipe and sends it back to the server. For example, it generates an image of bruschetta.

[0852] Step 10:

[0853] The server sends the generated recipe and image to the user's terminal, where the user can check the proposed recipe and image.

[0854] Step 11:

[0855] The server sets a scheduler to periodically check the object information in the refrigerator, retrieving and checking the object information from the database at a specific time every week.

[0856] Step 12:

[0857] The server retrieves all object information from the database and sends a request to the generative AI to generate a week's worth of menus.

[0858] Step 13:

[0859] Generative AI generates a weekly menu and creates a list of missing items. For example, it creates a menu that includes menus for Monday through Sunday.

[0860] Step 14:

[0861] The server transmits a list of missing objects to an online sales device, which automatically purchases the missing objects.

[0862] Step 15:

[0863] Users can pick up items they have purchased online at a specified location. By having any missing ingredients delivered to them, users can plan their meals for the week.

[0864] Example 2

[0865] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0866] In today's busy lifestyles, managing ingredients and suggesting effective cooking methods requires a great deal of effort. It is also difficult to find appropriate cooking methods that reduce stress and fatigue. Therefore, there is a need for a system that can effectively manage items in a refrigerator, suggest optimal cooking methods based on the user's emotions, and automatically purchase ingredients when they are in short supply.

[0867] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0868] In this invention, the server includes a detection means for detecting objects in the refrigeration unit, a transmission means for transmitting object information detected by the detection means to the processing device, a generation means for generating a cooking method based on the object information, an emotion recognition means for recognizing a user's emotion, a generation means for generating a cooking method based on the object information and emotion information, an image generation means for generating an image of the generated cooking method, and a means for transmitting the generated cooking method and image to a user terminal. This makes it possible to efficiently manage objects in the refrigeration unit and suggest appropriate cooking methods based on the user's emotion, not only facilitating meal preparation but also improving the user's psychological satisfaction. It also enables automatic purchasing of ingredients that are in short supply, further improving convenience.

[0869] "Detection means" refers to a device or system for detecting objects within the refrigeration unit.

[0870] The "transmitting means" refers to a device or system for transmitting object information detected by the detecting means to the processing device.

[0871] The term "processing device" refers to a device or system that generates and manages recipes based on object information received from a transmitting means.

[0872] The "generation means" refers to a device or system for generating a cooking method based on object information and user emotion information.

[0873] "Emotion recognition means" refers to a device or system for recognizing the emotional state of a user.

[0874] "Image generation means" refers to a device or system for generating an image based on the generated recipe.

[0875] The term "user terminal" refers to an apparatus or device for displaying generated recipes and images to a user.

[0876] "Database" refers to a system for storing object information, user emotion information, and other related data.

[0877] "Online sales device" refers to a system that processes orders online to purchase missing objects.

[0878] "Scheduler" refers to a system that manages a schedule for periodically checking information about objects in a refrigeration device.

[0879] The present invention relates to a system that manages items stored in a refrigerator and suggests optimal cooking methods based on the user's emotions. This system utilizes multiple hardware and software components to efficiently and automatically manage ingredients and suggest cooking methods.

[0880] Hardware and Software Configuration

[0881] 1. Sensors: Sensors with image recognition technology and RFID tag readers are placed inside the refrigerator, allowing for accurate detection of objects inside the refrigerator.

[0882] 2. Server: The central part of the system, it centrally manages and processes data from various devices and software, such as sensors, emotion recognition engines, generative AI, and image generation methods.

[0883] 3. Emotion Recognition Engine: Uses software to analyze user input (voice, facial expressions, etc.) and recognize emotions. For example, determining stress or happiness from the user's tone of voice and facial expressions.

[0884] 4. Generative AI: Uses AI models to generate optimal recipes based on object and emotion information. For example, this applies to recipe suggestion systems that use natural language processing.

[0885] 5. Image generation means: Software for generating food images based on the generated recipe. This includes DALL-E and similar image generation models.

[0886] 6. User device: A device (smartphone, tablet, etc.) on which the user checks the recipe and generated images. It displays the data sent from the server.

[0887] 7. Database: Use a system to store object information, user emotion information, recipe information, etc.

[0888] 8. Online Sales Device: A system for automatically purchasing missing items. The purchasing process is carried out using the online store API.

[0889] Specific actions

[0890] 1. Ingredient detection:

[0891] When a user places new ingredients in the refrigerator, the sensor scans them and sends the detection results to the server. For example, if a tomato and cheese are newly added to the refrigerator, image recognition technology and an RFID tag reader will detect this.

[0892] 2. Data storage:

[0893] The server analyzes the received data and stores it in a database. For example, it might register "2 tomatoes, 50g of cheese."

[0894] 3. Emotion Recognition:

[0895] When a user speaks into the smartphone or faces the camera, the emotion recognition engine analyzes this and sends the user's emotional information to the server. For example, it may determine that the user is feeling stressed.

[0896] 4. Generate the recipe:

[0897] The server retrieves object and emotion information from the database and sends it to the generative AI, which then generates the optimal recipe based on the user's emotions. For example, it suggests "relaxing tomato and basil bruschetta."

[0898] 5. Image Generation:

[0899] The server transmits the generated recipe to the image generating means, which generates a related food image, for example, an image of "bruschetta."

[0900] 6. Submit your recipe and images:

[0901] The server sends the generated recipe and images to the user terminal, where the user can check the proposed recipe and images.

[0902] 7. Regular Checks and Purchases:

[0903] The server periodically checks the information of the items in the refrigerator and lists the items that are missing, and automatically purchases the missing items through the online sales device based on the list.

[0904] Specific examples

[0905] Example prompt sentence:

[0906] "What are some relaxing dishes that you would suggest to users when they are feeling stressed?"

[0907] "If I have tomatoes, cheese, and basil in my refrigerator, can you suggest a recipe that works best?"

[0908] In this way, the present invention provides a system that allows users to efficiently manage ingredients and enjoy optimal recipes that suit their own emotional state.

[0909] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0910] Step 1:

[0911] Ingredient detection

[0912] Input: User stores new ingredients in the refrigerator.

[0913] Processing: The server activates sensors (image recognition technology and RFID tag readers) inside the refrigeration unit to scan the object.

[0914] Output: Data of detected ingredients.

[0915] What it does: When a user adds 2 tomatoes, 50g of cheese, and a little basil to the refrigerator, the sensor detects this and sends the data (2 tomatoes, 50g of cheese, a little basil) to the server.

[0916] Step 2:

[0917] Data storage

[0918] Input: Ingredient data sent from the sensor.

[0919] Processing: The server receives the data from the sensors and stores it in a database.

[0920] Output: Ingredient information stored in a database.

[0921] Specific operation: The server analyzes the data received from the sensor (2 tomatoes, 50g of cheese, and a small amount of basil) and registers it in a database.

[0922] Step 3:

[0923] Emotion recognition

[0924] Input: User voice and facial expression data.

[0925] Processing: The server uses an emotion recognition engine to analyze the user's emotions.

[0926] Output: The perceived emotional state of the user.

[0927] Specific operation: When a user speaks into the smartphone or faces the camera, the emotion recognition engine analyzes this and determines that the user is feeling stressed.

[0928] Step 4:

[0929] Recipe Generation

[0930] Input: Ingredient information obtained from the database and emotion information from the emotion recognition engine.

[0931] Processing: The server sends the ingredient information and emotion information to the generative AI and requests it to generate an appropriate recipe.

[0932] Output: Generated recipe information.

[0933] Specific operation: The server sends information about tomatoes, cheese, and basil, as well as the user's stress level, to the generative AI, which then generates a recipe for "Tomato and Basil Bruschetta."

[0934] Step 5:

[0935] Image generation

[0936] Input: Recipe information from a generative AI.

[0937] Processing: The server sends the recipe information to the image generation means (such as DALL-E), which generates an image based on the recipe.

[0938] Output: The generated food image.

[0939] Specific operation: The server sends the recipe information for "Tomato and Basil Bruschetta" received from the generative AI to DALL-E, and generates an image of the bruschetta.

[0940] Step 6:

[0941] Submit recipes and images

[0942] Input: Generated recipe information and images.

[0943] Processing: The server sends the generated recipe and image to the user device.

[0944] Output: Recipe and image displayed on user's device.

[0945] Specific operation: The server sends the generated recipe and image of "Tomato and Basil Bruschetta" to the user's smartphone or tablet, and the user confirms it.

[0946] Step 7:

[0947] Regular checks and purchases

[0948] Input: Ingredient information stored in the database.

[0949] Processing: The server periodically checks the ingredient information in the database using a scheduler and generates a list of missing items.

[0950] Output: A list of missing objects.

[0951] Specific operation: At a set time each week, the server checks the information about the objects in the refrigerator, requests a week's worth of menus from the generative AI, and creates a list of missing objects.

[0952] Step 8:

[0953] Buy Online

[0954] Input: A list of missing objects.

[0955] Processing: The server sends the list of missing objects to the online sales device, which automatically purchases the required objects.

[0956] Output: Purchased object information.

[0957] Specific operation: The server uses an online sales API to automatically order the missing items (e.g., 1 liter of milk, 6 eggs) and complete the purchase.

[0958] (Application example 2)

[0959] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0960] The present invention aims to provide a system that reduces the effort required for managing items in a refrigerator and provides optimal cooking methods according to the user's emotional state. Another objective is to make users' lives more convenient by automatically detecting shortages of ingredients and providing a mechanism for purchasing necessary items online.

[0961] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a detection means for detecting objects in the refrigeration device, a transmission means for transmitting object information detected by the detection means to the processing device, a generation means for the processing device to generate a cooking method based on the object information, a means for the generation means to customize the cooking method based on user emotion information, an image generation means for generating an image of the generated cooking method, and a means for transmitting the generated cooking method and image to a user terminal. This automates the management of objects in the refrigeration device, makes it possible to suggest optimal cooking methods based on the user's emotions, and further enables efficient use of the refrigeration device by automatically purchasing ingredients that are running low, without the user having to perform any cumbersome operations.

[0962] A "refrigeration unit" is a box-shaped electrical device used to keep food and beverages at low temperatures.

[0963] "Object" refers to the specific food or beverage stored within the refrigeration unit.

[0964] "Detection means" refers to devices and techniques for identifying and obtaining information about objects within the refrigeration unit.

[0965] "Transmitting means" refers to a device or technology for transmitting object information acquired by the detecting means to a processing device.

[0966] The "processing device" is a computer device that processes the received object information and generates cooking methods and recipes.

[0967] "Generation means" refers to technology or devices for generating optimal cooking methods and recipes based on object information and user emotional information.

[0968] "Emotion information" is information that represents the emotional state of the user, obtained by analyzing the user's facial expression or voice.

[0969] "Means for customizing" refers to a technology or device in which the generating means changes the cooking method based on the user's emotional information.

[0970] "Image generation means" refers to technology or equipment for generating an image of a dish based on the generated cooking method.

[0971] A "user terminal" is a device for displaying the generated recipes and images, and includes a smartphone, a computer, etc.

[0972] A "generative AI model" is an artificial intelligence model used to generate cooking instructions and images from input data.

[0973] A "prompt sentence" is an input text provided to a generative AI model, and is a sentence that indicates the direction of the content to be generated.

[0974] "Online sales device" refers to a system or device for purchasing missing items via the Internet.

[0975] The present invention is a system that manages objects in a refrigeration device and suggests optimal cooking methods based on the user's emotional information. This system includes multiple steps, such as object detection, information processing, emotion recognition, recipe generation, and image generation. Specific system embodiments are described below.

[0976] Hardware and software used

[0977] Refrigeration unit: An electrical device used to preserve food and beverages.

[0978] Detection methods: Image recognition technology (e.g., OpenCV) and radio frequency identification technology (e.g., RFID tags).

[0979] Transmission method: Data communication using Wi-Fi module.

[0980] Processing unit: A high-performance computer or cloud server.

[0981] Generation method: Cooking instructions are generated using AI technology (e.g., OpenAI GPT-3).

[0982] Emotion Recognition Engine: Facial expression recognition technology (e.g., facial_emotion_recognition) and voice analysis technology.

[0983] Image generation method: AI image generation technology (e.g., DALL-E).

[0984] User devices: smartphones and computers.

[0985] System Operation

[0986] 1. Object detection:

[0987] Objects inside the refrigerator are detected using image recognition and radio frequency identification technology.

[0988] The detected object information is transmitted from the sensor to a processing unit.

[0989] 2. Processing of information:

[0990] The processing device (server) stores the received object information in a database.

[0991] The system periodically checks the information about items in the refrigerator, generates a cooking plan for a certain period of time, and creates a list of items that are missing.

[0992] 3. User emotion recognition:

[0993] The emotion recognition engine analyzes the user's facial expressions and voice to recognize the user's emotional information.

[0994] For example, if the user is feeling stressed, that information is sent to the server.

[0995] 4. Generate the recipe:

[0996] The server generates recipes using a generative AI model based on emotional and object information.

[0997] An example prompt is, "The refrigerator contains the following ingredients: tomatoes, cheese, and basil. The user is feeling stressed. Please suggest a recipe."

[0998] 5. Image Generation:

[0999] Based on the generated recipe information, AI image generation technology is used to generate images of the dish. For example, DALL-E is used to generate an image of "Tomato and Basil Bruschetta."

[1000] 6. Transmission and Display of Information:

[1001] The server transmits the generated recipe and image to the user terminal.

[1002] Suggested recipes and images are displayed on the user's device (smartphone or computer), which can be used as a reference for cooking.

[1003] Specific examples

[1004] If the refrigerator contains tomatoes, cheese, and basil, it is recognized that the user is seeking relaxation.

[1005] The server sends prompts to the generative AI model based on object information and emotion information, generating the optimal recipe: "Tomato and Basil Bruschetta."

[1006] Based on the recipe generated by the generative AI model, DALL-E generates an image of "Tomato and Basil Bruschetta."

[1007] This information is sent to the user's terminal, and the user can cook while looking at the displayed recipe and images.

[1008] In this way, the present invention significantly improves user convenience through the management of objects in a refrigerator, the suggestion of cooking methods based on the user's emotional information, and the automatic purchase of ingredients that are in short supply.

[1009] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1010] Step 1:

[1011] The server detects objects in the refrigerator. Sensors in the refrigerator use image recognition technology or radio frequency identification technology (RFID) to detect objects (e.g., tomatoes, cheese, basil, etc.). The detected object information is sent to the server. The input is the image or data from the RFID tag, and the output is the object information.

[1012] Step 2:

[1013] The server stores the received object information in a database. The object information is sent to the server, which records the information in the database. The database contains the object type, quantity, and detection date and time. The input is the object information, and the output is the updated database.

[1014] Step 3:

[1015] The server starts an emotion recognition engine to recognize the user's emotional information. The emotion recognition engine analyzes facial expression images and voice data from the user's device to identify the user's emotional state (e.g., stress, relaxation, etc.). The input is facial expression images and voice data, and the output is emotional information.

[1016] Step 4:

[1017] The server generates a recipe using a generative AI model based on object information and emotional information. The server generates a prompt sentence and inputs it into the generative AI model to suggest the optimal cooking method. For example, a prompt sentence could be "The following ingredients are in the refrigerator: tomatoes, cheese, and basil. The user is feeling stressed. Please suggest a dish." The input is object information and emotional information, and the output is the generated recipe.

[1018] Step 5:

[1019] The server sends the generated recipe information to an image generation engine to generate a food image. Based on the generated recipe, an AI image generation technology such as DALL-E is used to generate a corresponding food image (e.g., "Tomato and Basil Bruschetta"). The input is the recipe information, and the output is the food image.

[1020] Step 6:

[1021] The server sends the generated recipe and image to the user's device. The generated cooking method and its image are then sent to the user's device, where the user can check the suggested recipe and image on their smartphone or computer. The input is the recipe information and image, and the output is the display on the user's device.

[1022] Step 7:

[1023] The server periodically checks the object information in the refrigeration unit. For example, daily or weekly, the server updates the object information in the database and identifies any missing objects. The input is the existing object information, and the output is a list of missing objects.

[1024] Step 8:

[1025] The server generates a cooking plan for one week and sends a list of missing items to the online sales device. A generative AI model is used to generate recipes for one week, list missing items, and automatically send order information to the online sales device. The input is ingredient information and the number of recipes required, and the output is a list of missing items and order information.

[1026] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1027] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1028] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1029] [Third embodiment]

[1030] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1031] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1033] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1034] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1035] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1037] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1038] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1039] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1040] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1041] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1042] This invention is a system that automates the management of food and beverages stored in a refrigerator and suggests effective cooking methods. The system detects objects in the refrigerator and generates cooking instructions and images based on that information, which are then provided to the user. It also has a function that periodically checks the information about the objects in the refrigerator and automatically purchases any missing items online.

[1043] (Program overview and processing explanation)

[1044] Ingredient detection and data generation

[1045] 1. A sensor scans the object inside the refrigerator.

[1046] The server installs multiple sensors inside the refrigeration unit and uses image recognition technology and RFID tags to detect food and beverages.

[1047] For example, the sensor detects two tomatoes, 50g of cheese, and a small amount of basil.

[1048] 2. The server receives object information from the sensor.

[1049] The server receives object information detected by the sensor in real time.

[1050] Examples of received data include {"Object": [{"Name": "Tomato", "Quantity": "2"}, {"Name": "Cheese", "Quantity": "50g"}, {"Name": "Basil", "Quantity": "Small"}]}.

[1051] 3. The server stores the received object information in a database.

[1052] The server stores the received object information in a database and manages it as the latest inventory information.

[1053] Recipe generation and image creation

[1054] 4. The server sends a recipe generation request to the generative AI based on the object information.

[1055] The server sends a request to the generative AI to generate a recipe based on information about the objects inside the refrigerator.

[1056] For example, the server generates a request containing tomato, cheese, basil, and bread.

[1057] 5. A generative AI receives the request and generates a recipe based on the object.

[1058] The generative AI generates the optimal recipe based on the requested object.

[1059] For example, the generated recipe is "Tomato and Basil Bruschetta."

[1060] 6. The server sends the generated recipe to DALL-E and generates an image of the finished dish.

[1061] The server sends an image generation request to DALL-E based on the recipe information received from the generative AI.

[1062] For example, send a request to "generate an image of bruschetta."

[1063] 7. The server sends the generated recipe and image to the user device.

[1064] The server transmits the generated recipe text and image file to the user terminal.

[1065] The content provided includes a "Tomato and Basil Bruschetta Recipe" and an image of it.

[1066] Regular menu planning and online purchasing

[1067] 8. The server periodically checks the information of objects in the refrigerator.

[1068] The server retrieves and checks information about objects in the refrigerator from the database at a specific time each week.

[1069] 9. The server requests a week's worth of menus from the generative AI.

[1070] To generate a week's worth of menus, the server requests information about all objects and the number of dishes required from the generative AI.

[1071] 10. Generative AI generates a week's worth of menus and creates a list of missing items.

[1072] The generative AI creates a weekly menu and identifies missing items and generates a list.

[1073] 11. The server sends the list of missing objects to the online sales device and purchases the required objects.

[1074] The server sends a list of missing objects to an online sales device, which automatically purchases the objects.

[1075] User Examples

[1076] For example, if tomatoes, cheese, basil, and bread are stored in a refrigerator, the user uses the system in the following steps:

[1077] 1. The user starts using the refrigeration unit.

[1078] A user places an object in the refrigerator and a sensor detects this.

[1079] 2. The server receives the object information and stores it in a database.

[1080] The server receives data from the sensors and registers the object information in a database.

[1081] 3. The server requests the generative AI to generate a recipe.

[1082] The server requests the generative AI to generate a recipe based on the object information, and the AI ​​generates a recipe for "Tomato and Basil Bruschetta."

[1083] 4. The server requests image generation from DALL-E.

[1084] The server sends the generated recipe to DALL-E, which generates an image of bruschetta.

[1085] 5. The server sends the generated recipe and image to the user device.

[1086] The recipe and images are displayed on the user's terminal, and the user cooks while looking at them.

[1087] 6. The server periodically checks the object information and creates a menu for the week.

[1088] The server periodically checks the object information and requests a week's worth of menus from the generative AI.

[1089] 7. Automatically purchase absent objects online.

[1090] The server transmits a list of missing items to the online sales device, which automatically carries out the purchase procedure.

[1091] In this way, the present invention automates ingredient management, recipe suggestions, menu creation, and online purchasing, supporting users' busy lives.

[1092] The processing flow will be explained below.

[1093] Step 1:

[1094] The server activates sensors to detect objects inside the refrigerator, which use image recognition technology and RFID tags to detect food and beverages inside the refrigerator.

[1095] Step 2:

[1096] The sensor detects the objects in the refrigerator and sends them to the server. For example, the sensor detects two tomatoes, 50g of cheese, and a small amount of basil.

[1097] Step 3:

[1098] The server stores the object information received from the sensor in a database, including the object name and quantity.

[1099] Step 4:

[1100] The server retrieves the latest object information from the database and sends a recipe generation request to the generative AI, which includes information on all ingredients in the refrigerator.

[1101] Step 5:

[1102] The generative AI receives requests from the server and generates recipes based on the object information. For example, it generates a recipe for "Tomato and Basil Bruschetta."

[1103] Step 6:

[1104] The server sends the generated recipe information to DALL-E, which is an image generation means, and sends a request to generate an image of the finished dish.

[1105] Step 7:

[1106] DALL-E generates an image based on the recipe and sends it back to the server. For example, it generates an "image of bruschetta."

[1107] Step 8:

[1108] The server sends the generated recipe and image to the user's device, where the user can view the proposed recipe and image.

[1109] Step 9:

[1110] The server periodically checks the information of objects in the refrigerator once a week. This check is performed automatically by the scheduler.

[1111] Step 10:

[1112] The server retrieves all object information from the database and sends a request to the generative AI to generate a week's worth of menus.

[1113] Step 11:

[1114] A generative AI receives the request and generates a week's worth of cooking plans and a list of ingredients needed, including any missing ingredients.

[1115] Step 12:

[1116] The server transmits the generated list of absent objects to an online sales device, which automatically purchases the missing objects.

[1117] Step 13:

[1118] Users can pick up ingredients purchased online at a designated location. By receiving any missing ingredients, users can prepare a week's worth of meals as planned.

[1119] Example 1

[1120] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1121] In today's busy world, users spend a lot of time managing ingredients and planning their cooking. In addition, improper management of food and beverages stored in refrigerators can lead to food waste. Furthermore, it can be cumbersome for users to regularly purchase ingredients needed for cooking, and poor management can make cooking difficult.

[1122] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1123] In this invention, the server includes a detection means for detecting objects in the refrigeration unit, a transmission means for transmitting object information detected by the detection means to a processing device, and a generation means for the processing device to generate a cooking method based on the object information. This enables automatic ingredient management and optimal cooking method suggestions. The server also includes an image generation means for generating an image of the generated cooking method, a means for transmitting the generated cooking method and the image to a user terminal, a means for storing the object information received by the processing device in a database, a means for the processing device to request recipe generation from a generative artificial intelligence model, a means for the generative artificial intelligence model to receive the request and generate a recipe based on the object, and a means for the processing device to transmit the generated recipe to an image generation model and generate an image of the finished dish. This allows users to consistently automate everything from daily ingredient management to cooking method suggestions and food image confirmation. It also reduces food waste and realizes efficient ingredient purchasing.

[1124] The "detection means" is a device installed to detect objects inside the refrigeration device, and may use image recognition technology or radio frequency identification technology.

[1125] The "transmitting means" is a device or system for transmitting object information detected by the detecting means to the processing device.

[1126] The "generation means" is a device or system that allows the processing device to generate a cooking method based on object information.

[1127] "Image generation means" refers to a device or system for generating an image of the generated recipe.

[1128] The "means for transmitting to the user terminal" refers to a device or system for transmitting the generated recipes and images to the terminal used by the user.

[1129] A "processing device" is a device or system for processing object information received from a sensor and storing it in a database.

[1130] A "generative artificial intelligence model" is an artificial intelligence model that generates recipes in natural language based on input prompts.

[1131] An "image generation model" is an artificial intelligence model for generating images based on input text information.

[1132] A "database" is a data management system for storing and managing received object information.

[1133] A "refrigeration device" is a device used to store food and beverages at a constant low temperature.

[1134] "Object information" is data about the names and quantities of food and beverages stored in the refrigerator.

[1135] A "recipe" is information that describes the steps and ingredients for cooking a particular dish.

[1136] This invention is a system that automates the management of food and beverages stored in a refrigerator and suggests effective cooking methods. The system detects objects in the refrigerator and generates cooking instructions and images based on that information, which are then provided to the user. It also has a function that periodically checks the information about the objects in the refrigerator and automatically purchases any missing items online.

[1137] This system mainly uses the following hardware and software:

[1138] 1. Sensors: Image recognition sensors and RFID readers are used to detect objects inside the refrigerator.

[1139] 2. Server: Receives object information from sensors, stores it in a database, and requests recipe and image generation from generative artificial intelligence models and image generation models.

[1140] 3. Generative AI models: For example, using natural language generation models such as GPT-3 to generate recipes based on ingredients.

[1141] 4. Image generation model: For example, a text-to-image generation model such as DALL-E is used to generate images of the generated recipe.

[1142] 5. User device: For example, the generated recipes and images are provided to the user using a smartphone or tablet.

[1143] As a concrete example, consider a case where a user stores two tomatoes, 50g of cheese, a little basil, and bread in a refrigerator. The system operates as follows:

[1144] First, sensors inside the refrigerator detect these ingredients and send object information to a server, which stores this information in a database and manages the overall inventory.

[1145] The server then sends a recipe generation request to the generative artificial intelligence model, such as a prompt like "Generate a recipe using tomatoes, cheese, basil, and bread."

[1146] The generative AI model receives this request and generates a recipe called "Tomato and Basil Bruschetta." The server then sends this generated recipe to the image generation model, requesting it to "generate an image of the bruschetta."

[1147] The image generation model receives the request and generates an image of the bruschetta. The server then sends the generated recipe and image to the user's device. The user then cooks the dish while viewing the recipe and image on the device.

[1148] Furthermore, the server can periodically check the information about the objects in the refrigerator and ask the generative AI model to create a week's worth of menus. Once a list of missing objects is created, the server sends the list to the online sales device, which then automatically carries out the purchasing process.

[1149] In this way, the present invention reduces the burden on the user and enables efficient and effective food ingredient management and cooking.

[1150] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1151] Step 1:

[1152] Sensors scan objects inside the refrigerator

[1153] Input: Current state of ingredients in the refrigerator

[1154] How it works: Image recognition sensors and RFID readers installed inside the refrigerator detect ingredients like tomatoes, cheese, and basil. Each sensor captures image data and RFID tag information for each ingredient.

[1155] Output: A list of ingredients and their amounts (e.g., "2 tomatoes, 50g cheese, a little basil")

[1156] Step 2:

[1157] The server receives object information from the sensor

[1158] Input: A list of ingredient names and quantities sent from the sensor

[1159] Specific operation: The server receives object information sent from the sensor in real time. The received data format is, for example, JSON: {"Object": [{"Name": "Tomato", "Quantity": "2"}, {"Name": "Cheese", "Quantity": "50g"}, {"Name": "Basil", "Quantity": "Small"}]}.

[1160] Output: Received object information data

[1161] Step 3:

[1162] The server stores the received object information in a database.

[1163] Input: Received object information data

[1164] Specific operation: The server analyzes the object information and stores it in a database. The database keeps up-to-date information on the types and quantities of ingredients. In this example, the database stores "2 tomatoes, 50g of cheese, and a little basil."

[1165] Output: Latest object information stored in the database

[1166] Step 4:

[1167] The server sends a recipe generation request to the generative AI based on the object information.

[1168] Input: The latest object information stored in the database

[1169] Specific operation: The server retrieves object information from the database and sends it to the generative AI (e.g., GPT-3) as a recipe generation request. The request is sent in the form of a prompt: "Please generate a recipe using tomatoes, cheese, basil, and bread."

[1170] Output: The prompt sent to the generative AI

[1171] Step 5:

[1172] A generative AI receives the request and generates a recipe based on the object.

[1173] Input: Prompt sent from the server

[1174] How it works: A generative AI (e.g., GPT-3) analyzes the prompt and generates an optimal recipe based on the input object information. For example, the generated recipe might be "Tomato and Basil Bruschetta."

[1175] Output: Generated recipe (text format)

[1176] Step 6:

[1177] The server sends the generated recipe to the image generation model, which generates an image of the finished dish.

[1178] Input: Recipe generated by generative AI

[1179] Specific operation: The server obtains the generated recipe information and sends an image generation request to the image generation model (e.g., DALL-E). The request is sent in the form of, for example, "Please generate an image of bruschetta."

[1180] Output: Generated food images

[1181] Step 7:

[1182] The server sends the generated recipe and image to the user's device.

[1183] Input: Generated recipes and food images

[1184] Specific operation: The server sends the generated recipe text and image file to the user's device so that the user can check it. For example, the user's smartphone will display "Tomato and Basil Bruschetta Recipe" and its image.

[1185] Output: Recipe and image displayed on the user's device

[1186] Step 8:

[1187] The server periodically checks the information about objects in the refrigerator.

[1188] Input: Object information stored in the database

[1189] Specific operation: The server periodically (e.g., every Monday at 9:00 AM) accesses the database and checks the information about objects in the refrigerator.

[1190] Output: Information about the most recent object in the refrigerator that was checked

[1191] Step 9:

[1192] The server requests a week's worth of menus from the generative AI

[1193] Input: Information about the object in the most recent refrigerator that was checked

[1194] Specific operation: To generate a week's worth of menus, the server requests all object information in the database and the number of dishes required from the generative AI. The request is sent in the form of a prompt, such as "Please generate a seven-day menu."

[1195] Output: The prompt sent to the generative AI

[1196] Step 10:

[1197] Generative AI generates a week's worth of meals and creates a list of missing items

[1198] Input: Prompt sent from the server

[1199] Specific operation: A generative AI (e.g., GPT-3) analyzes the prompts and generates a weekly menu, identifying and listing any missing items. For example, a generated menu and a "list of missing items" are created.

[1200] Output: Generated weekly menu and list of missing items

[1201] Step 11:

[1202] The server sends the list of missing objects to the online sales device and purchases the required objects.

[1203] Input: Generated missing object list

[1204] Specific operation: The server sends a list of missing items to the online sales device, which then automatically processes the purchase. For example, a list of "2 tomatoes, 100g of cheese" is sent, and an online order is placed.

[1205] Output: Required objects purchased

[1206] In this way, the entire system flow is completed, allowing users to smoothly carry out a series of processes, from managing ingredients and suggesting cooking methods to automatically purchasing the necessary ingredients.

[1207] (Application example 1)

[1208] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1209] Managing food and beverages in a refrigerator requires manual checking and replenishing, which is a significant time burden for users. Furthermore, since no cooking method suggestions are provided, it is difficult to effectively utilize the ingredients in the refrigerator. Furthermore, there is no system that automatically replenishes ingredients when they run out, so users must manually order them. To solve these issues, there is a need for a system that can automatically manage food and beverages, suggest effective cooking methods, and automatically replenish ingredients when they run out.

[1210] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1211] In this invention, the server includes a detection means for detecting objects in the refrigeration unit, a transmission means for transmitting object information detected by the detection means to a processing device, a generation means for the processing device to generate a cooking method based on the object information, an image generation means for generating an image of the generated cooking method, a means for transmitting the generated cooking method and the image to a user terminal, and a replenishment means for the processing device to generate order information for replenishing ingredients based on the object information and automatically order ingredients through an online service. This enables automated management of food and beverages in the refrigeration unit, suggestions for effective cooking methods, and automatic replenishment of ingredients when they run out.

[1212] A "refrigeration unit" is a device designed to maintain a low internal temperature and is used to preserve food, beverages, etc.

[1213] A "system" is a whole made up of multiple elements that are interrelated and combined to achieve a certain purpose.

[1214] "Detection means" means a device or function, including sensor technology, for accurately identifying and detecting the presence and type of object within the refrigeration unit.

[1215] "Transmitting means" refers to a device or function that includes communication technology for transmitting object information collected by the detecting means to the processing device.

[1216] The "generation means" is a device or function that includes an algorithm or artificial intelligence for automatically creating a cooking method based on object information.

[1217] "Image generation means" refers to a device or function including an artificial intelligence or image generation engine for creating visual food images based on the generated cooking instructions.

[1218] A "user terminal" is a device such as a smartphone or computer that a user uses to receive, check, or operate information provided by the system.

[1219] The "replenishment means" is a function or system for automatically ordering and replenishing missing food or beverages based on object information.

[1220] The "processing device" is a computer or control unit that receives and analyzes the object information transmitted from the detection means and controls the entire system.

[1221] The system to realize this application example utilizes various sensors, processing devices, and communication means built into the refrigeration equipment to automatically manage food and beverages, suggest effective cooking methods, and even automatically replenish ingredients when they run out.

[1222] The server includes a detection means that uses image recognition technology or radio frequency identification technology to detect objects in the refrigeration device. The detection means collects object information using, for example, a camera or an RFID reader. The collected object information is transmitted to the server via a transmission means.

[1223] The server receives the transmitted object information with a processing device and generates an optimal cooking method based on the object information using a generation means, such as an OpenAI generative AI model. Based on the generated cooking method, an image generation engine such as DALL-E is used to generate an image of the cooking method.

[1224] The image of the cooking method and the recipe generated by the image generating means are transmitted to a user terminal via a transmitting means, which may be a smartphone or tablet, and the user can check the cooking method and the image through the user terminal.

[1225] Furthermore, the server periodically checks the information on the items in the refrigerator and generates a cooking plan for the week. Based on the generated cooking plan, it creates a list of items that are in short supply and automatically orders them online. This eliminates the need for users to manually check and order.

[1226] For example, if tomatoes, cheese, and basil are stored in the refrigerator, the server will request a recipe from the generative AI model based on this information. The generative AI model will suggest "Tomato and Basil Bruschetta," and DALL-E will generate an image of it. Furthermore, if the bread needed for the bruschetta is in short supply, it will automatically be ordered from an online service.

[1227] Here are some examples of prompts for generative AI models:

[1228] If you have the following ingredients in your fridge, please suggest some possible dishes.

[1229] object:

[1230] Name: Tomato

[1231] Quantity: 2 pieces

[1232] Name: Cheese

[1233] Amount: 50g

[1234] Name: Basil

[1235] Quantity: small quantity

[1236] This system automates the management of food and beverages in refrigeration units, allowing users to cook efficiently while saving time and effort.

[1237] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1238] Step 1:

[1239] Detect objects inside a refrigeration unit.

[1240] How it works: The server collects data on food and beverages using cameras and RFID readers installed inside the refrigeration units.

[1241] Input: Image data or RFID tag data of objects in the refrigeration unit.

[1242] Data processing: Identifying the type and quantity of objects using image recognition and RFID technology.

[1243] Output: Identified object information (e.g., "2 tomatoes", "50g cheese", "small amount of basil").

[1244] Step 2:

[1245] The detected object information is transmitted to a processing device.

[1246] Specific operations: The server transmits the object information obtained by the detection means to the processing device.

[1247] Input: Identified object information.

[1248] Data calculation: Formatting object information according to the protocol.

[1249] Output: Formatted object information data.

[1250] Step 3:

[1251] Cooking methods are generated based on object information.

[1252] Specific operation: The server sends a cooking method generation request to the generative AI model based on the object information.

[1253] Input: Formatted object information data.

[1254] Data calculation: The generative AI model performs natural language processing and database search to generate optimal recipes based on object information.

[1255] Output: The generated recipe (e.g., "Tomato and Basil Bruschetta").

[1256] Step 4:

[1257] Generate an image of the cooking method.

[1258] Specific operation: The server sends the generated text information of the cooking method to the image generation engine to generate a visual image of the dish.

[1259] Input: The generated text information of the cooking instructions.

[1260] Data calculation: An AI model processes and generates images based on text information.

[1261] Output: The generated food image (e.g. "Image of bruschetta").

[1262] Step 5:

[1263] The generated recipe and image are sent to the user terminal.

[1264] Specific operation: The server sends cooking instructions and images to the user's smartphone or tablet.

[1265] Input: Generated recipes and images.

[1266] Data calculation: Processes data in a format suitable for the user terminal and sends it.

[1267] Output: Cooking instructions and images displayed on the user's device.

[1268] Step 6:

[1269] Regularly check the contents of the refrigerator.

[1270] Specific operation: The server rechecks the information about objects in the refrigerator at set intervals.

[1271] Input: Current object information data.

[1272] Data calculation: Compare with existing information in the database and update new object information.

[1273] Output: Updated object information data.

[1274] Step 7:

[1275] Generate cooking plans for a certain period of time.

[1276] Specific operation: The server sends a request to the generative AI model to generate a cooking plan for a certain period (e.g., one week) based on the collected object information.

[1277] Input: Updated object information data.

[1278] Data calculation: A generative AI model generates a cooking plan that combines multiple recipes based on object information.

[1279] Output: Generated cooking plan (e.g., recipes for a week).

[1280] Step 8:

[1281] Automatically order missing items online.

[1282] Specific operation: The server creates a list of ingredients that are missing based on the generated cooking plan and places an order with the online sales service.

[1283] Input: Generated cooking plan, missing object list.

[1284] Data processing: Generates order information and processes orders through the API of online services.

[1285] Output: Online order confirmation information.

[1286] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1287] This invention relates to a system that manages objects stored in a refrigerator and suggests optimal cooking methods based on the user's emotions. The system detects objects stored in the refrigerator, generates optimal recipes based on that information, and customizes cooking methods by taking the user's emotions into account using an emotion engine. It also has the functionality to automate object management and periodic online purchasing of items that are in short supply.

[1288] (Program overview and processing explanation)

[1289] Ingredient detection and data generation

[1290] 1. A sensor scans the object inside the refrigerator.

[1291] The server activates sensors installed inside the refrigerator that use image recognition technology and RFID tags to detect objects, such as two tomatoes, 50g of cheese, and a small amount of basil.

[1292] 2. The server receives object information from the sensor.

[1293] The sensor sends the detected object information to the server, which stores it in a database. The stored information includes the object name and quantity.

[1294] Recipe Generation and Emotion Recognition

[1295] 3. The server launches an emotion engine that recognizes the user's emotions.

[1296] The emotion engine analyzes user input, voice, and facial expressions to recognize the user's emotional state. For example, if the user is feeling stressed, it will suggest recipes to help them relax.

[1297] 4. The server sends the object information and the user's emotion information to the generative AI.

[1298] The server sends the object information obtained from the database and the user's emotion information recognized by the emotion engine to the generative AI.

[1299] 5. Generative AI generates recipes based on object information and emotional information.

[1300] The generative AI generates optimal recipes based on object information and the user's emotional information. For example, if the user is feeling stressed, it generates a relaxing recipe such as "Tomato and Basil Bruschetta."

[1301] Image generation and information transmission

[1302] 6. The server sends the generated recipe to DALL-E, which is the image generation means.

[1303] The server sends the recipe information received from the generative AI to DALL-E, which then generates an image of the dish based on that recipe.

[1304] 7. DALL-E generates an image based on the recipe and sends it back to the server.

[1305] For example, an "image of bruschetta" is generated and sent back to the server.

[1306] 8. The server sends the generated recipe and image to the user device.

[1307] The server sends the generated recipe text and image files to the user's terminal, where the user can view the proposed recipe and images.

[1308] Regular menu planning and online purchasing

[1309] 9. The server periodically checks the information of objects in the refrigerator once a week.

[1310] The server checks the database for information about objects in the refrigerator at a specific time each week according to a scheduler.

[1311] 10. The server requests a week's worth of menus from the generative AI.

[1312] To generate a week's worth of menus, the server requests information about all objects and the number of dishes required from the generative AI.

[1313] 11. Generative AI generates a week's worth of menus and creates a list of missing items.

[1314] The generative AI creates a weekly menu and identifies missing items and generates a list.

[1315] 12. The server sends the list of missing objects to the online sales device and purchases the required objects.

[1316] The server sends a list of missing objects to an online sales device, which automatically purchases the objects.

[1317] User Examples

[1318] For example, if tomatoes, cheese, basil, and bread are stored in a refrigerator, the user uses the system in the following steps:

[1319] 1. The user starts using the refrigeration unit.

[1320] A user places an object in the refrigerator and a sensor detects this.

[1321] 2. The server receives the object information and stores it in a database.

[1322] The server receives data from the sensors and registers the object information in a database.

[1323] 3. The emotion engine detects the user's emotions.

[1324] The emotion engine analyzes the user's voice and facial expressions and recognizes that the user is seeking relaxation.

[1325] 4. The server requests the generative AI to generate a recipe.

[1326] The server requests a generative AI to generate a recipe based on object information and emotional information, and the AI ​​generates a recipe for "Tomato and Basil Bruschetta."

[1327] 5. The server requests DALL-E to generate an image.

[1328] The server sends the generated recipe to DALL-E, which generates an image of bruschetta.

[1329] 6. The server sends the generated recipe and image to the user device.

[1330] The recipe and images are displayed on the user's terminal, and the user cooks while looking at them.

[1331] 7. The server periodically checks the object information and creates a menu for the week.

[1332] The server periodically checks the object information and requests a week's worth of menus from the generative AI.

[1333] 8. Automatically purchase absent objects online.

[1334] The server transmits a list of missing items to the online sales device, which automatically carries out the purchase procedure.

[1335] In this way, the present invention automates ingredient management, recipe suggestions, menu planning, emotion-based recipe optimization, and online purchasing to support users' busy lives.

[1336] The processing flow will be explained below.

[1337] Step 1:

[1338] The server activates the sensor to detect the object inside the refrigerator. The sensor uses image recognition technology and RFID tags to detect the object (food or drink) inside the refrigerator.

[1339] Step 2:

[1340] The sensor detects information about objects in the refrigerator and sends it to the server. For example, the sensor detects "2 tomatoes," "50g of cheese," and "a little basil."

[1341] Step 3:

[1342] The server stores the object information received from the sensor in a database, including the object name and quantity.

[1343] Step 4:

[1344] The server activates an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's input, voice, and facial expressions to recognize the user's emotional state.

[1345] Step 5:

[1346] The emotion engine analyzes the user's emotions and generates emotion information, for example, recognizing that the user is feeling stressed.

[1347] Step 6:

[1348] The server sends a recipe generation request to the generative AI based on the object information obtained from the database and the emotion information generated by the emotion engine.

[1349] Step 7:

[1350] Generative AI generates optimal recipes based on object and emotional information. For example, it generates a recipe for "Tomato and Basil Bruschetta" to help users relax.

[1351] Step 8:

[1352] The server sends the generated recipe to DALL-E, an image generation means, which generates an image of the dish based on the recipe.

[1353] Step 9:

[1354] DALL-E generates an image of the dish based on the recipe and sends it back to the server. For example, it generates an image of bruschetta.

[1355] Step 10:

[1356] The server sends the generated recipe and image to the user's terminal, where the user can check the proposed recipe and image.

[1357] Step 11:

[1358] The server sets a scheduler to periodically check the object information in the refrigerator, retrieving and checking the object information from the database at a specific time every week.

[1359] Step 12:

[1360] The server retrieves all object information from the database and sends a request to the generative AI to generate a week's worth of menus.

[1361] Step 13:

[1362] Generative AI generates a weekly menu and creates a list of missing items. For example, it creates a menu that includes menus for Monday through Sunday.

[1363] Step 14:

[1364] The server transmits a list of missing objects to an online sales device, which automatically purchases the missing objects.

[1365] Step 15:

[1366] Users can pick up items they have purchased online at a specified location. By having any missing ingredients delivered to them, users can plan their meals for the week.

[1367] Example 2

[1368] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1369] In today's busy lifestyles, managing ingredients and suggesting effective cooking methods requires a great deal of effort. It is also difficult to find appropriate cooking methods that reduce stress and fatigue. Therefore, there is a need for a system that can effectively manage items in a refrigerator, suggest optimal cooking methods based on the user's emotions, and automatically purchase ingredients when they are in short supply.

[1370] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1371] In this invention, the server includes a detection means for detecting objects in the refrigeration unit, a transmission means for transmitting object information detected by the detection means to the processing device, a generation means for generating a cooking method based on the object information, an emotion recognition means for recognizing a user's emotion, a generation means for generating a cooking method based on the object information and emotion information, an image generation means for generating an image of the generated cooking method, and a means for transmitting the generated cooking method and image to a user terminal. This makes it possible to efficiently manage objects in the refrigeration unit and suggest appropriate cooking methods based on the user's emotion, not only facilitating meal preparation but also improving the user's psychological satisfaction. It also enables automatic purchasing of ingredients that are in short supply, further improving convenience.

[1372] "Detection means" refers to a device or system for detecting objects within the refrigeration unit.

[1373] The "transmitting means" refers to a device or system for transmitting object information detected by the detecting means to the processing device.

[1374] The term "processing device" refers to a device or system that generates and manages recipes based on object information received from a transmitting means.

[1375] The "generation means" refers to a device or system for generating a cooking method based on object information and user emotion information.

[1376] "Emotion recognition means" refers to a device or system for recognizing the emotional state of a user.

[1377] "Image generation means" refers to a device or system for generating an image based on the generated recipe.

[1378] The term "user terminal" refers to an apparatus or device for displaying generated recipes and images to a user.

[1379] "Database" refers to a system for storing object information, user emotion information, and other related data.

[1380] "Online sales device" refers to a system that processes orders online to purchase missing objects.

[1381] "Scheduler" refers to a system that manages a schedule for periodically checking information about objects in a refrigeration device.

[1382] The present invention relates to a system that manages items stored in a refrigerator and suggests optimal cooking methods based on the user's emotions. This system utilizes multiple hardware and software components to efficiently and automatically manage ingredients and suggest cooking methods.

[1383] Hardware and Software Configuration

[1384] 1. Sensors: Sensors with image recognition technology and RFID tag readers are placed inside the refrigerator, allowing for accurate detection of objects inside the refrigerator.

[1385] 2. Server: The central part of the system, it centrally manages and processes data from various devices and software, such as sensors, emotion recognition engines, generative AI, and image generation methods.

[1386] 3. Emotion Recognition Engine: Uses software to analyze user input (voice, facial expressions, etc.) and recognize emotions. For example, determining stress or happiness from the user's tone of voice and facial expressions.

[1387] 4. Generative AI: Uses AI models to generate optimal recipes based on object and emotion information. For example, this applies to recipe suggestion systems that use natural language processing.

[1388] 5. Image generation means: Software for generating food images based on the generated recipe. This includes DALL-E and similar image generation models.

[1389] 6. User device: A device (smartphone, tablet, etc.) on which the user checks the recipe and generated images. It displays the data sent from the server.

[1390] 7. Database: Use a system to store object information, user emotion information, recipe information, etc.

[1391] 8. Online Sales Device: A system for automatically purchasing missing items. The purchasing process is carried out using the online store API.

[1392] Specific actions

[1393] 1. Ingredient detection:

[1394] When a user places new ingredients in the refrigerator, the sensor scans them and sends the detection results to the server. For example, if a tomato and cheese are newly added to the refrigerator, image recognition technology and an RFID tag reader will detect this.

[1395] 2. Data storage:

[1396] The server analyzes the received data and stores it in a database. For example, it might register "2 tomatoes, 50g of cheese."

[1397] 3. Emotion Recognition:

[1398] When a user speaks into the smartphone or faces the camera, the emotion recognition engine analyzes this and sends the user's emotional information to the server. For example, it may determine that the user is feeling stressed.

[1399] 4. Generate the recipe:

[1400] The server retrieves object and emotion information from the database and sends it to the generative AI, which then generates the optimal recipe based on the user's emotions. For example, it suggests "relaxing tomato and basil bruschetta."

[1401] 5. Image Generation:

[1402] The server transmits the generated recipe to the image generating means, which generates a related food image, for example, an image of "bruschetta."

[1403] 6. Submit your recipe and images:

[1404] The server sends the generated recipe and images to the user terminal, where the user can check the proposed recipe and images.

[1405] 7. Regular Checks and Purchases:

[1406] The server periodically checks the information of the items in the refrigerator and lists the items that are missing, and automatically purchases the missing items through the online sales device based on the list.

[1407] Specific examples

[1408] Example prompt sentence:

[1409] "What are some relaxing dishes that you would suggest to users when they are feeling stressed?"

[1410] "If I have tomatoes, cheese, and basil in my refrigerator, can you suggest a recipe that works best?"

[1411] In this way, the present invention provides a system that allows users to efficiently manage ingredients and enjoy optimal recipes that suit their own emotional state.

[1412] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1413] Step 1:

[1414] Ingredient detection

[1415] Input: User stores new ingredients in the refrigerator.

[1416] Processing: The server activates sensors (image recognition technology and RFID tag readers) inside the refrigeration unit to scan the object.

[1417] Output: Data of detected ingredients.

[1418] What it does: When a user adds 2 tomatoes, 50g of cheese, and a little basil to the refrigerator, the sensor detects this and sends the data (2 tomatoes, 50g of cheese, a little basil) to the server.

[1419] Step 2:

[1420] Data storage

[1421] Input: Ingredient data sent from the sensor.

[1422] Processing: The server receives the data from the sensors and stores it in a database.

[1423] Output: Ingredient information stored in a database.

[1424] Specific operation: The server analyzes the data received from the sensor (2 tomatoes, 50g of cheese, and a small amount of basil) and registers it in a database.

[1425] Step 3:

[1426] Emotion recognition

[1427] Input: User voice and facial expression data.

[1428] Processing: The server uses an emotion recognition engine to analyze the user's emotions.

[1429] Output: The perceived emotional state of the user.

[1430] Specific operation: When a user speaks into the smartphone or faces the camera, the emotion recognition engine analyzes this and determines that the user is feeling stressed.

[1431] Step 4:

[1432] Recipe Generation

[1433] Input: Ingredient information obtained from the database and emotion information from the emotion recognition engine.

[1434] Processing: The server sends the ingredient information and emotion information to the generative AI and requests it to generate an appropriate recipe.

[1435] Output: Generated recipe information.

[1436] Specific operation: The server sends information about tomatoes, cheese, and basil, as well as the user's stress level, to the generative AI, which then generates a recipe for "Tomato and Basil Bruschetta."

[1437] Step 5:

[1438] Image generation

[1439] Input: Recipe information from a generative AI.

[1440] Processing: The server sends the recipe information to the image generation means (such as DALL-E), which generates an image based on the recipe.

[1441] Output: The generated food image.

[1442] Specific operation: The server sends the recipe information for "Tomato and Basil Bruschetta" received from the generative AI to DALL-E, and generates an image of the bruschetta.

[1443] Step 6:

[1444] Submit recipes and images

[1445] Input: Generated recipe information and images.

[1446] Processing: The server sends the generated recipe and image to the user device.

[1447] Output: Recipe and image displayed on user's device.

[1448] Specific operation: The server sends the generated recipe and image of "Tomato and Basil Bruschetta" to the user's smartphone or tablet, and the user confirms it.

[1449] Step 7:

[1450] Regular checks and purchases

[1451] Input: Ingredient information stored in the database.

[1452] Processing: The server periodically checks the ingredient information in the database using a scheduler and generates a list of missing items.

[1453] Output: A list of missing objects.

[1454] Specific operation: At a set time each week, the server checks the information about the objects in the refrigerator, requests a week's worth of menus from the generative AI, and creates a list of missing objects.

[1455] Step 8:

[1456] Buy Online

[1457] Input: A list of missing objects.

[1458] Processing: The server sends the list of missing objects to the online sales device, which automatically purchases the required objects.

[1459] Output: Purchased object information.

[1460] Specific operation: The server uses an online sales API to automatically order the missing items (e.g., 1 liter of milk, 6 eggs) and complete the purchase.

[1461] (Application example 2)

[1462] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1463] The present invention aims to provide a system that reduces the effort required for managing items in a refrigerator and provides optimal cooking methods according to the user's emotional state. Another objective is to make users' lives more convenient by automatically detecting shortages of ingredients and providing a mechanism for purchasing necessary items online.

[1464] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a detection means for detecting objects in the refrigeration device, a transmission means for transmitting object information detected by the detection means to the processing device, a generation means for the processing device to generate a cooking method based on the object information, a means for the generation means to customize the cooking method based on user emotion information, an image generation means for generating an image of the generated cooking method, and a means for transmitting the generated cooking method and image to a user terminal. This automates the management of objects in the refrigeration device, makes it possible to suggest optimal cooking methods based on the user's emotions, and further enables efficient use of the refrigeration device by automatically purchasing ingredients that are running low, without the user having to perform any cumbersome operations.

[1465] A "refrigeration unit" is a box-shaped electrical device used to keep food and beverages at low temperatures.

[1466] "Object" refers to the specific food or beverage stored within the refrigeration unit.

[1467] "Detection means" refers to devices and techniques for identifying and obtaining information about objects within the refrigeration unit.

[1468] "Transmitting means" refers to a device or technology for transmitting object information acquired by the detecting means to a processing device.

[1469] The "processing device" is a computer device that processes the received object information and generates cooking methods and recipes.

[1470] "Generation means" refers to technology or devices for generating optimal cooking methods and recipes based on object information and user emotional information.

[1471] "Emotion information" is information that represents the emotional state of the user, obtained by analyzing the user's facial expression or voice.

[1472] "Means for customizing" refers to a technology or device in which the generating means changes the cooking method based on the user's emotional information.

[1473] "Image generation means" refers to technology or equipment for generating an image of a dish based on the generated cooking method.

[1474] A "user terminal" is a device for displaying the generated recipes and images, and includes a smartphone, a computer, etc.

[1475] A "generative AI model" is an artificial intelligence model used to generate cooking instructions and images from input data.

[1476] A "prompt sentence" is an input text provided to a generative AI model, and is a sentence that indicates the direction of the content to be generated.

[1477] "Online sales device" refers to a system or device for purchasing missing items via the Internet.

[1478] The present invention is a system that manages objects in a refrigeration device and suggests optimal cooking methods based on the user's emotional information. This system includes multiple steps, such as object detection, information processing, emotion recognition, recipe generation, and image generation. Specific system embodiments are described below.

[1479] Hardware and software used

[1480] Refrigeration unit: An electrical device used to preserve food and beverages.

[1481] Detection methods: Image recognition technology (e.g., OpenCV) and radio frequency identification technology (e.g., RFID tags).

[1482] Transmission method: Data communication using Wi-Fi module.

[1483] Processing unit: A high-performance computer or cloud server.

[1484] Generation method: Cooking instructions are generated using AI technology (e.g., OpenAI GPT-3).

[1485] Emotion Recognition Engine: Facial expression recognition technology (e.g., facial_emotion_recognition) and voice analysis technology.

[1486] Image generation method: AI image generation technology (e.g., DALL-E).

[1487] User devices: smartphones and computers.

[1488] System Operation

[1489] 1. Object detection:

[1490] Objects inside the refrigerator are detected using image recognition and radio frequency identification technology.

[1491] The detected object information is transmitted from the sensor to a processing unit.

[1492] 2. Processing of information:

[1493] The processing device (server) stores the received object information in a database.

[1494] The system periodically checks the information about items in the refrigerator, generates a cooking plan for a certain period of time, and creates a list of items that are missing.

[1495] 3. User emotion recognition:

[1496] The emotion recognition engine analyzes the user's facial expressions and voice to recognize the user's emotional information.

[1497] For example, if the user is feeling stressed, that information is sent to the server.

[1498] 4. Generate the recipe:

[1499] The server generates recipes using a generative AI model based on emotional and object information.

[1500] An example prompt is, "The refrigerator contains the following ingredients: tomatoes, cheese, and basil. The user is feeling stressed. Please suggest a recipe."

[1501] 5. Image Generation:

[1502] Based on the generated recipe information, AI image generation technology is used to generate images of the dish. For example, DALL-E is used to generate an image of "Tomato and Basil Bruschetta."

[1503] 6. Transmission and Display of Information:

[1504] The server transmits the generated recipe and image to the user terminal.

[1505] Suggested recipes and images are displayed on the user's device (smartphone or computer), which can be used as a reference for cooking.

[1506] Specific examples

[1507] If the refrigerator contains tomatoes, cheese, and basil, it is recognized that the user is seeking relaxation.

[1508] The server sends prompts to the generative AI model based on object information and emotion information, generating the optimal recipe: "Tomato and Basil Bruschetta."

[1509] Based on the recipe generated by the generative AI model, DALL-E generates an image of "Tomato and Basil Bruschetta."

[1510] This information is sent to the user's terminal, and the user can cook while looking at the displayed recipe and images.

[1511] In this way, the present invention significantly improves user convenience through the management of objects in a refrigerator, the suggestion of cooking methods based on the user's emotional information, and the automatic purchase of ingredients that are in short supply.

[1512] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1513] Step 1:

[1514] The server detects objects in the refrigerator. Sensors in the refrigerator use image recognition technology or radio frequency identification technology (RFID) to detect objects (e.g., tomatoes, cheese, basil, etc.). The detected object information is sent to the server. The input is the image or data from the RFID tag, and the output is the object information.

[1515] Step 2:

[1516] The server stores the received object information in a database. The object information is sent to the server, which records the information in the database. The database contains the object type, quantity, and detection date and time. The input is the object information, and the output is the updated database.

[1517] Step 3:

[1518] The server starts an emotion recognition engine to recognize the user's emotional information. The emotion recognition engine analyzes facial expression images and voice data from the user's device to identify the user's emotional state (e.g., stress, relaxation, etc.). The input is facial expression images and voice data, and the output is emotional information.

[1519] Step 4:

[1520] The server generates a recipe using a generative AI model based on object information and emotional information. The server generates a prompt sentence and inputs it into the generative AI model to suggest the optimal cooking method. For example, a prompt sentence could be "The following ingredients are in the refrigerator: tomatoes, cheese, and basil. The user is feeling stressed. Please suggest a dish." The input is object information and emotional information, and the output is the generated recipe.

[1521] Step 5:

[1522] The server sends the generated recipe information to an image generation engine to generate a food image. Based on the generated recipe, an AI image generation technology such as DALL-E is used to generate a corresponding food image (e.g., "Tomato and Basil Bruschetta"). The input is the recipe information, and the output is the food image.

[1523] Step 6:

[1524] The server sends the generated recipe and image to the user's device. The generated cooking method and its image are then sent to the user's device, where the user can check the suggested recipe and image on their smartphone or computer. The input is the recipe information and image, and the output is the display on the user's device.

[1525] Step 7:

[1526] The server periodically checks the object information in the refrigeration unit. For example, daily or weekly, the server updates the object information in the database and identifies any missing objects. The input is the existing object information, and the output is a list of missing objects.

[1527] Step 8:

[1528] The server generates a cooking plan for one week and sends a list of missing items to the online sales device. A generative AI model is used to generate recipes for one week, list missing items, and automatically send order information to the online sales device. The input is ingredient information and the number of recipes required, and the output is a list of missing items and order information.

[1529] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1530] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1531] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1532] [Fourth embodiment]

[1533] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1534] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1535] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1536] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1537] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1538] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1539] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1540] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1541] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1542] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1543] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1544] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1545] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1546] This invention is a system that automates the management of food and beverages stored in a refrigerator and suggests effective cooking methods. The system detects objects in the refrigerator and generates cooking instructions and images based on that information, which are then provided to the user. It also has a function that periodically checks the information about the objects in the refrigerator and automatically purchases any missing items online.

[1547] (Program overview and processing explanation)

[1548] Ingredient detection and data generation

[1549] 1. A sensor scans the object inside the refrigerator.

[1550] The server installs multiple sensors inside the refrigeration unit and uses image recognition technology and RFID tags to detect food and beverages.

[1551] For example, the sensor detects two tomatoes, 50g of cheese, and a small amount of basil.

[1552] 2. The server receives object information from the sensor.

[1553] The server receives object information detected by the sensor in real time.

[1554] Examples of received data include {"Object": [{"Name": "Tomato", "Quantity": "2"}, {"Name": "Cheese", "Quantity": "50g"}, {"Name": "Basil", "Quantity": "Small"}]}.

[1555] 3. The server stores the received object information in a database.

[1556] The server stores the received object information in a database and manages it as the latest inventory information.

[1557] Recipe generation and image creation

[1558] 4. The server sends a recipe generation request to the generative AI based on the object information.

[1559] The server sends a request to the generative AI to generate a recipe based on information about the objects inside the refrigerator.

[1560] For example, the server generates a request containing tomato, cheese, basil, and bread.

[1561] 5. A generative AI receives the request and generates a recipe based on the object.

[1562] The generative AI generates the optimal recipe based on the requested object.

[1563] For example, the generated recipe is "Tomato and Basil Bruschetta."

[1564] 6. The server sends the generated recipe to DALL-E and generates an image of the finished dish.

[1565] The server sends an image generation request to DALL-E based on the recipe information received from the generative AI.

[1566] For example, send a request to "generate an image of bruschetta."

[1567] 7. The server sends the generated recipe and image to the user device.

[1568] The server transmits the generated recipe text and image file to the user terminal.

[1569] The content provided includes a "Tomato and Basil Bruschetta Recipe" and an image of it.

[1570] Regular menu planning and online purchasing

[1571] 8. The server periodically checks the information of objects in the refrigerator.

[1572] The server retrieves and checks information about objects in the refrigerator from the database at a specific time each week.

[1573] 9. The server requests a week's worth of menus from the generative AI.

[1574] To generate a week's worth of menus, the server requests information about all objects and the number of dishes required from the generative AI.

[1575] 10. Generative AI generates a week's worth of menus and creates a list of missing items.

[1576] The generative AI creates a weekly menu and identifies missing items and generates a list.

[1577] 11. The server sends the list of missing objects to the online sales device and purchases the required objects.

[1578] The server sends a list of missing objects to an online sales device, which automatically purchases the objects.

[1579] User Examples

[1580] For example, if tomatoes, cheese, basil, and bread are stored in a refrigerator, the user uses the system in the following steps:

[1581] 1. The user starts using the refrigeration unit.

[1582] A user places an object in the refrigerator and a sensor detects this.

[1583] 2. The server receives the object information and stores it in a database.

[1584] The server receives data from the sensors and registers the object information in a database.

[1585] 3. The server requests the generative AI to generate a recipe.

[1586] The server requests the generative AI to generate a recipe based on the object information, and the AI ​​generates a recipe for "Tomato and Basil Bruschetta."

[1587] 4. The server requests image generation from DALL-E.

[1588] The server sends the generated recipe to DALL-E, which generates an image of bruschetta.

[1589] 5. The server sends the generated recipe and image to the user device.

[1590] The recipe and images are displayed on the user's terminal, and the user cooks while looking at them.

[1591] 6. The server periodically checks the object information and creates a menu for the week.

[1592] The server periodically checks the object information and requests a week's worth of menus from the generative AI.

[1593] 7. Automatically purchase absent objects online.

[1594] The server transmits a list of missing items to the online sales device, which automatically carries out the purchase procedure.

[1595] In this way, the present invention automates ingredient management, recipe suggestions, menu creation, and online purchasing, supporting users' busy lives.

[1596] The processing flow will be explained below.

[1597] Step 1:

[1598] The server activates sensors to detect objects inside the refrigerator, which use image recognition technology and RFID tags to detect food and beverages inside the refrigerator.

[1599] Step 2:

[1600] The sensor detects the objects in the refrigerator and sends them to the server. For example, the sensor detects two tomatoes, 50g of cheese, and a small amount of basil.

[1601] Step 3:

[1602] The server stores the object information received from the sensor in a database, including the object name and quantity.

[1603] Step 4:

[1604] The server retrieves the latest object information from the database and sends a recipe generation request to the generative AI, which includes information on all ingredients in the refrigerator.

[1605] Step 5:

[1606] The generative AI receives requests from the server and generates recipes based on the object information. For example, it generates a recipe for "Tomato and Basil Bruschetta."

[1607] Step 6:

[1608] The server sends the generated recipe information to DALL-E, which is an image generation means, and sends a request to generate an image of the finished dish.

[1609] Step 7:

[1610] DALL-E generates an image based on the recipe and sends it back to the server. For example, it generates an "image of bruschetta."

[1611] Step 8:

[1612] The server sends the generated recipe and image to the user's device, where the user can view the proposed recipe and image.

[1613] Step 9:

[1614] The server periodically checks the information of objects in the refrigerator once a week. This check is performed automatically by the scheduler.

[1615] Step 10:

[1616] The server retrieves all object information from the database and sends a request to the generative AI to generate a week's worth of menus.

[1617] Step 11:

[1618] A generative AI receives the request and generates a week's worth of cooking plans and a list of ingredients needed, including any missing ingredients.

[1619] Step 12:

[1620] The server transmits the generated list of absent objects to an online sales device, which automatically purchases the missing objects.

[1621] Step 13:

[1622] Users can pick up ingredients purchased online at a designated location. By receiving any missing ingredients, users can prepare a week's worth of meals as planned.

[1623] Example 1

[1624] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1625] In today's busy world, users spend a lot of time managing ingredients and planning their cooking. In addition, improper management of food and beverages stored in refrigerators can lead to food waste. Furthermore, it can be cumbersome for users to regularly purchase ingredients needed for cooking, and poor management can make cooking difficult.

[1626] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1627] In this invention, the server includes a detection means for detecting objects in the refrigeration unit, a transmission means for transmitting object information detected by the detection means to a processing device, and a generation means for the processing device to generate a cooking method based on the object information. This enables automatic ingredient management and optimal cooking method suggestions. The server also includes an image generation means for generating an image of the generated cooking method, a means for transmitting the generated cooking method and the image to a user terminal, a means for storing the object information received by the processing device in a database, a means for the processing device to request recipe generation from a generative artificial intelligence model, a means for the generative artificial intelligence model to receive the request and generate a recipe based on the object, and a means for the processing device to transmit the generated recipe to an image generation model and generate an image of the finished dish. This allows users to consistently automate everything from daily ingredient management to cooking method suggestions and food image confirmation. It also reduces food waste and realizes efficient ingredient purchasing.

[1628] The "detection means" is a device installed to detect objects inside the refrigeration device, and may use image recognition technology or radio frequency identification technology.

[1629] The "transmitting means" is a device or system for transmitting object information detected by the detecting means to the processing device.

[1630] The "generation means" is a device or system that allows the processing device to generate a cooking method based on object information.

[1631] "Image generation means" refers to a device or system for generating an image of the generated recipe.

[1632] The "means for transmitting to the user terminal" refers to a device or system for transmitting the generated recipes and images to the terminal used by the user.

[1633] A "processing device" is a device or system for processing object information received from a sensor and storing it in a database.

[1634] A "generative artificial intelligence model" is an artificial intelligence model that generates recipes in natural language based on input prompts.

[1635] An "image generation model" is an artificial intelligence model for generating images based on input text information.

[1636] A "database" is a data management system for storing and managing received object information.

[1637] A "refrigeration device" is a device used to store food and beverages at a constant low temperature.

[1638] "Object information" is data about the names and quantities of food and beverages stored in the refrigerator.

[1639] A "recipe" is information that describes the steps and ingredients for cooking a particular dish.

[1640] This invention is a system that automates the management of food and beverages stored in a refrigerator and suggests effective cooking methods. The system detects objects in the refrigerator and generates cooking instructions and images based on that information, which are then provided to the user. It also has a function that periodically checks the information about the objects in the refrigerator and automatically purchases any missing items online.

[1641] This system mainly uses the following hardware and software:

[1642] 1. Sensors: Image recognition sensors and RFID readers are used to detect objects inside the refrigerator.

[1643] 2. Server: Receives object information from sensors, stores it in a database, and requests recipe and image generation from generative artificial intelligence models and image generation models.

[1644] 3. Generative AI models: For example, using natural language generation models such as GPT-3 to generate recipes based on ingredients.

[1645] 4. Image generation model: For example, a text-to-image generation model such as DALL-E is used to generate images of the generated recipe.

[1646] 5. User device: For example, the generated recipes and images are provided to the user using a smartphone or tablet.

[1647] As a concrete example, consider a case where a user stores two tomatoes, 50g of cheese, a little basil, and bread in a refrigerator. The system operates as follows:

[1648] First, sensors inside the refrigerator detect these ingredients and send object information to a server, which stores this information in a database and manages the overall inventory.

[1649] The server then sends a recipe generation request to the generative artificial intelligence model, such as a prompt like "Generate a recipe using tomatoes, cheese, basil, and bread."

[1650] The generative AI model receives this request and generates a recipe called "Tomato and Basil Bruschetta." The server then sends this generated recipe to the image generation model, requesting it to "generate an image of the bruschetta."

[1651] The image generation model receives the request and generates an image of the bruschetta. The server then sends the generated recipe and image to the user's device. The user then cooks the dish while viewing the recipe and image on the device.

[1652] Furthermore, the server can periodically check the information about the objects in the refrigerator and ask the generative AI model to create a week's worth of menus. Once a list of missing objects is created, the server sends the list to the online sales device, which then automatically carries out the purchasing process.

[1653] In this way, the present invention reduces the burden on the user and enables efficient and effective food ingredient management and cooking.

[1654] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1655] Step 1:

[1656] Sensors scan objects inside the refrigerator

[1657] Input: Current state of ingredients in the refrigerator

[1658] How it works: Image recognition sensors and RFID readers installed inside the refrigerator detect ingredients like tomatoes, cheese, and basil. Each sensor captures image data and RFID tag information for each ingredient.

[1659] Output: A list of ingredients and their amounts (e.g., "2 tomatoes, 50g cheese, a little basil")

[1660] Step 2:

[1661] The server receives object information from the sensor

[1662] Input: A list of ingredient names and quantities sent from the sensor

[1663] Specific operation: The server receives object information sent from the sensor in real time. The received data format is, for example, JSON: {"Object": [{"Name": "Tomato", "Quantity": "2"}, {"Name": "Cheese", "Quantity": "50g"}, {"Name": "Basil", "Quantity": "Small"}]}.

[1664] Output: Received object information data

[1665] Step 3:

[1666] The server stores the received object information in a database.

[1667] Input: Received object information data

[1668] Specific operation: The server analyzes the object information and stores it in a database. The database keeps up-to-date information on the types and quantities of ingredients. In this example, the database stores "2 tomatoes, 50g of cheese, and a little basil."

[1669] Output: Latest object information stored in the database

[1670] Step 4:

[1671] The server sends a recipe generation request to the generative AI based on the object information.

[1672] Input: The latest object information stored in the database

[1673] Specific operation: The server retrieves object information from the database and sends it to the generative AI (e.g., GPT-3) as a recipe generation request. The request is sent in the form of a prompt: "Please generate a recipe using tomatoes, cheese, basil, and bread."

[1674] Output: The prompt sent to the generative AI

[1675] Step 5:

[1676] A generative AI receives the request and generates a recipe based on the object.

[1677] Input: Prompt sent from the server

[1678] How it works: A generative AI (e.g., GPT-3) analyzes the prompt and generates an optimal recipe based on the input object information. For example, the generated recipe might be "Tomato and Basil Bruschetta."

[1679] Output: Generated recipe (text format)

[1680] Step 6:

[1681] The server sends the generated recipe to the image generation model, which generates an image of the finished dish.

[1682] Input: Recipe generated by generative AI

[1683] Specific operation: The server obtains the generated recipe information and sends an image generation request to the image generation model (e.g., DALL-E). The request is sent in the form of, for example, "Please generate an image of bruschetta."

[1684] Output: Generated food images

[1685] Step 7:

[1686] The server sends the generated recipe and image to the user's device.

[1687] Input: Generated recipes and food images

[1688] Specific operation: The server sends the generated recipe text and image file to the user's device so that the user can check it. For example, the user's smartphone will display "Tomato and Basil Bruschetta Recipe" and its image.

[1689] Output: Recipe and image displayed on the user's device

[1690] Step 8:

[1691] The server periodically checks the information about objects in the refrigerator.

[1692] Input: Object information stored in the database

[1693] Specific operation: The server periodically (e.g., every Monday at 9:00 AM) accesses the database and checks the information about objects in the refrigerator.

[1694] Output: Information about the most recent object in the refrigerator that was checked

[1695] Step 9:

[1696] The server requests a week's worth of menus from the generative AI

[1697] Input: Information about the object in the most recent refrigerator that was checked

[1698] Specific operation: To generate a week's worth of menus, the server requests all object information in the database and the number of dishes required from the generative AI. The request is sent in the form of a prompt, such as "Please generate a seven-day menu."

[1699] Output: The prompt sent to the generative AI

[1700] Step 10:

[1701] Generative AI generates a week's worth of meals and creates a list of missing items

[1702] Input: Prompt sent from the server

[1703] Specific operation: A generative AI (e.g., GPT-3) analyzes the prompts and generates a weekly menu, identifying and listing any missing items. For example, a generated menu and a "list of missing items" are created.

[1704] Output: Generated weekly menu and list of missing items

[1705] Step 11:

[1706] The server sends the list of missing objects to the online sales device and purchases the required objects.

[1707] Input: Generated missing object list

[1708] Specific operation: The server sends a list of missing items to the online sales device, which then automatically processes the purchase. For example, a list of "2 tomatoes, 100g of cheese" is sent, and an online order is placed.

[1709] Output: Required objects purchased

[1710] In this way, the entire system flow is completed, allowing users to smoothly carry out a series of processes, from managing ingredients and suggesting cooking methods to automatically purchasing the necessary ingredients.

[1711] (Application example 1)

[1712] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1713] Managing food and beverages in a refrigerator requires manual checking and replenishing, which is a significant time burden for users. Furthermore, since no cooking method suggestions are provided, it is difficult to effectively utilize the ingredients in the refrigerator. Furthermore, there is no system that automatically replenishes ingredients when they run out, so users must manually order them. To solve these issues, there is a need for a system that can automatically manage food and beverages, suggest effective cooking methods, and automatically replenish ingredients when they run out.

[1714] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1715] In this invention, the server includes a detection means for detecting objects in the refrigeration unit, a transmission means for transmitting object information detected by the detection means to a processing device, a generation means for the processing device to generate a cooking method based on the object information, an image generation means for generating an image of the generated cooking method, a means for transmitting the generated cooking method and the image to a user terminal, and a replenishment means for the processing device to generate order information for replenishing ingredients based on the object information and automatically order ingredients through an online service. This enables automated management of food and beverages in the refrigeration unit, suggestions for effective cooking methods, and automatic replenishment of ingredients when they run out.

[1716] A "refrigeration unit" is a device designed to maintain a low internal temperature and is used to preserve food, beverages, etc.

[1717] A "system" is a whole made up of multiple elements that are interrelated and combined to achieve a certain purpose.

[1718] "Detection means" means a device or function, including sensor technology, for accurately identifying and detecting the presence and type of object within the refrigeration unit.

[1719] "Transmitting means" refers to a device or function that includes communication technology for transmitting object information collected by the detecting means to the processing device.

[1720] The "generation means" is a device or function that includes an algorithm or artificial intelligence for automatically creating a cooking method based on object information.

[1721] "Image generation means" refers to a device or function including an artificial intelligence or image generation engine for creating visual food images based on the generated cooking instructions.

[1722] A "user terminal" is a device such as a smartphone or computer that a user uses to receive, check, or operate information provided by the system.

[1723] The "replenishment means" is a function or system for automatically ordering and replenishing missing food or beverages based on object information.

[1724] The "processing device" is a computer or control unit that receives and analyzes the object information transmitted from the detection means and controls the entire system.

[1725] The system to realize this application example utilizes various sensors, processing devices, and communication means built into the refrigeration equipment to automatically manage food and beverages, suggest effective cooking methods, and even automatically replenish ingredients when they run out.

[1726] The server includes a detection means that uses image recognition technology or radio frequency identification technology to detect objects in the refrigeration device. The detection means collects object information using, for example, a camera or an RFID reader. The collected object information is transmitted to the server via a transmission means.

[1727] The server receives the transmitted object information with a processing device and generates an optimal cooking method based on the object information using a generation means, such as an OpenAI generative AI model. Based on the generated cooking method, an image generation engine such as DALL-E is used to generate an image of the cooking method.

[1728] The image of the cooking method and the recipe generated by the image generating means are transmitted to a user terminal via a transmitting means, which may be a smartphone or tablet, and the user can check the cooking method and the image through the user terminal.

[1729] Furthermore, the server periodically checks the information on the items in the refrigerator and generates a cooking plan for the week. Based on the generated cooking plan, it creates a list of items that are in short supply and automatically orders them online. This eliminates the need for users to manually check and order.

[1730] For example, if tomatoes, cheese, and basil are stored in the refrigerator, the server will request a recipe from the generative AI model based on this information. The generative AI model will suggest "Tomato and Basil Bruschetta," and DALL-E will generate an image of it. Furthermore, if the bread needed for the bruschetta is in short supply, it will automatically be ordered from an online service.

[1731] Here are some examples of prompts for generative AI models:

[1732] If you have the following ingredients in your fridge, please suggest some possible dishes.

[1733] object:

[1734] Name: Tomato

[1735] Quantity: 2 pieces

[1736] Name: Cheese

[1737] Amount: 50g

[1738] Name: Basil

[1739] Quantity: small quantity

[1740] This system automates the management of food and beverages in refrigeration units, allowing users to cook efficiently while saving time and effort.

[1741] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1742] Step 1:

[1743] Detect objects inside a refrigeration unit.

[1744] How it works: The server collects data on food and beverages using cameras and RFID readers installed inside the refrigeration units.

[1745] Input: Image data or RFID tag data of objects in the refrigeration unit.

[1746] Data processing: Identifying the type and quantity of objects using image recognition and RFID technology.

[1747] Output: Identified object information (e.g., "2 tomatoes", "50g cheese", "small amount of basil").

[1748] Step 2:

[1749] The detected object information is transmitted to a processing device.

[1750] Specific operations: The server transmits the object information obtained by the detection means to the processing device.

[1751] Input: Identified object information.

[1752] Data calculation: Formatting object information according to the protocol.

[1753] Output: Formatted object information data.

[1754] Step 3:

[1755] Cooking methods are generated based on object information.

[1756] Specific operation: The server sends a cooking method generation request to the generative AI model based on the object information.

[1757] Input: Formatted object information data.

[1758] Data calculation: The generative AI model performs natural language processing and database search to generate optimal recipes based on object information.

[1759] Output: The generated recipe (e.g., "Tomato and Basil Bruschetta").

[1760] Step 4:

[1761] Generate an image of the cooking method.

[1762] Specific operation: The server sends the generated text information of the cooking method to the image generation engine to generate a visual image of the dish.

[1763] Input: The generated text information of the cooking instructions.

[1764] Data calculation: An AI model processes and generates images based on text information.

[1765] Output: The generated food image (e.g. "Image of bruschetta").

[1766] Step 5:

[1767] The generated recipe and image are sent to the user terminal.

[1768] Specific operation: The server sends cooking instructions and images to the user's smartphone or tablet.

[1769] Input: Generated recipes and images.

[1770] Data calculation: Processes data in a format suitable for the user terminal and sends it.

[1771] Output: Cooking instructions and images displayed on the user's device.

[1772] Step 6:

[1773] Regularly check the contents of the refrigerator.

[1774] Specific operation: The server rechecks the information about objects in the refrigerator at set intervals.

[1775] Input: Current object information data.

[1776] Data calculation: Compare with existing information in the database and update new object information.

[1777] Output: Updated object information data.

[1778] Step 7:

[1779] Generate cooking plans for a certain period of time.

[1780] Specific operation: The server sends a request to the generative AI model to generate a cooking plan for a certain period (e.g., one week) based on the collected object information.

[1781] Input: Updated object information data.

[1782] Data calculation: A generative AI model generates a cooking plan that combines multiple recipes based on object information.

[1783] Output: Generated cooking plan (e.g., recipes for a week).

[1784] Step 8:

[1785] Automatically order missing items online.

[1786] Specific operation: The server creates a list of ingredients that are missing based on the generated cooking plan and places an order with the online sales service.

[1787] Input: Generated cooking plan, missing object list.

[1788] Data processing: Generates order information and processes orders through the API of online services.

[1789] Output: Online order confirmation information.

[1790] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1791] This invention relates to a system that manages objects stored in a refrigerator and suggests optimal cooking methods based on the user's emotions. The system detects objects stored in the refrigerator, generates optimal recipes based on that information, and customizes cooking methods by taking the user's emotions into account using an emotion engine. It also has the functionality to automate object management and periodic online purchasing of items that are in short supply.

[1792] (Program overview and processing explanation)

[1793] Ingredient detection and data generation

[1794] 1. A sensor scans the object inside the refrigerator.

[1795] The server activates sensors installed inside the refrigerator that use image recognition technology and RFID tags to detect objects, such as two tomatoes, 50g of cheese, and a small amount of basil.

[1796] 2. The server receives object information from the sensor.

[1797] The sensor sends the detected object information to the server, which stores it in a database. The stored information includes the object name and quantity.

[1798] Recipe Generation and Emotion Recognition

[1799] 3. The server launches an emotion engine that recognizes the user's emotions.

[1800] The emotion engine analyzes user input, voice, and facial expressions to recognize the user's emotional state. For example, if the user is feeling stressed, it will suggest recipes to help them relax.

[1801] 4. The server sends the object information and the user's emotion information to the generative AI.

[1802] The server sends the object information obtained from the database and the user's emotion information recognized by the emotion engine to the generative AI.

[1803] 5. Generative AI generates recipes based on object information and emotional information.

[1804] The generative AI generates optimal recipes based on object information and the user's emotional information. For example, if the user is feeling stressed, it generates a relaxing recipe such as "Tomato and Basil Bruschetta."

[1805] Image generation and information transmission

[1806] 6. The server sends the generated recipe to DALL-E, which is the image generation means.

[1807] The server sends the recipe information received from the generative AI to DALL-E, which then generates an image of the dish based on that recipe.

[1808] 7. DALL-E generates an image based on the recipe and sends it back to the server.

[1809] For example, an "image of bruschetta" is generated and sent back to the server.

[1810] 8. The server sends the generated recipe and image to the user device.

[1811] The server sends the generated recipe text and image files to the user's terminal, where the user can view the proposed recipe and images.

[1812] Regular menu planning and online purchasing

[1813] 9. The server periodically checks the information of objects in the refrigerator once a week.

[1814] The server checks the database for information about objects in the refrigerator at a specific time each week according to a scheduler.

[1815] 10. The server requests a week's worth of menus from the generative AI.

[1816] To generate a week's worth of menus, the server requests information about all objects and the number of dishes required from the generative AI.

[1817] 11. Generative AI generates a week's worth of menus and creates a list of missing items.

[1818] The generative AI creates a weekly menu and identifies missing items and generates a list.

[1819] 12. The server sends the list of missing objects to the online sales device and purchases the required objects.

[1820] The server sends a list of missing objects to an online sales device, which automatically purchases the objects.

[1821] User Examples

[1822] For example, if tomatoes, cheese, basil, and bread are stored in a refrigerator, the user uses the system in the following steps:

[1823] 1. The user starts using the refrigeration unit.

[1824] A user places an object in the refrigerator and a sensor detects this.

[1825] 2. The server receives the object information and stores it in a database.

[1826] The server receives data from the sensors and registers the object information in a database.

[1827] 3. The emotion engine detects the user's emotions.

[1828] The emotion engine analyzes the user's voice and facial expressions and recognizes that the user is seeking relaxation.

[1829] 4. The server requests the generative AI to generate a recipe.

[1830] The server requests a generative AI to generate a recipe based on object information and emotional information, and the AI ​​generates a recipe for "Tomato and Basil Bruschetta."

[1831] 5. The server requests DALL-E to generate an image.

[1832] The server sends the generated recipe to DALL-E, which generates an image of bruschetta.

[1833] 6. The server sends the generated recipe and image to the user device.

[1834] The recipe and images are displayed on the user's terminal, and the user cooks while looking at them.

[1835] 7. The server periodically checks the object information and creates a menu for the week.

[1836] The server periodically checks the object information and requests a week's worth of menus from the generative AI.

[1837] 8. Automatically purchase absent objects online.

[1838] The server transmits a list of missing items to the online sales device, which automatically carries out the purchase procedure.

[1839] In this way, the present invention automates ingredient management, recipe suggestions, menu planning, emotion-based recipe optimization, and online purchasing to support users' busy lives.

[1840] The processing flow will be explained below.

[1841] Step 1:

[1842] The server activates the sensor to detect the object inside the refrigerator. The sensor uses image recognition technology and RFID tags to detect the object (food or drink) inside the refrigerator.

[1843] Step 2:

[1844] The sensor detects information about objects in the refrigerator and sends it to the server. For example, the sensor detects "2 tomatoes," "50g of cheese," and "a little basil."

[1845] Step 3:

[1846] The server stores the object information received from the sensor in a database, including the object name and quantity.

[1847] Step 4:

[1848] The server activates an emotion engine to recognize the user's emotions. The emotion engine analyzes the user's input, voice, and facial expressions to recognize the user's emotional state.

[1849] Step 5:

[1850] The emotion engine analyzes the user's emotions and generates emotion information, for example, recognizing that the user is feeling stressed.

[1851] Step 6:

[1852] The server sends a recipe generation request to the generative AI based on the object information obtained from the database and the emotion information generated by the emotion engine.

[1853] Step 7:

[1854] Generative AI generates optimal recipes based on object and emotional information. For example, it generates a recipe for "Tomato and Basil Bruschetta" to help users relax.

[1855] Step 8:

[1856] The server sends the generated recipe to DALL-E, an image generation means, which generates an image of the dish based on the recipe.

[1857] Step 9:

[1858] DALL-E generates an image of the dish based on the recipe and sends it back to the server. For example, it generates an image of bruschetta.

[1859] Step 10:

[1860] The server sends the generated recipe and image to the user's terminal, where the user can check the proposed recipe and image.

[1861] Step 11:

[1862] The server sets a scheduler to periodically check the object information in the refrigerator, retrieving and checking the object information from the database at a specific time every week.

[1863] Step 12:

[1864] The server retrieves all object information from the database and sends a request to the generative AI to generate a week's worth of menus.

[1865] Step 13:

[1866] Generative AI generates a weekly menu and creates a list of missing items. For example, it creates a menu that includes menus for Monday through Sunday.

[1867] Step 14:

[1868] The server transmits a list of missing objects to an online sales device, which automatically purchases the missing objects.

[1869] Step 15:

[1870] Users can pick up items they have purchased online at a specified location. By having any missing ingredients delivered to them, users can plan their meals for the week.

[1871] Example 2

[1872] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1873] In today's busy lifestyles, managing ingredients and suggesting effective cooking methods requires a great deal of effort. It is also difficult to find appropriate cooking methods that reduce stress and fatigue. Therefore, there is a need for a system that can effectively manage items in a refrigerator, suggest optimal cooking methods based on the user's emotions, and automatically purchase ingredients when they are in short supply.

[1874] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1875] In this invention, the server includes a detection means for detecting objects in the refrigeration unit, a transmission means for transmitting object information detected by the detection means to the processing device, a generation means for generating a cooking method based on the object information, an emotion recognition means for recognizing a user's emotion, a generation means for generating a cooking method based on the object information and emotion information, an image generation means for generating an image of the generated cooking method, and a means for transmitting the generated cooking method and image to a user terminal. This makes it possible to efficiently manage objects in the refrigeration unit and suggest appropriate cooking methods based on the user's emotion, not only facilitating meal preparation but also improving the user's psychological satisfaction. It also enables automatic purchasing of ingredients that are in short supply, further improving convenience.

[1876] "Detection means" refers to a device or system for detecting objects within the refrigeration unit.

[1877] The "transmitting means" refers to a device or system for transmitting object information detected by the detecting means to the processing device.

[1878] The term "processing device" refers to a device or system that generates and manages recipes based on object information received from a transmitting means.

[1879] The "generation means" refers to a device or system for generating a cooking method based on object information and user emotion information.

[1880] "Emotion recognition means" refers to a device or system for recognizing the emotional state of a user.

[1881] "Image generation means" refers to a device or system for generating an image based on the generated recipe.

[1882] The term "user terminal" refers to an apparatus or device for displaying generated recipes and images to a user.

[1883] "Database" refers to a system for storing object information, user emotion information, and other related data.

[1884] "Online sales device" refers to a system that processes orders online to purchase missing objects.

[1885] "Scheduler" refers to a system that manages a schedule for periodically checking information about objects in a refrigeration device.

[1886] The present invention relates to a system that manages items stored in a refrigerator and suggests optimal cooking methods based on the user's emotions. This system utilizes multiple hardware and software components to efficiently and automatically manage ingredients and suggest cooking methods.

[1887] Hardware and Software Configuration

[1888] 1. Sensors: Sensors with image recognition technology and RFID tag readers are placed inside the refrigerator, allowing for accurate detection of objects inside the refrigerator.

[1889] 2. Server: The central part of the system, it centrally manages and processes data from various devices and software, such as sensors, emotion recognition engines, generative AI, and image generation methods.

[1890] 3. Emotion Recognition Engine: Uses software to analyze user input (voice, facial expressions, etc.) and recognize emotions. For example, determining stress or happiness from the user's tone of voice and facial expressions.

[1891] 4. Generative AI: Uses AI models to generate optimal recipes based on object and emotion information. For example, this applies to recipe suggestion systems that use natural language processing.

[1892] 5. Image generation means: Software for generating food images based on the generated recipe. This includes DALL-E and similar image generation models.

[1893] 6. User device: A device (smartphone, tablet, etc.) on which the user checks the recipe and generated images. It displays the data sent from the server.

[1894] 7. Database: Use a system to store object information, user emotion information, recipe information, etc.

[1895] 8. Online Sales Device: A system for automatically purchasing missing items. The purchasing process is carried out using the online store API.

[1896] Specific actions

[1897] 1. Ingredient detection:

[1898] When a user places new ingredients in the refrigerator, the sensor scans them and sends the detection results to the server. For example, if a tomato and cheese are newly added to the refrigerator, image recognition technology and an RFID tag reader will detect this.

[1899] 2. Data storage:

[1900] The server analyzes the received data and stores it in a database. For example, it might register "2 tomatoes, 50g of cheese."

[1901] 3. Emotion Recognition:

[1902] When a user speaks into the smartphone or faces the camera, the emotion recognition engine analyzes this and sends the user's emotional information to the server. For example, it may determine that the user is feeling stressed.

[1903] 4. Generate the recipe:

[1904] The server retrieves object and emotion information from the database and sends it to the generative AI, which then generates the optimal recipe based on the user's emotions. For example, it suggests "relaxing tomato and basil bruschetta."

[1905] 5. Image Generation:

[1906] The server transmits the generated recipe to the image generating means, which generates a related food image, for example, an image of "bruschetta."

[1907] 6. Submit your recipe and images:

[1908] The server sends the generated recipe and images to the user terminal, where the user can check the proposed recipe and images.

[1909] 7. Regular Checks and Purchases:

[1910] The server periodically checks the information of the items in the refrigerator and lists the items that are missing, and automatically purchases the missing items through the online sales device based on the list.

[1911] Specific examples

[1912] Example prompt sentence:

[1913] "What are some relaxing dishes that you would suggest to users when they are feeling stressed?"

[1914] "If I have tomatoes, cheese, and basil in my refrigerator, can you suggest a recipe that works best?"

[1915] In this way, the present invention provides a system that allows users to efficiently manage ingredients and enjoy optimal recipes that suit their own emotional state.

[1916] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1917] Step 1:

[1918] Ingredient detection

[1919] Input: User stores new ingredients in the refrigerator.

[1920] Processing: The server activates sensors (image recognition technology and RFID tag readers) inside the refrigeration unit to scan the object.

[1921] Output: Data of detected ingredients.

[1922] What it does: When a user adds 2 tomatoes, 50g of cheese, and a little basil to the refrigerator, the sensor detects this and sends the data (2 tomatoes, 50g of cheese, a little basil) to the server.

[1923] Step 2:

[1924] Data storage

[1925] Input: Ingredient data sent from the sensor.

[1926] Processing: The server receives the data from the sensors and stores it in a database.

[1927] Output: Ingredient information stored in a database.

[1928] Specific operation: The server analyzes the data received from the sensor (2 tomatoes, 50g of cheese, and a small amount of basil) and registers it in a database.

[1929] Step 3:

[1930] Emotion recognition

[1931] Input: User voice and facial expression data.

[1932] Processing: The server uses an emotion recognition engine to analyze the user's emotions.

[1933] Output: The perceived emotional state of the user.

[1934] Specific operation: When a user speaks into the smartphone or faces the camera, the emotion recognition engine analyzes this and determines that the user is feeling stressed.

[1935] Step 4:

[1936] Recipe Generation

[1937] Input: Ingredient information obtained from the database and emotion information from the emotion recognition engine.

[1938] Processing: The server sends the ingredient information and emotion information to the generative AI and requests it to generate an appropriate recipe.

[1939] Output: Generated recipe information.

[1940] Specific operation: The server sends information about tomatoes, cheese, and basil, as well as the user's stress level, to the generative AI, which then generates a recipe for "Tomato and Basil Bruschetta."

[1941] Step 5:

[1942] Image generation

[1943] Input: Recipe information from a generative AI.

[1944] Processing: The server sends the recipe information to the image generation means (such as DALL-E), which generates an image based on the recipe.

[1945] Output: The generated food image.

[1946] Specific operation: The server sends the recipe information for "Tomato and Basil Bruschetta" received from the generative AI to DALL-E, and generates an image of the bruschetta.

[1947] Step 6:

[1948] Submit recipes and images

[1949] Input: Generated recipe information and images.

[1950] Processing: The server sends the generated recipe and image to the user device.

[1951] Output: Recipe and image displayed on user's device.

[1952] Specific operation: The server sends the generated recipe and image of "Tomato and Basil Bruschetta" to the user's smartphone or tablet, and the user confirms it.

[1953] Step 7:

[1954] Regular checks and purchases

[1955] Input: Ingredient information stored in the database.

[1956] Processing: The server periodically checks the ingredient information in the database using a scheduler and generates a list of missing items.

[1957] Output: A list of missing objects.

[1958] Specific operation: At a set time each week, the server checks the information about the objects in the refrigerator, requests a week's worth of menus from the generative AI, and creates a list of missing objects.

[1959] Step 8:

[1960] Buy Online

[1961] Input: A list of missing objects.

[1962] Processing: The server sends the list of missing objects to the online sales device, which automatically purchases the required objects.

[1963] Output: Purchased object information.

[1964] Specific operation: The server uses an online sales API to automatically order the missing items (e.g., 1 liter of milk, 6 eggs) and complete the purchase.

[1965] (Application example 2)

[1966] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1967] The present invention aims to provide a system that reduces the effort required for managing items in a refrigerator and provides optimal cooking methods according to the user's emotional state. Another objective is to make users' lives more convenient by automatically detecting shortages of ingredients and providing a mechanism for purchasing necessary items online.

[1968] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a detection means for detecting objects in the refrigeration device, a transmission means for transmitting object information detected by the detection means to the processing device, a generation means for the processing device to generate a cooking method based on the object information, a means for the generation means to customize the cooking method based on user emotion information, an image generation means for generating an image of the generated cooking method, and a means for transmitting the generated cooking method and image to a user terminal. This automates the management of objects in the refrigeration device, makes it possible to suggest optimal cooking methods based on the user's emotions, and further enables efficient use of the refrigeration device by automatically purchasing ingredients that are running low, without the user having to perform any cumbersome operations.

[1969] A "refrigeration unit" is a box-shaped electrical device used to keep food and beverages at low temperatures.

[1970] "Object" refers to the specific food or beverage stored within the refrigeration unit.

[1971] "Detection means" refers to devices and techniques for identifying and obtaining information about objects within the refrigeration unit.

[1972] "Transmitting means" refers to a device or technology for transmitting object information acquired by the detecting means to a processing device.

[1973] The "processing device" is a computer device that processes the received object information and generates cooking methods and recipes.

[1974] "Generation means" refers to technology or devices for generating optimal cooking methods and recipes based on object information and user emotional information.

[1975] "Emotion information" is information that represents the emotional state of the user, obtained by analyzing the user's facial expression or voice.

[1976] "Means for customizing" refers to a technology or device in which the generating means changes the cooking method based on the user's emotional information.

[1977] "Image generation means" refers to technology or equipment for generating an image of a dish based on the generated cooking method.

[1978] A "user terminal" is a device for displaying the generated recipes and images, and includes a smartphone, a computer, etc.

[1979] A "generative AI model" is an artificial intelligence model used to generate cooking instructions and images from input data.

[1980] A "prompt sentence" is an input text provided to a generative AI model, and is a sentence that indicates the direction of the content to be generated.

[1981] "Online sales device" refers to a system or device for purchasing missing items via the Internet.

[1982] The present invention is a system that manages objects in a refrigeration device and suggests optimal cooking methods based on the user's emotional information. This system includes multiple steps, such as object detection, information processing, emotion recognition, recipe generation, and image generation. Specific system embodiments are described below.

[1983] Hardware and software used

[1984] Refrigeration unit: An electrical device used to preserve food and beverages.

[1985] Detection methods: Image recognition technology (e.g., OpenCV) and radio frequency identification technology (e.g., RFID tags).

[1986] Transmission method: Data communication using Wi-Fi module.

[1987] Processing unit: A high-performance computer or cloud server.

[1988] Generation method: Cooking instructions are generated using AI technology (e.g., OpenAI GPT-3).

[1989] Emotion Recognition Engine: Facial expression recognition technology (e.g., facial_emotion_recognition) and voice analysis technology.

[1990] Image generation method: AI image generation technology (e.g., DALL-E).

[1991] User devices: smartphones and computers.

[1992] System Operation

[1993] 1. Object detection:

[1994] Objects inside the refrigerator are detected using image recognition and radio frequency identification technology.

[1995] The detected object information is transmitted from the sensor to a processing unit.

[1996] 2. Processing of information:

[1997] The processing device (server) stores the received object information in a database.

[1998] The system periodically checks the information about items in the refrigerator, generates a cooking plan for a certain period of time, and creates a list of items that are missing.

[1999] 3. User emotion recognition:

[2000] The emotion recognition engine analyzes the user's facial expressions and voice to recognize the user's emotional information.

[2001] For example, if the user is feeling stressed, that information is sent to the server.

[2002] 4. Generate the recipe:

[2003] The server generates recipes using a generative AI model based on emotional and object information.

[2004] An example prompt is, "The refrigerator contains the following ingredients: tomatoes, cheese, and basil. The user is feeling stressed. Please suggest a recipe."

[2005] 5. Image Generation:

[2006] Based on the generated recipe information, AI image generation technology is used to generate images of the dish. For example, DALL-E is used to generate an image of "Tomato and Basil Bruschetta."

[2007] 6. Transmission and Display of Information:

[2008] The server transmits the generated recipe and image to the user terminal.

[2009] Suggested recipes and images are displayed on the user's device (smartphone or computer), which can be used as a reference for cooking.

[2010] Specific examples

[2011] If the refrigerator contains tomatoes, cheese, and basil, it is recognized that the user is seeking relaxation.

[2012] The server sends prompts to the generative AI model based on object information and emotion information, generating the optimal recipe: "Tomato and Basil Bruschetta."

[2013] Based on the recipe generated by the generative AI model, DALL-E generates an image of "Tomato and Basil Bruschetta."

[2014] This information is sent to the user's terminal, and the user can cook while looking at the displayed recipe and images.

[2015] In this way, the present invention significantly improves user convenience through the management of objects in a refrigerator, the suggestion of cooking methods based on the user's emotional information, and the automatic purchase of ingredients that are in short supply.

[2016] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2017] Step 1:

[2018] The server detects objects in the refrigerator. Sensors in the refrigerator use image recognition technology or radio frequency identification technology (RFID) to detect objects (e.g., tomatoes, cheese, basil, etc.). The detected object information is sent to the server. The input is the image or data from the RFID tag, and the output is the object information.

[2019] Step 2:

[2020] The server stores the received object information in a database. The object information is sent to the server, which records the information in the database. The database contains the object type, quantity, and detection date and time. The input is the object information, and the output is the updated database.

[2021] Step 3:

[2022] The server starts an emotion recognition engine to recognize the user's emotional information. The emotion recognition engine analyzes facial expression images and voice data from the user's device to identify the user's emotional state (e.g., stress, relaxation, etc.). The input is facial expression images and voice data, and the output is emotional information.

[2023] Step 4:

[2024] The server generates a recipe using a generative AI model based on object information and emotional information. The server generates a prompt sentence and inputs it into the generative AI model to suggest the optimal cooking method. For example, a prompt sentence could be "The following ingredients are in the refrigerator: tomatoes, cheese, and basil. The user is feeling stressed. Please suggest a dish." The input is object information and emotional information, and the output is the generated recipe.

[2025] Step 5:

[2026] The server sends the generated recipe information to an image generation engine to generate a food image. Based on the generated recipe, an AI image generation technology such as DALL-E is used to generate a corresponding food image (e.g., "Tomato and Basil Bruschetta"). The input is the recipe information, and the output is the food image.

[2027] Step 6:

[2028] The server sends the generated recipe and image to the user's device. The generated cooking method and its image are then sent to the user's device, where the user can check the suggested recipe and image on their smartphone or computer. The input is the recipe information and image, and the output is the display on the user's device.

[2029] Step 7:

[2030] The server periodically checks the object information in the refrigeration unit. For example, daily or weekly, the server updates the object information in the database and identifies any missing objects. The input is the existing object information, and the output is a list of missing objects.

[2031] Step 8:

[2032] The server generates a cooking plan for one week and sends a list of missing items to the online sales device. A generative AI model is used to generate recipes for one week, list missing items, and automatically send order information to the online sales device. The input is ingredient information and the number of recipes required, and the output is a list of missing items and order information.

[2033] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2034] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2035] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2036] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2037] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2038] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2039] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2040] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2041] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2042] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2043] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2044] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2045] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2046] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2047] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2048] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2049] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2050] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2051] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2052] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2053] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2054] The following is further disclosed regarding the above embodiment.

[2055] (Claim 1)

[2056] a detection means for detecting an object within the refrigeration device;

[2057] a transmitting means for transmitting object information detected by the detecting means to a processing device;

[2058] a generation means for generating a cooking method based on the object information by the processing device;

[2059] an image generating means for generating an image of the generated cooking method;

[2060] means for transmitting the generated recipes and images to a user terminal;

[2061] A system including:

[2062] (Claim 2)

[2063] The processing device further comprises means for periodically checking information on objects in the refrigeration device, generating a cooking plan for a certain period of time, and generating a list of missing objects;

[2064] 10. The system of claim 1, further comprising means for purchasing the missing objects from an online sales device based on said list.

[2065] (Claim 3)

[2066] 10. The system of claim 1, wherein the detecting means comprises means for detecting objects using image recognition or radio frequency identification techniques.

[2067] "Example 1"

[2068] (Claim 1)

[2069] a detection means for detecting an object within the refrigeration device;

[2070] a transmitting means for transmitting object information detected by the detecting means to a processing device;

[2071] a generation means for generating a cooking method based on the object information by the processing device;

[2072] an image generating means for generating an image of the generated cooking method;

[2073] means for transmitting the generated recipes and images to a user terminal;

[2074] means for storing the object information received by the processing device in a database;

[2075] a means for the processing device to request recipe generation from a generative artificial intelligence model;

[2076] means for the generative artificial intelligence model to receive a request and generate a recipe based on the object;

[2077] a means for transmitting the generated recipe to an image generation model by the processing device to generate an image of the finished dish;

[2078] A system including:

[2079] (Claim 2)

[2080] The processing device further comprises means for periodically checking information on objects in the refrigeration device, generating a cooking plan for a certain period of time, and generating a list of missing objects;

[2081] 10. The system of claim 1, further comprising means for purchasing the missing objects from an online sales device based on said list.

[2082] (Claim 3)

[2083] 10. The system of claim 1, wherein the detecting means comprises means for detecting objects using image recognition or radio frequency identification techniques.

[2084] "Application Example 1"

[2085] (Claim 1)

[2086] a detection means for detecting an object within the refrigeration device;

[2087] a transmitting means for transmitting object information detected by the detecting means to a processing device;

[2088] a generation means for generating a cooking method based on the object information by the processing device;

[2089] an image generating means for generating an image of the generated cooking method;

[2090] means for transmitting the generated recipes and images to a user terminal;

[2091] a replenishment means for generating order information for replenishing ingredients based on the object information by the processing device and automatically ordering ingredients through an online service;

[2092] A system including:

[2093] (Claim 2)

[2094] The processing device further comprises means for periodically checking information on objects in the refrigeration device, generating a cooking plan for a certain period of time, and generating a list of missing objects;

[2095] 10. The system of claim 1, further comprising means for purchasing the missing objects from an online sales device based on said list.

[2096] (Claim 3)

[2097] 10. The system of claim 1, wherein the detecting means comprises means for detecting objects using image recognition or radio frequency identification techniques.

[2098] "Example 2: Combining Emotion Engines"

[2099] (Claim 1)

[2100] a detection means for detecting an object within the refrigeration device;

[2101] a transmitting means for transmitting object information detected by the detecting means to a processing device;

[2102] a generation means for generating a cooking method based on the object information by the processing device;

[2103] emotion recognition means for recognizing an emotion of a user;

[2104] a generation means for generating a cooking method based on object information and emotion information;

[2105] an image generating means for generating an image of the generated cooking method;

[2106] means for transmitting the generated recipes and images to a user terminal;

[2107] A system including:

[2108] (Claim 2)

[2109] The processing device further comprises means for periodically checking information on objects in the refrigeration device, generating a cooking plan for a certain period of time, and generating a list of missing objects;

[2110] 10. The system of claim 1, further comprising means for purchasing the missing objects from an online sales device based on said list.

[2111] (Claim 3)

[2112] 10. The system of claim 1, wherein the detecting means comprises means for detecting objects using image recognition or radio frequency identification techniques.

[2113] "Application example 2 when combining emotion engines"

[2114] (Claim 1)

[2115] a detection means for detecting an object within the refrigeration device;

[2116] a transmitting means for transmitting object information detected by the detecting means to a processing device;

[2117] a generation means for generating a cooking method based on the object information by the processing device;

[2118] a means for customizing a cooking method based on the user's emotion information;

[2119] an image generating means for generating an image of the generated cooking method;

[2120] means for transmitting the generated recipes and images to a user terminal;

[2121] ...

Claims

1. a detection means for detecting an object within the refrigeration device; a transmitting means for transmitting object information detected by the detecting means to a processing device; a generation means for generating a cooking method based on the object information by the processing device; an image generating means for generating an image of the generated cooking method; means for transmitting the generated recipes and images to a user terminal; A system including:

2. The processing device further comprises means for periodically checking information on objects in the refrigeration device, generating a cooking plan for a certain period of time, and generating a list of missing objects; 2. The system of claim 1, further comprising means for purchasing the missing objects from an online sales device based on said list.

3. 10. The system of claim 1, wherein said detecting means comprises means for detecting objects using image recognition or radio frequency identification techniques.

Images