system

Abstract

Provide a system. 【Solution means】 Means for acquiring an image of food ingredients by an image acquisition means, Means for analyzing the acquired image to identify food ingredients, Means for registering information on the identified food ingredients in a database, Means for managing the expiration date of the registered food ingredients, Means for proposing recipes based on food ingredients with a near expiration date, Means for notifying the user of the proposed recipe information, Means for periodically photographing the food ingredients in the cooling device by a household automatic machine and executing each stage of processing, Means for guiding the user through cooking procedures using a voice support device, A system including the above.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Food loss is a major social and economic issue, and in particular, over-purchase of food ingredients in households and waste due to expiration dates occur frequently. Such situations are caused by the complexity of managing the expiration dates of food ingredients or insufficient cooking plans. Therefore, there is a need for a system that can easily and efficiently manage food ingredients in households, appropriately utilize them before they reach their expiration dates, and reduce food loss.

Means for Solving the Problems

[0005] This invention utilizes images of food items in a refrigerator taken with a terminal device to automatically recognize the food items using image analysis, and records and manages the food item information in a database. Furthermore, it identifies food items nearing their expiration date and notifies the user by suggesting related recipes. Through this system, users can visually understand the current state of their food items, enabling planned consumption and reducing food waste.

[0006] "Image acquisition means" refers to a device or function that uses a camera or sensor to capture video data of food ingredients.

[0007] "Means for identifying food ingredients" refers to a function or algorithm that analyzes acquired images to recognize and classify specific food ingredients.

[0008] "Means of registering in a database" refers to an information storage system for storing and managing information on identified food ingredients.

[0009] A "means of managing expiration dates" refers to a system that tracks the expiration dates of food items registered in a database and monitors them based on certain conditions.

[0010] "Methods for suggesting recipes" refers to a function that searches for relevant cooking procedures and ingredient lists based on specified conditions and recommends them to the user.

[0011] "Means of notifying users" refers to a system that generates alerts or messages and sends the information to the user's device to inform them. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0013] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described according to the accompanying drawings.

[0014] First, the language used in the following description will be explained.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0018] In the following embodiments, the numbered communication I / F (Interface) is an interface that includes a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), and the like.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention is a system that begins with the user taking a picture of the food inside the refrigerator using their own device. The user takes an image of the food stored in the refrigerator using a smartphone or a dedicated camera device, and the device uploads this image to a server in the cloud.

[0034] The server analyzes received images using AI image recognition technology and automatically identifies the ingredients within the images. The server registers this identified data in a database, storing it along with the expiration date entered by the user for each ingredient. The information in the database is updated periodically, and the server uses an algorithm to identify ingredients that are nearing their expiration date.

[0035] The server searches a recipe database based on ingredients nearing their expiration date and selects a suitable recipe. In this process, the server determines the optimal recipe, considering factors such as ingredient combinations, cooking procedures, and preparation time. The server then generates a notification message containing the selected recipe information and information about ingredients nearing their expiration date, and sends it to the user's device.

[0036] The device displays received notifications to the user and suggests available recipes. The user checks the notifications and cooks according to the recipes, efficiently consuming ingredients. When cooking is complete, the user updates the information on the ingredients consumed using the device, keeping the database up-to-date.

[0037] As a concrete example, suppose a user takes a photo of the inside of their refrigerator, and the server identifies "chicken," "spinach," and "carrots." The server checks the expiration dates of these ingredients and detects that the chicken is nearing its expiration date. Based on this, the server suggests a recipe for "Creamy Chicken and Spinach Stew" and notifies the user of the recipe. By receiving this suggestion, the user can use the ingredients efficiently and prevent food waste.

[0038] The following describes the processing flow.

[0039] Step 1:

[0040] The user takes a photo of the food items inside the refrigerator using their device. It is recommended to take the photo in sufficient light so that the food items are clearly visible.

[0041] Step 2:

[0042] The device uploads images taken by the user to a cloud server. This upload is performed via an internet connection.

[0043] Step 3:

[0044] The server receives the uploaded image and sends it to an AI image analysis module. There, machine learning algorithms are used to recognize the food items in the image and identify their names.

[0045] Step 4:

[0046] The server records identified food ingredient information in a database. For each ingredient, the expiration date previously registered by the user is also stored in the database.

[0047] Step 5:

[0048] The server periodically scans the database to check the expiration dates of ingredients. It extracts ingredients that are nearing their expiration date and creates a specific list.

[0049] Step 6:

[0050] The server searches the recipe database based on the expiration date list. It selects the optimal recipe by considering factors such as ingredient combinations, user preferences, and cooking time.

[0051] Step 7:

[0052] The server generates a notification message containing the selected recipe. This notification includes the name of the ingredient that is nearing its expiration date and details of the recipe that uses that ingredient.

[0053] Step 8:

[0054] The server sends the notification message to the user's device and delivers it as a push notification.

[0055] Step 9:

[0056] The device displays the received notification to the user. Here, the user can review the contents of the suggested recipe.

[0057] Step 10:

[0058] The user checks the notification and cooks according to the provided recipe. After cooking is complete, they update the information on the ingredients consumed from their device to keep the database information up to date.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] Inadequate inventory management and expiration date tracking of essential goods in storage areas can lead to unnecessary waste and excess inventory. Furthermore, a lack of efficient means to consume essential goods nearing their expiration date contributes to food waste. Solving these problems, streamlining the management of essential goods, and minimizing waste are crucial.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes means for acquiring images of daily necessities in a storage location using a camera, means for analyzing the acquired images using artificial intelligence technology to identify the daily necessities, and means for registering information about the identified daily necessities in a storage device. This makes it possible to automatically grasp the inventory status of daily necessities and propose an appropriate consumption plan based on their expiration dates.

[0064] "Photography device" refers to hardware used to acquire images of storage locations, including those containing essential goods.

[0065] "Daily necessities" refer to consumable items needed in daily life.

[0066] "Artificial intelligence technology" refers to advanced algorithms and models used by computers to analyze images and data, specifically deep learning models.

[0067] A "memory device" refers to a data storage means for registering and retaining information about identified essential goods and their expiration dates.

[0068] "Expiration date" refers to information indicating the period during which essential goods can be consumed.

[0069] "Cooking instructions" refer to information that includes specific consumption methods and cooking procedures, based on essential goods nearing their expiration date.

[0070] "Notification" refers to messages or alerts used by a server to transmit relevant information to a user.

[0071] A "generative artificial intelligence model" refers to a machine learning model that learns patterns and features from data in order to perform a specific task.

[0072] A "short message notification" refers to a simple message format that instantly conveys the minimum necessary information to the user.

[0073] This invention is a system that supports users in managing and efficiently consuming their daily necessities. The system aims to minimize waste of daily necessities by primarily acquiring and analyzing images of these necessities and providing recommended cooking instructions based on that information.

[0074] First, users use their smartphones or dedicated cameras to photograph essential household items in storage areas such as refrigerators and pantries. The device then acquires high-resolution image data and sends it to a cloud server.

[0075] Upon receiving the acquired images, the server uses artificial intelligence technology for image analysis to identify the food items. In this process, machine learning frameworks such as TENSORFLOW® and PyTorch are used, with deep learning models supporting the identification of everyday necessities. This allows for the identification of specific types of food items.

[0076] The server stores the identification results in storage and manages them along with the expiration date information for each essential item. Database management systems such as MySQL® or PostgreSQL are used for this information management. Based on this data, the server identifies essential items that are nearing their expiration date and generates optimal cooking instructions by comparing the recorded data with the recipe database.

[0077] The selected cooking instructions are notified from the server to the terminal. The terminal displays the notification to the user in the form of a pop-up or alert, offering options tailored to the user's lifestyle. For example, a prompt such as "Please suggest the best recipe based on the expiration dates of chicken, spinach, and carrots" is input into the AI ​​model, and based on the received information, a specific cooking method is presented.

[0078] The user consumes daily necessities according to the cooking instructions received, and then updates the information about the ingredients actually consumed via their terminal. The updated information is sent back to the server, and the database is kept up-to-date.

[0079] Through this series of processes, users can efficiently manage their daily necessities and reduce food waste.

[0080] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0081] Step 1:

[0082] Users photograph essential household items within the storage area using their smartphones or dedicated cameras. Image data from the camera device is obtained as input. This image needs to be clear, with adjustments to brightness and focus, for later analysis.

[0083] Step 2:

[0084] The device uploads the captured image to a server in the cloud. The input is an image file, and the output is the image data stored in cloud storage. During this process, data is transmitted appropriately based on the data transfer speed and security protocol (e.g., HTTPS).

[0085] Step 3:

[0086] The server receives uploaded images and analyzes them using a generative AI model. The input is image data from cloud storage. This AI model (e.g., a TensorFlow model) identifies the types and characteristics of everyday necessities within the images. The output is category information for the identified everyday necessities.

[0087] Step 4:

[0088] The server registers the identification results in a storage device (database) and manages the expiration dates of each essential item. The inputs are information on the identified essential items and expiration date information provided by the user. SQL queries are used to insert the information into the database, and the output is an updated database.

[0089] Step 5:

[0090] The server identifies essential goods nearing their expiration date based on information in the database. The input is the entire database data. Expiration dates are calculated using date calculations, and the output is a list of essential goods nearing their expiration date.

[0091] Step 6:

[0092] The server searches a recipe database based on the expiration dates of essential goods and selects the most suitable cooking instructions. The input consists of a list of essential goods nearing their expiration date and the recipe database. Considering the combination of ingredients and cooking procedures, the server outputs the selected cooking instructions.

[0093] Step 7:

[0094] The server sends a notification message containing the generated cooking instructions to the terminal. The input is selected cooking instruction information. A network protocol is used for transmission to the terminal, and the output is the user terminal receiving the notification.

[0095] Step 8:

[0096] The device displays received notifications to the user. These notifications include cooking instructions and information about essential goods nearing their expiration date. The input is the notification message, and the output is displayed on the user interface.

[0097] Step 9:

[0098] The user consumes daily necessities according to the suggested cooking instructions and updates their consumption status via a terminal after consumption. The user's consumption history is used as input, and the updated consumption data is sent back to the server as output. The server reflects this data in its storage device, keeping the database up to date.

[0099] (Application Example 1)

[0100] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0101] In modern households, managing food stored in refrigeration systems is crucial, but there is a lack of systems that effectively manage expiration dates and efficiently utilize food. Furthermore, manual food management is time-consuming and contributes to food waste. Therefore, there is a need for automated household appliances to properly manage food and automatically suggest cooking plans based on expiration dates.

[0102] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0103] In this invention, the server includes means for acquiring images of food ingredients using image acquisition means, means for analyzing the acquired images to identify the food ingredients, and means for registering information about the identified food ingredients in a database. This makes it possible to efficiently manage food ingredients in a cooling device and automate the suggestion of optimal recipes that take expiration dates into consideration.

[0104] "Image acquisition means" refers to a means for taking images of food stored in a cooling device and acquiring the digital data thereof.

[0105] "Means for identifying food ingredients" refers to methods for analyzing acquired image data and automatically identifying food ingredients.

[0106] "Methods for registering in a database" refers to methods for recording information about identified ingredients and storing it in a database for searching and updating as needed.

[0107] "Means for managing expiration dates" refers to a means of monitoring the expiration dates of registered food items and sending notifications when the expiration date is approaching.

[0108] "Methods for suggesting recipes" refer to methods that search for cooking methods based on ingredients nearing their expiration date and present the optimal cooking procedure.

[0109] "Means of notifying the user" refers to methods of displaying the suggested recipe information on the user's device or the screen of a home appliance, or communicating it via voice.

[0110] "Household automated machinery" refers to automated devices used in a home environment, specifically for managing ingredients and assisting with cooking.

[0111] A "voice support device" is a device that provides guidance and instructions to the user via voice, and is used to assist with cooking procedures.

[0112] The system for realizing this invention includes a household automated machine, a cloud server, and a user's portable terminal. At the heart of the system is a program for photographing and managing food items inside a cooling device. The household automated machine periodically acquires images of the conditions inside the cooling device using a camera. These images are transmitted via a network to a server in the cloud.

[0113] The server analyzes received images using image analysis software and a generative AI model to identify the type of food ingredient. The identified food ingredient information is stored in a database, and the expiration date is also recorded. The database has a function to manage the expiration dates of registered food ingredients, and as the expiration date approaches, it effectively manages it using expiration date management methods.

[0114] When ingredients nearing their expiration date are detected, the server searches a recipe database to find the optimal cooking method and selects a cooking method through a system that suggests recipes. The selected recipe information is notified to the user via a notification system, appearing on the user's mobile device or the display of a home-use automated appliance. It is also possible to guide the user directly through the cooking method using a voice assistance device.

[0115] For example, if a user enters the prompt "Please tell me a recommended recipe for tonight's dinner," the system will provide a suitable recipe based on the information about the ingredients in the cooling unit. This system reduces food waste and enables more efficient cooking.

[0116] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0117] Step 1:

[0118] The home-use automated machine uses a built-in camera to photograph food items inside the cooling unit. The input is visual information from inside the cooling unit, and the output is digital image data. This image data is transmitted to a cloud server via a network.

[0119] Step 2:

[0120] The server inputs the received image data into the AI ​​image analysis system. Using a generative AI model, the system analyzes the image data and identifies the food items within the image. The input is the transmitted image data, and the output is information about the identified food items. Processing this information prepares it for registration in the database.

[0121] Step 3:

[0122] The server records identified food information and its associated expiration date in a database. Input is the identified food information and expiration date data, while output is the food information stored in the database. This allows for efficient management of the food's condition.

[0123] Step 4:

[0124] The system periodically checks the expiration date information in the database to identify ingredients nearing their expiration date. The input consists of ingredient and expiration date information stored in the database. The identified ingredient information is then prepared for use in the next step.

[0125] Step 5:

[0126] The server searches a recipe database based on ingredients nearing their expiration date. The input is information about ingredients nearing their expiration date, and the output is the optimal recipe information using these ingredients. This process includes recipe selection that takes into account the type and quantity of ingredients, the user's available cooking time, and other factors.

[0127] Step 6:

[0128] The system notifies the user's device or home-use automated appliances of the found recipe information. The input is the selected recipe information, and the output is a visual or audio notification to the user. Home-use automated appliances can use voice assistance devices to provide voice guidance for the cooking procedure.

[0129] Step 7:

[0130] The user cooks according to the suggested recipe. Once cooking is complete, the user reports information about the ingredients consumed to the server via their terminal. The input is the information about the ingredients consumed entered by the user after cooking, and the output is the latest database information. This update ensures that the ingredient list is always accurate and up-to-date.

[0131] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0132] This invention is a system for efficiently managing food stored in a user's refrigerator and consuming the food before its expiration date. This system begins by recognizing food items using images taken by a terminal. The terminal takes pictures of the food items in the refrigerator and uploads the images to a server. The server uses image analysis technology to identify each food item and registers it in a database.

[0133] Furthermore, the present invention is characterized by its incorporation of an emotion engine. The user's terminal or server uses cameras and sensors to analyze the user's emotions from their facial expressions and voice, and uses this information to determine the user's current emotional state. This emotional information is used in the recipe suggestion process to reflect the user's cooking motivation and preferences. For example, if the user is feeling stressed, it is possible to suggest a recipe that can be prepared quickly and easily.

[0134] Furthermore, the server stores the user's emotional history and uses this to customize future recipe suggestions based on past emotional patterns. In this way, the system can provide personalized recipes that match the user's preferences and emotional state. For example, if the system determines that the user is in a relaxed state, it might suggest a dish that can be enjoyed over a longer period of time.

[0135] The combination of these elements allows users to receive recipes best suited to their emotional state, effectively contributing to the reduction of food waste. The system also supports users in cooking at the optimal time by providing timely notifications based on the expiration dates of ingredients.

[0136] The following describes the processing flow.

[0137] Step 1:

[0138] The user takes a picture of the food in the refrigerator with their device. The device adjusts the brightness to the appropriate level to ensure the image is clear while taking the picture.

[0139] Step 2:

[0140] The user uploads the captured image from their device to the server. The device instantly transmits the image via the internet connection.

[0141] Step 3:

[0142] The server uses AI image recognition technology to analyze the received images. Here, it identifies the types of ingredients and extracts their names.

[0143] Step 4:

[0144] The server registers the recognized food ingredient information in the database. The database also stores the expiration date of each ingredient.

[0145] Step 5:

[0146] To recognize the user's emotions, the device uses its built-in camera and microphone to capture facial expressions and voice. This prepares the device for analyzing the user's emotional state.

[0147] Step 6:

[0148] The server uses the latest emotion recognition algorithms to determine the user's emotions based on the emotional information sent from the terminal. This allows it to evaluate the user's current mood and state.

[0149] Step 7:

[0150] The server identifies ingredients nearing their expiration date and selects an appropriate recipe, taking emotional information into account. For example, if the user is tired, it will choose an easy-to-make recipe.

[0151] Step 8:

[0152] The server provides optimal suggestions to the user based on the emotions associated with the selected recipe. These suggestions are emotionally sensitive and customized to enhance the user experience.

[0153] Step 9:

[0154] The server sends the final recipe suggestion as a notification to the user's device. A list of required ingredients is provided along with detailed recipe information.

[0155] Step 10:

[0156] The device displays received notifications to the user. The user cooks according to the suggested recipe and updates the consumed ingredients from the device to record them in the database.

[0157] (Example 2)

[0158] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0159] To address the inefficiencies of food management and food waste in households, it is necessary to accurately understand the condition of individual ingredients and consume them effectively before their expiration date. Furthermore, a challenge is the decline in cooking motivation due to the lack of personalized suggestions that take into account the user's emotional state.

[0160] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0161] In this invention, the server includes means for capturing an image of an object with an image acquisition device, means for analyzing the acquired image to identify the object, and means for registering information about the identified object in a storage device. As a result, the user is offered processing methods based on objects nearing their expiration date, enabling consumption at the optimal time. Furthermore, by detecting the user's emotions and selecting a processing method based on that information, it becomes possible to offer suggestions tailored to individual preferences and circumstances.

[0162] An "image acquisition device" is a device that has the function of capturing images of objects and recording them as digital data.

[0163] "Means for identifying objects" refers to technologies that analyze captured image data to identify the type and characteristics of an object.

[0164] A "memory device" is a device or system that stores information about an identified object and allows it to be retrieved as needed.

[0165] "Shelf life" refers to information indicating the period during which an object is considered usable while maintaining its quality.

[0166] "Real-time notification" is a function that provides information to users immediately based on pre-set conditions.

[0167] "Means of detecting emotions" refers to technologies that analyze a user's physiological and emotional state using cameras, sensors, and other means.

[0168] "Means for suggesting processing methods" refers to a function that, based on pre-registered information and the current situation, indicates to the user the optimal way to use an object.

[0169] This invention is a system that streamlines object management in users' homes and ensures appropriate consumption within the expiration date. This system provides highly personalized suggestions by utilizing an image acquisition device, object recognition technology, a storage device, and user emotion analysis technology.

[0170] The device's role is to acquire images of objects inside the refrigerator taken by the user. The images are uploaded to the server in real time, and image analysis software such as Google Cloud Vision API is used to identify the type and number of objects.

[0171] The server registers the analysis results in storage and manages the expiration date for each object. This information serves as foundational data for proposing individually optimized processing methods, taking into account the user's emotional state. User emotional analysis is performed using data acquired through the terminal's camera and microphone, and is carried out using Microsoft® Azure® sentiment analysis APIs, etc. Cooking methods and ingredient consumption methods are selected according to the user's emotional state.

[0172] For example, when a user takes a picture, the prompt might say, "Please take a picture of the food. Analysis will begin," and during emotion analysis, the prompt might say, "Please show your face to the camera." In this way, users are automatically supported in optimizing the management and consumption of their food ingredients.

[0173] For example, when a user is feeling stressed, the server can suggest a dish that is easy to prepare. Conversely, if the user is relaxed, it can suggest a dish that takes longer to prepare. This allows for a service tailored to the user's individual state and preferences, minimizing food waste.

[0174] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0175] Step 1:

[0176] The user takes an image of an object inside the refrigerator using the device's camera. The image data is saved on the device as input. The device then sends this image data to the server. In this step, the user takes an image according to the prompt message "Please take a picture of the food. Analysis will begin."

[0177] Step 2:

[0178] The server receives image data sent from the terminal. Using the received image data as input, it identifies objects using the Google Cloud Vision API and extracts their types and characteristics. The output is data containing the types and numbers of objects. The server then registers this identification data in its storage device.

[0179] Step 3:

[0180] The server manages the expiration dates of each object based on the registered object data. It retrieves expiration date data from the storage device as input and identifies objects that are nearing their expiration date. A list of objects nearing their expiration date is generated as output. The server then uses this list to prepare notifications to the user.

[0181] Step 4:

[0182] The device acquires the user's emotional state through its camera or microphone. Based on the acquired audio and image data, the server analyzes the user's emotions using Microsoft Azure's Sentiment Analysis API. Data indicating the emotional state is generated as output.

[0183] Step 5:

[0184] The server integrates the object's expiration date with the user's emotional state and uses a generative AI model to select the optimal processing method. The input is expiration date information and emotional data, and the output is a proposed recipe or processing method. The server then prepares to notify the terminal of this information. In this step, if the user's emotional state is stress, a simple recipe is selected.

[0185] Step 6:

[0186] The terminal notifies the user of the processing instructions received from the server. The user then checks the suggested information in response to the prompt "See recommended recipes." Specifically, an interface is provided that allows the user to check the recipes and begin cooking.

[0187] (Application Example 2)

[0188] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0189] Modern households are required to efficiently manage ingredients and consume them without waste amidst busy lifestyles. However, it is difficult for consumers to keep track of what they have in their refrigerators and find appropriate recipes, leading to food waste. Furthermore, the lack of personalized suggestions tailored to the user's emotional state makes it difficult to maintain motivation for cooking. To solve these problems, a system is needed that can simultaneously manage ingredients and provide personalized support based on the user's emotions.

[0190] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0191] In this invention, the server includes means for acquiring images of ingredients using image acquisition means, means for acquiring the user's facial expressions or voice using sensors and analyzing their emotional state, and means for formulating a recipe suitable for the user based on ingredients nearing their expiration date. This enables not only efficient management of ingredients in the refrigerator but also personalized recipe suggestions tailored to the user's emotional state.

[0192] "Image acquisition means" refers to technology that uses cameras or sensors to acquire images of food items inside a refrigerator and transmit them to a system.

[0193] "Food ingredient identification means" refers to a technology that analyzes acquired images and individually recognizes food ingredients that can be stored or cooked.

[0194] A "database registration method" is a technology that records information about identified food ingredients in a database within the system to prepare for future data retrieval.

[0195] A "shelf-life management system" is a system function that tracks the expiration dates of registered food items and sends notifications when the expiration date is approaching.

[0196] A "recipe suggestion method" is a technology that generates and suggests recipes to facilitate cooking based on the expiration date of ingredients and the user's emotional state.

[0197] "Emotional analysis methods" refer to technologies that use sensors to capture a user's facial expressions and voice, analyze them, and determine their current emotional state.

[0198] "Push notification" refers to electronic notification technology used to inform users in real time about information such as the expiration date of ingredients or selected recipes.

[0199] A system for carrying out this invention includes a user terminal, a server, and sensors for performing sentiment analysis.

[0200] The user's device uses its camera to capture images of food items in the refrigerator and uploads them to a server. The server uses image analysis software such as OpenCV to identify the food items and registers the information in a database. This database records the type of food item and its expiration date. The system also has a function to notify the user via push notification when the expiration date of the food items is approaching.

[0201] Furthermore, emotion analysis is performed using software such as Amazon Rekognition, which captures the user's facial expressions or voice using sensors. This emotion data is sent to a server, where the user's emotional state is determined.

[0202] Using generative AI models such as ChatGPT®, the server generates recipes considering ingredient information and the user's emotional state. These recipes are provided as feedback tailored to the user's individual state. For example, if the user is tired, the system suggests an easy-to-make dish. Conversational recipe suggestions are also possible, improving the user experience.

[0203] For example, if a user living alone enters the prompt "I have eggs, butter, and milk" via their smartphone, the system can suggest a recipe such as "Since you seem to want to relax, why not try making an omelet?"

[0204] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0205] Step 1:

[0206] The user's device takes pictures of the food items inside the refrigerator using its camera and sends the image data to the server. The input is the images of the food items, and the output is the image data sent to the server. This step includes the user using the device to check the contents of the refrigerator.

[0207] Step 2:

[0208] The server analyzes the received image data using OpenCV to identify the type and quantity of ingredients. The input is image data sent from the user's terminal, and the output is the identified ingredient information. At this time, the information obtained through image analysis is recorded in a database.

[0209] Step 3:

[0210] After ingredient information is registered in the database, the server tracks and manages the expiration date and sends a push notification to the user as the expiration date approaches. The input is the ingredient information registered in the database, and the output is the push notification. By receiving this notification, users can plan how to use the ingredients before they expire.

[0211] Step 4:

[0212] The user's device or attached sensors capture the user's facial expressions and voice, and transmit this data to the server. The input is data about the user's emotional state, and the output is the analyzed emotional state. This step includes actions to acquire the emotional data.

[0213] Step 5:

[0214] The server uses Amazon Rekognition to analyze emotional data and determine the user's emotional state. The input is facial or voice data sent by the user, and the output is the result of the determination of the user's emotional state. This information is used for recipe selection in the next step.

[0215] Step 6:

[0216] The server uses a generative AI model to generate recipes based on the user's emotional state and ingredient information. The input is the emotional state and identified ingredient information, and the output is a individually customized recipe. This step involves the AI ​​formulating a recipe suitable for the user.

[0217] Step 7:

[0218] The user's device notifies the user of recipes received from the server and suggests them as cooking options. The input is recipe information sent from the server, and the output is the recipe notified to the user. Upon receiving this notification, the user can then plan their next meal.

[0219] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0220] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0221] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0222] [Second Embodiment]

[0223] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0224] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0225] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0226] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0227] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0228] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0229] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0230] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0231] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0232] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0233] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0234] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0235] This invention is a system that begins with the user taking a picture of the food inside the refrigerator using their own device. The user takes an image of the food stored in the refrigerator using a smartphone or a dedicated camera device, and the device uploads this image to a server in the cloud.

[0236] The server analyzes received images using AI image recognition technology and automatically identifies the ingredients within the images. The server registers this identified data in a database, storing it along with the expiration date entered by the user for each ingredient. The information in the database is updated periodically, and the server uses an algorithm to identify ingredients that are nearing their expiration date.

[0237] The server searches a recipe database based on ingredients nearing their expiration date and selects a suitable recipe. In this process, the server determines the optimal recipe, considering factors such as ingredient combinations, cooking procedures, and preparation time. The server then generates a notification message containing the selected recipe information and information about ingredients nearing their expiration date, and sends it to the user's device.

[0238] The device displays received notifications to the user and suggests available recipes. The user checks the notifications and cooks according to the recipes, efficiently consuming ingredients. When cooking is complete, the user updates the information on the ingredients consumed using the device, keeping the database up-to-date.

[0239] As a concrete example, suppose a user takes a photo of the inside of their refrigerator, and the server identifies "chicken," "spinach," and "carrots." The server checks the expiration dates of these ingredients and detects that the chicken is nearing its expiration date. Based on this, the server suggests a recipe for "Creamy Chicken and Spinach Stew" and notifies the user of the recipe. By receiving this suggestion, the user can use the ingredients efficiently and prevent food waste.

[0240] The following describes the processing flow.

[0241] Step 1:

[0242] The user takes a photo of the food items inside the refrigerator using their device. It is recommended to take the photo in sufficient light so that the food items are clearly visible.

[0243] Step 2:

[0244] The device uploads images taken by the user to a cloud server. This upload is performed via an internet connection.

[0245] Step 3:

[0246] The server receives the uploaded image and sends it to an AI image analysis module. There, machine learning algorithms are used to recognize the food items in the image and identify their names.

[0247] Step 4:

[0248] The server records identified food ingredient information in a database. For each ingredient, the expiration date previously registered by the user is also stored in the database.

[0249] Step 5:

[0250] The server periodically scans the database to check the expiration dates of ingredients. It extracts ingredients that are nearing their expiration date and creates a specific list.

[0251] Step 6:

[0252] The server searches the recipe database based on the expiration date list. It selects the optimal recipe by considering factors such as ingredient combinations, user preferences, and cooking time.

[0253] Step 7:

[0254] The server generates a notification message containing the selected recipe. This notification includes the name of the ingredient that is nearing its expiration date and details of the recipe that uses that ingredient.

[0255] Step 8:

[0256] The server sends the notification message to the user's device and delivers it as a push notification.

[0257] Step 9:

[0258] The device displays the received notification to the user. Here, the user can review the contents of the suggested recipe.

[0259] Step 10:

[0260] The user checks the notification and cooks according to the provided recipe. After cooking is complete, they update the information on the ingredients consumed from their device to keep the database information up to date.

[0261] (Example 1)

[0262] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0263] Inadequate inventory management and expiration date tracking of essential goods in storage areas can lead to unnecessary waste and excess inventory. Furthermore, a lack of efficient means to consume essential goods nearing their expiration date contributes to food waste. Solving these problems, streamlining the management of essential goods, and minimizing waste are crucial.

[0264] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0265] In this invention, the server includes means for acquiring images of daily necessities in a storage location using a camera, means for analyzing the acquired images using artificial intelligence technology to identify the daily necessities, and means for registering information about the identified daily necessities in a storage device. This makes it possible to automatically grasp the inventory status of daily necessities and propose an appropriate consumption plan based on their expiration dates.

[0266] "Photography device" refers to hardware used to acquire images of storage locations, including those containing essential goods.

[0267] "Daily necessities" refer to consumable items needed in daily life.

[0268] "Artificial intelligence technology" refers to advanced algorithms and models used by computers to analyze images and data, specifically deep learning models.

[0269] A "memory device" refers to a data storage means for registering and retaining information about identified essential goods and their expiration dates.

[0270] "Expiration date" refers to information indicating the period during which essential goods can be consumed.

[0271] "Cooking instructions" refer to information that includes specific consumption methods and cooking procedures, based on essential goods nearing their expiration date.

[0272] "Notification" refers to messages or alerts used by a server to transmit relevant information to a user.

[0273] A "generative artificial intelligence model" refers to a machine learning model that learns patterns and features from data in order to perform a specific task.

[0274] A "short message notification" refers to a simple message format that instantly conveys the minimum necessary information to the user.

[0275] This invention is a system that supports users in managing and efficiently consuming their daily necessities. The system aims to minimize waste of daily necessities by primarily acquiring and analyzing images of these necessities and providing recommended cooking instructions based on that information.

[0276] First, users use their smartphones or dedicated cameras to photograph essential household items in storage areas such as refrigerators and pantries. The device then acquires high-resolution image data and sends it to a cloud server.

[0277] Upon receiving the acquired images, the server uses artificial intelligence technology for image analysis to identify the food items. TensorFlow and PyTorch are used as machine learning frameworks, with deep learning models supporting the identification of everyday necessities. This allows for the identification of specific types of food items.

[0278] The server stores the identification results in a storage device and manages them together with the expiration date information of each daily necessity. For this information management, a database management system such as MySQL or PostgreSQL is used. Based on this data, the server identifies daily necessities with approaching expiration dates and generates optimal cooking instructions by collating the recorded data with the recipe database.

[0279] The selected cooking instructions are notified from the server to the terminal. The terminal displays the notification to the user in the form of a pop-up or alert and provides options according to the user's lifestyle. For example, a prompt sentence such as "Please propose an optimal recipe based on the expiration dates of chicken, spinach, and carrots" is input into the AI model, and specific cooking methods are presented based on the received information.

[0280] The user consumes the daily necessities according to the received cooking instructions and then updates the information on the actually consumed food ingredients through the terminal. The updated information is sent to the server again, and the contents of the database are brought up to date.

[0281] Through this series of processes, it is possible for the user to efficiently manage daily necessities and reduce food waste.

[0282] The flow of the specific process in Example 1 will be described using FIG. 11.

[0283] Step 1:

[0284] The user takes pictures of the daily necessities in the storage location using a smartphone or a dedicated photographing device. As input, image data from the camera device is obtained. This image needs to be clearly taken with adjusted brightness and focus for later analysis.

[0285] Step 2:

[0286] The terminal uploads the captured image to a server on the cloud. An image file is obtained as input, and the image data is stored in cloud storage as output. In this process, the data is properly transmitted based on the data transfer speed and security protocol (e.g., HTTPS).

[0287] Step 3:

[0288] The server receives the uploaded image and analyzes the image using a generated AI model. There is image data from cloud storage as input. This AI model (e.g., TensorFlow model) identifies the types and characteristics of daily necessities in the image. The category information of the identified daily necessities is obtained as output.

[0289] Step 4:

[0290] The server registers the identification result in a storage device (database) and manages the expiration dates of each daily necessity. There is information on the identified daily necessities and expiration date information provided by the user as input. SQL queries are used to insert the information into the database, and an updated database is obtained as output.

[0291] Step 5:

[0292] The server identifies the daily necessities with approaching expiration dates based on the information in the database. There is all the data in the database as input. Date calculations are used to perform the expiration calculation, and a list of daily necessities with approaching expiration dates is obtained as output.

[0293] Step 6:

[0294] The server searches the recipe database based on the daily necessities with approaching expiration dates and selects the optimal cooking instructions. There is a list of daily necessities with approaching expiration dates and the recipe database as input. Considering the combination of ingredients and cooking procedures, the selected cooking instructions are obtained as output.

[0295] Step 7:

[0296] The server sends a notification message containing the generated cooking instructions to the terminal. The input is selected cooking instruction information. A network protocol is used for transmission to the terminal, and the output is the user terminal receiving the notification.

[0297] Step 8:

[0298] The device displays received notifications to the user. These notifications include cooking instructions and information about essential goods nearing their expiration date. The input is the notification message, and the output is displayed on the user interface.

[0299] Step 9:

[0300] The user consumes daily necessities according to the suggested cooking instructions and updates their consumption status via a terminal after consumption. The user's consumption history is used as input, and the updated consumption data is sent back to the server as output. The server reflects this data in its storage device, keeping the database up to date.

[0301] (Application Example 1)

[0302] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0303] In modern households, managing food stored in refrigeration systems is crucial, but there is a lack of systems that effectively manage expiration dates and efficiently utilize food. Furthermore, manual food management is time-consuming and contributes to food waste. Therefore, there is a need for automated household appliances to properly manage food and automatically suggest cooking plans based on expiration dates.

[0304] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0305] In this invention, the server includes means for acquiring an image of food ingredients by an image acquisition means, means for analyzing the acquired image to identify the food ingredients, and means for registering information on the identified food ingredients in a database. Thereby, it becomes possible to efficiently manage the food ingredients in the cooling device and automate the proposal of an optimal recipe considering the expiration date.

[0306] The "image acquisition means" is means for taking a picture of an image of food ingredients stored in the cooling device and acquiring the digital data thereof.

[0307] The "means for identifying food ingredients" is means for analyzing the acquired image data and automatically identifying the food ingredients.

[0308] The "means for registering in the database" is means for recording information on the identified food ingredients and storing it in the database for retrieval and update as necessary.

[0309] The "means for managing the expiration date" is means for monitoring the expiration date of the registered food ingredients and giving a notice when the expiration date is approaching.

[0310] The "means for proposing a recipe" is means for searching for a cooking method based on food ingredients with an approaching expiration date and presenting an optimal cooking procedure.

[0311] The "means for notifying the user" is means for displaying the proposed recipe information on the screen of the user's terminal or a household automatic machine, or communicating it by voice.

[0312] The "household automatic machine" is an automated device used in a household environment and is a device for managing food ingredients and assisting in cooking.

[0313] The "voice support device" is a device for guiding and instructing the user by voice and assisting in the cooking procedure.

[0314] The system for realizing this invention includes a household automated machine, a cloud server, and a user's portable terminal. At the heart of the system is a program for photographing and managing food items inside a cooling device. The household automated machine periodically acquires images of the conditions inside the cooling device using a camera. These images are transmitted via a network to a server in the cloud.

[0315] The server analyzes received images using image analysis software and a generative AI model to identify the type of food ingredient. The identified food ingredient information is stored in a database, and the expiration date is also recorded. The database has a function to manage the expiration dates of registered food ingredients, and as the expiration date approaches, it effectively manages it using expiration date management methods.

[0316] When ingredients nearing their expiration date are detected, the server searches a recipe database to find the optimal cooking method and selects a cooking method through a system that suggests recipes. The selected recipe information is notified to the user via a notification system, appearing on the user's mobile device or the display of a home-use automated appliance. It is also possible to guide the user directly through the cooking method using a voice assistance device.

[0317] For example, if a user enters the prompt "Please tell me a recommended recipe for tonight's dinner," the system will provide a suitable recipe based on the information about the ingredients in the cooling unit. This system reduces food waste and enables more efficient cooking.

[0318] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0319] Step 1:

[0320] The home-use automated machine uses a built-in camera to photograph food items inside the cooling unit. The input is visual information from inside the cooling unit, and the output is digital image data. This image data is transmitted to a cloud server via a network.

[0321] Step 2:

[0322] The server inputs the received image data into the AI ​​image analysis system. Using a generative AI model, the system analyzes the image data and identifies the food items within the image. The input is the transmitted image data, and the output is information about the identified food items. Processing this information prepares it for registration in the database.

[0323] Step 3:

[0324] The server records identified food information and its associated expiration date in a database. Input is the identified food information and expiration date data, while output is the food information stored in the database. This allows for efficient management of the food's condition.

[0325] Step 4:

[0326] The system periodically checks the expiration date information in the database to identify ingredients nearing their expiration date. The input consists of ingredient and expiration date information stored in the database. The identified ingredient information is then prepared for use in the next step.

[0327] Step 5:

[0328] The server searches a recipe database based on ingredients nearing their expiration date. The input is information about ingredients nearing their expiration date, and the output is the optimal recipe information using these ingredients. This process includes recipe selection that takes into account the type and quantity of ingredients, the user's available cooking time, and other factors.

[0329] Step 6:

[0330] The system notifies the user's device or home-use automated appliances of the found recipe information. The input is the selected recipe information, and the output is a visual or audio notification to the user. Home-use automated appliances can use voice assistance devices to provide voice guidance for the cooking procedure.

[0331] Step 7:

[0332] The user cooks according to the suggested recipe. Once cooking is complete, the user reports information about the ingredients consumed to the server via their terminal. The input is the information about the ingredients consumed entered by the user after cooking, and the output is the latest database information. This update ensures that the ingredient list is always accurate and up-to-date.

[0333] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0334] This invention is a system for efficiently managing food stored in a user's refrigerator and consuming the food before its expiration date. This system begins by recognizing food items using images taken by a terminal. The terminal takes pictures of the food items in the refrigerator and uploads the images to a server. The server uses image analysis technology to identify each food item and registers it in a database.

[0335] Furthermore, the present invention is characterized by its incorporation of an emotion engine. The user's terminal or server uses cameras and sensors to analyze the user's emotions from their facial expressions and voice, and uses this information to determine the user's current emotional state. This emotional information is used in the recipe suggestion process to reflect the user's cooking motivation and preferences. For example, if the user is feeling stressed, it is possible to suggest a recipe that can be prepared quickly and easily.

[0336] Furthermore, the server stores the user's emotional history and uses this to customize future recipe suggestions based on past emotional patterns. In this way, the system can provide personalized recipes that match the user's preferences and emotional state. For example, if the system determines that the user is in a relaxed state, it might suggest a dish that can be enjoyed over a longer period of time.

[0337] The combination of these elements allows users to receive recipes best suited to their emotional state, effectively contributing to the reduction of food waste. The system also supports users in cooking at the optimal time by providing timely notifications based on the expiration dates of ingredients.

[0338] The following describes the processing flow.

[0339] Step 1:

[0340] The user takes a picture of the food in the refrigerator with their device. The device adjusts the brightness to the appropriate level to ensure the image is clear while taking the picture.

[0341] Step 2:

[0342] The user uploads the captured image from their device to the server. The device instantly transmits the image via the internet connection.

[0343] Step 3:

[0344] The server uses AI image recognition technology to analyze the received images. Here, it identifies the types of ingredients and extracts their names.

[0345] Step 4:

[0346] The server registers the recognized food ingredient information in the database. The database also stores the expiration date of each ingredient.

[0347] Step 5:

[0348] To recognize the user's emotions, the device uses its built-in camera and microphone to capture facial expressions and voice. This prepares the device for analyzing the user's emotional state.

[0349] Step 6:

[0350] The server uses the latest emotion recognition algorithms to determine the user's emotions based on the emotional information sent from the terminal. This allows it to evaluate the user's current mood and state.

[0351] Step 7:

[0352] The server identifies ingredients nearing their expiration date and selects an appropriate recipe, taking emotional information into account. For example, if the user is tired, it will choose an easy-to-make recipe.

[0353] Step 8:

[0354] The server provides optimal suggestions to the user based on the emotions associated with the selected recipe. These suggestions are emotionally sensitive and customized to enhance the user experience.

[0355] Step 9:

[0356] The server sends the final recipe suggestion as a notification to the user's device. A list of required ingredients is provided along with detailed recipe information.

[0357] Step 10:

[0358] The device displays received notifications to the user. The user cooks according to the suggested recipe and updates the consumed ingredients from the device to record them in the database.

[0359] (Example 2)

[0360] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0361] To address the inefficiencies of food management and food waste in households, it is necessary to accurately understand the condition of individual ingredients and consume them effectively before their expiration date. Furthermore, a challenge is the decline in cooking motivation due to the lack of personalized suggestions that take into account the user's emotional state.

[0362] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0363] In this invention, the server includes means for capturing an image of an object with an image acquisition device, means for analyzing the acquired image to identify the object, and means for registering information about the identified object in a storage device. As a result, the user is offered processing methods based on objects nearing their expiration date, enabling consumption at the optimal time. Furthermore, by detecting the user's emotions and selecting a processing method based on that information, it becomes possible to offer suggestions tailored to individual preferences and circumstances.

[0364] An "image acquisition device" is a device that has the function of capturing images of objects and recording them as digital data.

[0365] "Means for identifying objects" refers to technologies that analyze captured image data to identify the type and characteristics of an object.

[0366] A "memory device" is a device or system that stores information about an identified object and allows it to be retrieved as needed.

[0367] "Shelf life" refers to information indicating the period during which an object is considered usable while maintaining its quality.

[0368] "Real-time notification" is a function that provides information to users immediately based on pre-set conditions.

[0369] "Means of detecting emotions" refers to technologies that analyze a user's physiological and emotional state using cameras, sensors, and other means.

[0370] "Means for suggesting processing methods" refers to a function that, based on pre-registered information and the current situation, indicates to the user the optimal way to use an object.

[0371] This invention is a system that streamlines object management in users' homes and ensures appropriate consumption within the expiration date. This system provides highly personalized suggestions by utilizing an image acquisition device, object recognition technology, a storage device, and user emotion analysis technology.

[0372] The device's role is to acquire images of objects inside the refrigerator taken by the user. The images are uploaded to the server in real time, and image analysis software such as the Google Cloud Vision API is used to identify the type and number of objects.

[0373] The server registers the analysis results in storage and manages the expiration date for each object. This information serves as foundational data for proposing individually optimized processing methods, taking into account the user's emotional state. User emotional analysis is performed using data acquired through the terminal's camera and microphone, and is carried out using Microsoft Azure's emotion analysis API, among others. Cooking methods and ingredient consumption methods are selected according to the user's emotional state.

[0374] For example, when a user takes a picture, the prompt might say, "Please take a picture of the food. Analysis will begin," and during emotion analysis, the prompt might say, "Please show your face to the camera." In this way, users are automatically supported in optimizing the management and consumption of their food ingredients.

[0375] For example, when a user is feeling stressed, the server can suggest a dish that is easy to prepare. Conversely, if the user is relaxed, it can suggest a dish that takes longer to prepare. This allows for a service tailored to the user's individual state and preferences, minimizing food waste.

[0376] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0377] Step 1:

[0378] The user takes an image of an object inside the refrigerator using the device's camera. The image data is saved on the device as input. The device then sends this image data to the server. In this step, the user takes an image according to the prompt message "Please take a picture of the food. Analysis will begin."

[0379] Step 2:

[0380] The server receives image data sent from the terminal. Using the received image data as input, it identifies objects using the Google Cloud Vision API and extracts their types and characteristics. The output is data containing the types and numbers of objects. The server then registers this identification data in its storage device.

[0381] Step 3:

[0382] The server manages the expiration dates of each object based on the registered object data. It retrieves expiration date data from the storage device as input and identifies objects that are nearing their expiration date. A list of objects nearing their expiration date is generated as output. The server then uses this list to prepare notifications to the user.

[0383] Step 4:

[0384] The device acquires the user's emotional state through its camera or microphone. Based on the acquired audio and image data, the server analyzes the user's emotions using Microsoft Azure's Sentiment Analysis API. Data indicating the emotional state is generated as output.

[0385] Step 5:

[0386] The server integrates the object's expiration date with the user's emotional state and uses a generative AI model to select the optimal processing method. The input is expiration date information and emotional data, and the output is a proposed recipe or processing method. The server then prepares to notify the terminal of this information. In this step, if the user's emotional state is stress, a simple recipe is selected.

[0387] Step 6:

[0388] The terminal notifies the user of the processing instructions received from the server. The user then checks the suggested information in response to the prompt "See recommended recipes." Specifically, an interface is provided that allows the user to check the recipes and begin cooking.

[0389] (Application Example 2)

[0390] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0391] Modern households are required to efficiently manage ingredients and consume them without waste amidst busy lifestyles. However, it is difficult for consumers to keep track of what they have in their refrigerators and find appropriate recipes, leading to food waste. Furthermore, the lack of personalized suggestions tailored to the user's emotional state makes it difficult to maintain motivation for cooking. To solve these problems, a system is needed that can simultaneously manage ingredients and provide personalized support based on the user's emotions.

[0392] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0393] In this invention, the server includes means for acquiring images of ingredients using image acquisition means, means for acquiring the user's facial expressions or voice using sensors and analyzing their emotional state, and means for formulating a recipe suitable for the user based on ingredients nearing their expiration date. This enables not only efficient management of ingredients in the refrigerator but also personalized recipe suggestions tailored to the user's emotional state.

[0394] "Image acquisition means" refers to technology that uses cameras or sensors to acquire images of food items inside a refrigerator and transmit them to a system.

[0395] "Food ingredient identification means" refers to a technology that analyzes acquired images and individually recognizes food ingredients that can be stored or cooked.

[0396] A "database registration method" is a technology that records information about identified food ingredients in a database within the system to prepare for future data retrieval.

[0397] A "shelf-life management system" is a system function that tracks the expiration dates of registered food items and sends notifications when the expiration date is approaching.

[0398] A "recipe suggestion method" is a technology that generates and suggests recipes to facilitate cooking based on the expiration date of ingredients and the user's emotional state.

[0399] "Emotional analysis methods" refer to technologies that use sensors to capture a user's facial expressions and voice, analyze them, and determine their current emotional state.

[0400] "Push notification" refers to electronic notification technology used to inform users in real time about information such as the expiration date of ingredients or selected recipes.

[0401] A system for carrying out this invention includes a user terminal, a server, and sensors for performing sentiment analysis.

[0402] The user's device uses its camera to capture images of food items in the refrigerator and uploads them to a server. The server uses image analysis software such as OpenCV to identify the food items and registers the information in a database. This database records the type of food item and its expiration date. The system also has a function to notify the user via push notification when the expiration date of the food items is approaching.

[0403] Furthermore, emotion analysis is performed using software such as Amazon Rekognition, which captures the user's facial expressions or voice using sensors. This emotion data is sent to a server, where the user's emotional state is determined.

[0404] Using generative AI models such as ChatGPT, the server generates recipes considering ingredient information and the user's emotional state. These recipes are provided as feedback tailored to the user's individual state. For example, if the user is tired, the system suggests an easy-to-make dish. Conversational recipe suggestions are also possible, improving the user experience.

[0405] For example, if a user living alone enters the prompt "I have eggs, butter, and milk" via their smartphone, the system can suggest a recipe such as "Since you seem to want to relax, why not try making an omelet?"

[0406] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0407] Step 1:

[0408] The user's device takes pictures of the food items inside the refrigerator using its camera and sends the image data to the server. The input is the images of the food items, and the output is the image data sent to the server. This step includes the user using the device to check the contents of the refrigerator.

[0409] Step 2:

[0410] The server analyzes the received image data using OpenCV to identify the type and quantity of ingredients. The input is image data sent from the user's terminal, and the output is the identified ingredient information. At this time, the information obtained through image analysis is recorded in a database.

[0411] Step 3:

[0412] After ingredient information is registered in the database, the server tracks and manages the expiration date and sends a push notification to the user as the expiration date approaches. The input is the ingredient information registered in the database, and the output is the push notification. By receiving this notification, users can plan how to use the ingredients before they expire.

[0413] Step 4:

[0414] The user's device or attached sensors capture the user's facial expressions and voice, and transmit this data to the server. The input is data about the user's emotional state, and the output is the analyzed emotional state. This step includes actions to acquire the emotional data.

[0415] Step 5:

[0416] The server uses Amazon Rekognition to analyze emotional data and determine the user's emotional state. The input is facial or voice data sent by the user, and the output is the result of the determination of the user's emotional state. This information is used for recipe selection in the next step.

[0417] Step 6:

[0418] The server uses a generative AI model to generate recipes based on the user's emotional state and ingredient information. The input is the emotional state and identified ingredient information, and the output is a individually customized recipe. This step involves the AI ​​formulating a recipe suitable for the user.

[0419] Step 7:

[0420] The user's device notifies the user of recipes received from the server and suggests them as cooking options. The input is recipe information sent from the server, and the output is the recipe notified to the user. Upon receiving this notification, the user can then plan their next meal.

[0421] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0422] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0423] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0424] [Third Embodiment]

[0425] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0426] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0427] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0428] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0429] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0430] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0431] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0432] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0433] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0434] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0435] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0436] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0437] This invention is a system that begins with the user taking a picture of the food inside the refrigerator using their own device. The user takes an image of the food stored in the refrigerator using a smartphone or a dedicated camera device, and the device uploads this image to a server in the cloud.

[0438] The server analyzes received images using AI image recognition technology and automatically identifies the ingredients within the images. The server registers this identified data in a database, storing it along with the expiration date entered by the user for each ingredient. The information in the database is updated periodically, and the server uses an algorithm to identify ingredients that are nearing their expiration date.

[0439] The server searches a recipe database based on ingredients nearing their expiration date and selects a suitable recipe. In this process, the server determines the optimal recipe, considering factors such as ingredient combinations, cooking procedures, and preparation time. The server then generates a notification message containing the selected recipe information and information about ingredients nearing their expiration date, and sends it to the user's device.

[0440] The device displays received notifications to the user and suggests available recipes. The user checks the notifications and cooks according to the recipes, efficiently consuming ingredients. When cooking is complete, the user updates the information on the ingredients consumed using the device, keeping the database up-to-date.

[0441] As a concrete example, suppose a user takes a photo of the inside of their refrigerator, and the server identifies "chicken," "spinach," and "carrots." The server checks the expiration dates of these ingredients and detects that the chicken is nearing its expiration date. Based on this, the server suggests a recipe for "Creamy Chicken and Spinach Stew" and notifies the user of the recipe. By receiving this suggestion, the user can use the ingredients efficiently and prevent food waste.

[0442] The following describes the processing flow.

[0443] Step 1:

[0444] The user takes a photo of the food items inside the refrigerator using their device. It is recommended to take the photo in sufficient light so that the food items are clearly visible.

[0445] Step 2:

[0446] The device uploads images taken by the user to a cloud server. This upload is performed via an internet connection.

[0447] Step 3:

[0448] The server receives the uploaded image and sends it to an AI image analysis module. There, machine learning algorithms are used to recognize the food items in the image and identify their names.

[0449] Step 4:

[0450] The server records identified food ingredient information in a database. For each ingredient, the expiration date previously registered by the user is also stored in the database.

[0451] Step 5:

[0452] The server periodically scans the database to check the expiration dates of ingredients. It extracts ingredients that are nearing their expiration date and creates a specific list.

[0453] Step 6:

[0454] The server searches the recipe database based on the expiration date list. It selects the optimal recipe by considering factors such as ingredient combinations, user preferences, and cooking time.

[0455] Step 7:

[0456] The server generates a notification message containing the selected recipe. This notification includes the name of the ingredient that is nearing its expiration date and details of the recipe that uses that ingredient.

[0457] Step 8:

[0458] The server sends the notification message to the user's device and delivers it as a push notification.

[0459] Step 9:

[0460] The device displays the received notification to the user. Here, the user can review the contents of the suggested recipe.

[0461] Step 10:

[0462] The user checks the notification and cooks according to the provided recipe. After cooking is complete, they update the information on the ingredients consumed from their device to keep the database information up to date.

[0463] (Example 1)

[0464] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0465] Inadequate inventory management and expiration date tracking of essential goods in storage areas can lead to unnecessary waste and excess inventory. Furthermore, a lack of efficient means to consume essential goods nearing their expiration date contributes to food waste. Solving these problems, streamlining the management of essential goods, and minimizing waste are crucial.

[0466] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0467] In this invention, the server includes means for acquiring images of daily necessities in a storage location using a camera, means for analyzing the acquired images using artificial intelligence technology to identify the daily necessities, and means for registering information about the identified daily necessities in a storage device. This makes it possible to automatically grasp the inventory status of daily necessities and propose an appropriate consumption plan based on their expiration dates.

[0468] "Photography device" refers to hardware used to acquire images of storage locations, including those containing essential goods.

[0469] "Daily necessities" refer to consumable items needed in daily life.

[0470] "Artificial intelligence technology" refers to advanced algorithms and models used by computers to analyze images and data, specifically deep learning models.

[0471] A "memory device" refers to a data storage means for registering and retaining information about identified essential goods and their expiration dates.

[0472] "Expiration date" refers to information indicating the period during which essential goods can be consumed.

[0473] "Cooking instructions" refer to information that includes specific consumption methods and cooking procedures, based on essential goods nearing their expiration date.

[0474] "Notification" refers to messages or alerts used by a server to transmit relevant information to a user.

[0475] A "generative artificial intelligence model" refers to a machine learning model that learns patterns and features from data in order to perform a specific task.

[0476] A "short message notification" refers to a simple message format that instantly conveys the minimum necessary information to the user.

[0477] This invention is a system that supports users in managing and efficiently consuming their daily necessities. The system aims to minimize waste of daily necessities by primarily acquiring and analyzing images of these necessities and providing recommended cooking instructions based on that information.

[0478] First, users use their smartphones or dedicated cameras to photograph essential household items in storage areas such as refrigerators and pantries. The device then acquires high-resolution image data and sends it to a cloud server.

[0479] Upon receiving the acquired images, the server uses artificial intelligence technology for image analysis to identify the food items. TensorFlow and PyTorch are used as machine learning frameworks, with deep learning models supporting the identification of everyday necessities. This allows for the identification of specific types of food items.

[0480] The server stores the identification results in storage and manages them along with the expiration date information for each essential item. Database management systems such as MySQL and PostgreSQL are used for this information management. Based on this data, the server identifies essential items that are nearing their expiration date and generates optimal cooking instructions by comparing the recorded data with the recipe database.

[0481] The selected cooking instructions are notified from the server to the terminal. The terminal displays the notification to the user in the form of a pop-up or alert, offering options tailored to the user's lifestyle. For example, a prompt such as "Please suggest the best recipe based on the expiration dates of chicken, spinach, and carrots" is input into the AI ​​model, and based on the received information, a specific cooking method is presented.

[0482] The user consumes daily necessities according to the cooking instructions received, and then updates the information about the ingredients actually consumed via their terminal. The updated information is sent back to the server, and the database is kept up-to-date.

[0483] Through this series of processes, users can efficiently manage their daily necessities and reduce food waste.

[0484] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0485] Step 1:

[0486] Users photograph essential household items within the storage area using their smartphones or dedicated cameras. Image data from the camera device is obtained as input. This image needs to be clear, with adjustments to brightness and focus, for later analysis.

[0487] Step 2:

[0488] The device uploads the captured image to a server in the cloud. The input is an image file, and the output is the image data stored in cloud storage. During this process, data is transmitted appropriately based on the data transfer speed and security protocol (e.g., HTTPS).

[0489] Step 3:

[0490] The server receives uploaded images and analyzes them using a generative AI model. The input is image data from cloud storage. This AI model (e.g., a TensorFlow model) identifies the types and characteristics of everyday necessities within the images. The output is category information for the identified everyday necessities.

[0491] Step 4:

[0492] The server registers the identification results in a storage device (database) and manages the expiration dates of each essential item. The inputs are information on the identified essential items and expiration date information provided by the user. SQL queries are used to insert the information into the database, and the output is an updated database.

[0493] Step 5:

[0494] The server identifies essential goods nearing their expiration date based on information in the database. The input is the entire database data. Expiration dates are calculated using date calculations, and the output is a list of essential goods nearing their expiration date.

[0495] Step 6:

[0496] The server searches a recipe database based on the expiration dates of essential goods and selects the most suitable cooking instructions. The input consists of a list of essential goods nearing their expiration date and the recipe database. Considering the combination of ingredients and cooking procedures, the server outputs the selected cooking instructions.

[0497] Step 7:

[0498] The server sends a notification message containing the generated cooking instructions to the terminal. The input is selected cooking instruction information. A network protocol is used for transmission to the terminal, and the output is the user terminal receiving the notification.

[0499] Step 8:

[0500] The device displays received notifications to the user. These notifications include cooking instructions and information about essential goods nearing their expiration date. The input is the notification message, and the output is displayed on the user interface.

[0501] Step 9:

[0502] The user consumes daily necessities according to the suggested cooking instructions and updates their consumption status via a terminal after consumption. The user's consumption history is used as input, and the updated consumption data is sent back to the server as output. The server reflects this data in its storage device, keeping the database up to date.

[0503] (Application Example 1)

[0504] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0505] In modern households, managing food stored in refrigeration systems is crucial, but there is a lack of systems that effectively manage expiration dates and efficiently utilize food. Furthermore, manual food management is time-consuming and contributes to food waste. Therefore, there is a need for automated household appliances to properly manage food and automatically suggest cooking plans based on expiration dates.

[0506] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0507] In this invention, the server includes means for acquiring images of food ingredients using image acquisition means, means for analyzing the acquired images to identify the food ingredients, and means for registering information about the identified food ingredients in a database. This makes it possible to efficiently manage food ingredients in a cooling device and automate the suggestion of optimal recipes that take expiration dates into consideration.

[0508] "Image acquisition means" refers to a means for taking images of food stored in a cooling device and acquiring the digital data thereof.

[0509] "Means for identifying food ingredients" refers to methods for analyzing acquired image data and automatically identifying food ingredients.

[0510] "Methods for registering in a database" refers to methods for recording information about identified ingredients and storing it in a database for searching and updating as needed.

[0511] "Means for managing expiration dates" refers to a means of monitoring the expiration dates of registered food items and sending notifications when the expiration date is approaching.

[0512] "Methods for suggesting recipes" refer to methods that search for cooking methods based on ingredients nearing their expiration date and present the optimal cooking procedure.

[0513] "Means of notifying the user" refers to methods of displaying the suggested recipe information on the user's device or the screen of a home appliance, or communicating it via voice.

[0514] "Household automated machinery" refers to automated devices used in a home environment, specifically for managing ingredients and assisting with cooking.

[0515] A "voice support device" is a device that provides guidance and instructions to the user via voice, and is used to assist with cooking procedures.

[0516] The system for realizing this invention includes a household automated machine, a cloud server, and a user's portable terminal. At the heart of the system is a program for photographing and managing food items inside a cooling device. The household automated machine periodically acquires images of the conditions inside the cooling device using a camera. These images are transmitted via a network to a server in the cloud.

[0517] The server analyzes received images using image analysis software and a generative AI model to identify the type of food ingredient. The identified food ingredient information is stored in a database, and the expiration date is also recorded. The database has a function to manage the expiration dates of registered food ingredients, and as the expiration date approaches, it effectively manages it using expiration date management methods.

[0518] When ingredients nearing their expiration date are detected, the server searches a recipe database to find the optimal cooking method and selects a cooking method through a system that suggests recipes. The selected recipe information is notified to the user via a notification system, appearing on the user's mobile device or the display of a home-use automated appliance. It is also possible to guide the user directly through the cooking method using a voice assistance device.

[0519] For example, if a user enters the prompt "Please tell me a recommended recipe for tonight's dinner," the system will provide a suitable recipe based on the information about the ingredients in the cooling unit. This system reduces food waste and enables more efficient cooking.

[0520] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0521] Step 1:

[0522] The home-use automated machine uses a built-in camera to photograph food items inside the cooling unit. The input is visual information from inside the cooling unit, and the output is digital image data. This image data is transmitted to a cloud server via a network.

[0523] Step 2:

[0524] The server inputs the received image data into the AI ​​image analysis system. Using a generative AI model, the system analyzes the image data and identifies the food items within the image. The input is the transmitted image data, and the output is information about the identified food items. Processing this information prepares it for registration in the database.

[0525] Step 3:

[0526] The server records identified food information and its associated expiration date in a database. Input is the identified food information and expiration date data, while output is the food information stored in the database. This allows for efficient management of the food's condition.

[0527] Step 4:

[0528] The system periodically checks the expiration date information in the database to identify ingredients nearing their expiration date. The input consists of ingredient and expiration date information stored in the database. The identified ingredient information is then prepared for use in the next step.

[0529] Step 5:

[0530] The server searches a recipe database based on ingredients nearing their expiration date. The input is information about ingredients nearing their expiration date, and the output is the optimal recipe information using these ingredients. This process includes recipe selection that takes into account the type and quantity of ingredients, the user's available cooking time, and other factors.

[0531] Step 6:

[0532] The system notifies the user's device or home-use automated appliances of the found recipe information. The input is the selected recipe information, and the output is a visual or audio notification to the user. Home-use automated appliances can use voice assistance devices to provide voice guidance for the cooking procedure.

[0533] Step 7:

[0534] The user cooks according to the suggested recipe. Once cooking is complete, the user reports information about the ingredients consumed to the server via their terminal. The input is the information about the ingredients consumed entered by the user after cooking, and the output is the latest database information. This update ensures that the ingredient list is always accurate and up-to-date.

[0535] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0536] This invention is a system for efficiently managing food stored in a user's refrigerator and consuming the food before its expiration date. This system begins by recognizing food items using images taken by a terminal. The terminal takes pictures of the food items in the refrigerator and uploads the images to a server. The server uses image analysis technology to identify each food item and registers it in a database.

[0537] Furthermore, the present invention is characterized by its incorporation of an emotion engine. The user's terminal or server uses cameras and sensors to analyze the user's emotions from their facial expressions and voice, and uses this information to determine the user's current emotional state. This emotional information is used in the recipe suggestion process to reflect the user's cooking motivation and preferences. For example, if the user is feeling stressed, it is possible to suggest a recipe that can be prepared quickly and easily.

[0538] Furthermore, the server stores the user's emotional history and uses this to customize future recipe suggestions based on past emotional patterns. In this way, the system can provide personalized recipes that match the user's preferences and emotional state. For example, if the system determines that the user is in a relaxed state, it might suggest a dish that can be enjoyed over a longer period of time.

[0539] The combination of these elements allows users to receive recipes best suited to their emotional state, effectively contributing to the reduction of food waste. The system also supports users in cooking at the optimal time by providing timely notifications based on the expiration dates of ingredients.

[0540] The following describes the processing flow.

[0541] Step 1:

[0542] The user takes a picture of the food in the refrigerator with their device. The device adjusts the brightness to the appropriate level to ensure the image is clear while taking the picture.

[0543] Step 2:

[0544] The user uploads the captured image from their device to the server. The device instantly transmits the image via the internet connection.

[0545] Step 3:

[0546] The server uses AI image recognition technology to analyze the received images. Here, it identifies the types of ingredients and extracts their names.

[0547] Step 4:

[0548] The server registers the recognized food ingredient information in the database. The database also stores the expiration date of each ingredient.

[0549] Step 5:

[0550] To recognize the user's emotions, the device uses its built-in camera and microphone to capture facial expressions and voice. This prepares the device for analyzing the user's emotional state.

[0551] Step 6:

[0552] The server uses the latest emotion recognition algorithms to determine the user's emotions based on the emotional information sent from the terminal. This allows it to evaluate the user's current mood and state.

[0553] Step 7:

[0554] The server identifies ingredients nearing their expiration date and selects an appropriate recipe, taking emotional information into account. For example, if the user is tired, it will choose an easy-to-make recipe.

[0555] Step 8:

[0556] The server provides optimal suggestions to the user based on the emotions associated with the selected recipe. These suggestions are emotionally sensitive and customized to enhance the user experience.

[0557] Step 9:

[0558] The server sends the final recipe suggestion as a notification to the user's device. A list of required ingredients is provided along with detailed recipe information.

[0559] Step 10:

[0560] The device displays received notifications to the user. The user cooks according to the suggested recipe and updates the consumed ingredients from the device to record them in the database.

[0561] (Example 2)

[0562] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0563] To address the inefficiencies of food management and food waste in households, it is necessary to accurately understand the condition of individual ingredients and consume them effectively before their expiration date. Furthermore, a challenge is the decline in cooking motivation due to the lack of personalized suggestions that take into account the user's emotional state.

[0564] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0565] In this invention, the server includes means for capturing an image of an object with an image acquisition device, means for analyzing the acquired image to identify the object, and means for registering information about the identified object in a storage device. As a result, the user is offered processing methods based on objects nearing their expiration date, enabling consumption at the optimal time. Furthermore, by detecting the user's emotions and selecting a processing method based on that information, it becomes possible to offer suggestions tailored to individual preferences and circumstances.

[0566] An "image acquisition device" is a device that has the function of capturing images of objects and recording them as digital data.

[0567] "Means for identifying objects" refers to technologies that analyze captured image data to identify the type and characteristics of an object.

[0568] A "memory device" is a device or system that stores information about an identified object and allows it to be retrieved as needed.

[0569] "Shelf life" refers to information indicating the period during which an object is considered usable while maintaining its quality.

[0570] "Real-time notification" is a function that provides information to users immediately based on pre-set conditions.

[0571] "Means of detecting emotions" refers to technologies that analyze a user's physiological and emotional state using cameras, sensors, and other means.

[0572] "Means for suggesting processing methods" refers to a function that, based on pre-registered information and the current situation, indicates to the user the optimal way to use an object.

[0573] This invention is a system that streamlines object management in users' homes and ensures appropriate consumption within the expiration date. This system provides highly personalized suggestions by utilizing an image acquisition device, object recognition technology, a storage device, and user emotion analysis technology.

[0574] The device's role is to acquire images of objects inside the refrigerator taken by the user. The images are uploaded to the server in real time, and image analysis software such as the Google Cloud Vision API is used to identify the type and number of objects.

[0575] The server registers the analysis results in storage and manages the expiration date for each object. This information serves as foundational data for proposing individually optimized processing methods, taking into account the user's emotional state. User emotional analysis is performed using data acquired through the terminal's camera and microphone, and is carried out using Microsoft Azure's emotion analysis API, among others. Cooking methods and ingredient consumption methods are selected according to the user's emotional state.

[0576] For example, when a user takes a picture, the prompt might say, "Please take a picture of the food. Analysis will begin," and during emotion analysis, the prompt might say, "Please show your face to the camera." In this way, users are automatically supported in optimizing the management and consumption of their food ingredients.

[0577] For example, when a user is feeling stressed, the server can suggest a dish that is easy to prepare. Conversely, if the user is relaxed, it can suggest a dish that takes longer to prepare. This allows for a service tailored to the user's individual state and preferences, minimizing food waste.

[0578] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0579] Step 1:

[0580] The user takes an image of an object inside the refrigerator using the device's camera. The image data is saved on the device as input. The device then sends this image data to the server. In this step, the user takes an image according to the prompt message "Please take a picture of the food. Analysis will begin."

[0581] Step 2:

[0582] The server receives image data sent from the terminal. Using the received image data as input, it identifies objects using the Google Cloud Vision API and extracts their types and characteristics. The output is data containing the types and numbers of objects. The server then registers this identification data in its storage device.

[0583] Step 3:

[0584] The server manages the expiration dates of each object based on the registered object data. It retrieves expiration date data from the storage device as input and identifies objects that are nearing their expiration date. A list of objects nearing their expiration date is generated as output. The server then uses this list to prepare notifications to the user.

[0585] Step 4:

[0586] The device acquires the user's emotional state through its camera or microphone. Based on the acquired audio and image data, the server analyzes the user's emotions using Microsoft Azure's Sentiment Analysis API. Data indicating the emotional state is generated as output.

[0587] Step 5:

[0588] The server integrates the object's expiration date with the user's emotional state and uses a generative AI model to select the optimal processing method. The input is expiration date information and emotional data, and the output is a proposed recipe or processing method. The server then prepares to notify the terminal of this information. In this step, if the user's emotional state is stress, a simple recipe is selected.

[0589] Step 6:

[0590] The terminal notifies the user of the processing instructions received from the server. The user then checks the suggested information in response to the prompt "See recommended recipes." Specifically, an interface is provided that allows the user to check the recipes and begin cooking.

[0591] (Application Example 2)

[0592] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0593] Modern households are required to efficiently manage ingredients and consume them without waste amidst busy lifestyles. However, it is difficult for consumers to keep track of what they have in their refrigerators and find appropriate recipes, leading to food waste. Furthermore, the lack of personalized suggestions tailored to the user's emotional state makes it difficult to maintain motivation for cooking. To solve these problems, a system is needed that can simultaneously manage ingredients and provide personalized support based on the user's emotions.

[0594] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0595] In this invention, the server includes means for acquiring images of ingredients using image acquisition means, means for acquiring the user's facial expressions or voice using sensors and analyzing their emotional state, and means for formulating a recipe suitable for the user based on ingredients nearing their expiration date. This enables not only efficient management of ingredients in the refrigerator but also personalized recipe suggestions tailored to the user's emotional state.

[0596] "Image acquisition means" refers to technology that uses cameras or sensors to acquire images of food items inside a refrigerator and transmit them to a system.

[0597] "Food ingredient identification means" refers to a technology that analyzes acquired images and individually recognizes food ingredients that can be stored or cooked.

[0598] A "database registration method" is a technology that records information about identified food ingredients in a database within the system to prepare for future data retrieval.

[0599] A "shelf-life management system" is a system function that tracks the expiration dates of registered food items and sends notifications when the expiration date is approaching.

[0600] A "recipe suggestion method" is a technology that generates and suggests recipes to facilitate cooking based on the expiration date of ingredients and the user's emotional state.

[0601] "Emotional analysis methods" refer to technologies that use sensors to capture a user's facial expressions and voice, analyze them, and determine their current emotional state.

[0602] "Push notification" refers to electronic notification technology used to inform users in real time about information such as the expiration date of ingredients or selected recipes.

[0603] A system for carrying out this invention includes a user terminal, a server, and sensors for performing sentiment analysis.

[0604] The user's device uses its camera to capture images of food items in the refrigerator and uploads them to a server. The server uses image analysis software such as OpenCV to identify the food items and registers the information in a database. This database records the type of food item and its expiration date. The system also has a function to notify the user via push notification when the expiration date of the food items is approaching.

[0605] Furthermore, emotion analysis is performed using software such as Amazon Rekognition, which captures the user's facial expressions or voice using sensors. This emotion data is sent to a server, where the user's emotional state is determined.

[0606] Using generative AI models such as ChatGPT, the server generates recipes considering ingredient information and the user's emotional state. These recipes are provided as feedback tailored to the user's individual state. For example, if the user is tired, the system suggests an easy-to-make dish. Conversational recipe suggestions are also possible, improving the user experience.

[0607] For example, if a user living alone enters the prompt "I have eggs, butter, and milk" via their smartphone, the system can suggest a recipe such as "Since you seem to want to relax, why not try making an omelet?"

[0608] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0609] Step 1:

[0610] The user's device takes pictures of the food items inside the refrigerator using its camera and sends the image data to the server. The input is the images of the food items, and the output is the image data sent to the server. This step includes the user using the device to check the contents of the refrigerator.

[0611] Step 2:

[0612] The server analyzes the received image data using OpenCV to identify the type and quantity of ingredients. The input is image data sent from the user's terminal, and the output is the identified ingredient information. At this time, the information obtained through image analysis is recorded in a database.

[0613] Step 3:

[0614] After ingredient information is registered in the database, the server tracks and manages the expiration date and sends a push notification to the user as the expiration date approaches. The input is the ingredient information registered in the database, and the output is the push notification. By receiving this notification, users can plan how to use the ingredients before they expire.

[0615] Step 4:

[0616] The user's device or attached sensors capture the user's facial expressions and voice, and transmit this data to the server. The input is data about the user's emotional state, and the output is the analyzed emotional state. This step includes actions to acquire the emotional data.

[0617] Step 5:

[0618] The server uses Amazon Rekognition to analyze emotional data and determine the user's emotional state. The input is facial or voice data sent by the user, and the output is the result of the determination of the user's emotional state. This information is used for recipe selection in the next step.

[0619] Step 6:

[0620] The server uses a generative AI model to generate recipes based on the user's emotional state and ingredient information. The input is the emotional state and identified ingredient information, and the output is a individually customized recipe. This step involves the AI ​​formulating a recipe suitable for the user.

[0621] Step 7:

[0622] The user's device notifies the user of recipes received from the server and suggests them as cooking options. The input is recipe information sent from the server, and the output is the recipe notified to the user. Upon receiving this notification, the user can then plan their next meal.

[0623] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0624] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0625] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0626] [Fourth Embodiment]

[0627] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0628] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0629] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0630] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0631] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0632] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0633] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0634] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0635] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0636] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0637] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0638] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0639] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0640] This invention is a system that begins with the user taking a picture of the food inside the refrigerator using their own device. The user takes an image of the food stored in the refrigerator using a smartphone or a dedicated camera device, and the device uploads this image to a server in the cloud.

[0641] The server analyzes received images using AI image recognition technology and automatically identifies the ingredients within the images. The server registers this identified data in a database, storing it along with the expiration date entered by the user for each ingredient. The information in the database is updated periodically, and the server uses an algorithm to identify ingredients that are nearing their expiration date.

[0642] The server searches a recipe database based on ingredients nearing their expiration date and selects a suitable recipe. In this process, the server determines the optimal recipe, considering factors such as ingredient combinations, cooking procedures, and preparation time. The server then generates a notification message containing the selected recipe information and information about ingredients nearing their expiration date, and sends it to the user's device.

[0643] The device displays received notifications to the user and suggests available recipes. The user checks the notifications and cooks according to the recipes, efficiently consuming ingredients. When cooking is complete, the user updates the information on the ingredients consumed using the device, keeping the database up-to-date.

[0644] As a concrete example, suppose a user takes a photo of the inside of their refrigerator, and the server identifies "chicken," "spinach," and "carrots." The server checks the expiration dates of these ingredients and detects that the chicken is nearing its expiration date. Based on this, the server suggests a recipe for "Creamy Chicken and Spinach Stew" and notifies the user of the recipe. By receiving this suggestion, the user can use the ingredients efficiently and prevent food waste.

[0645] The following describes the processing flow.

[0646] Step 1:

[0647] The user takes a photo of the food items inside the refrigerator using their device. It is recommended to take the photo in sufficient light so that the food items are clearly visible.

[0648] Step 2:

[0649] The device uploads images taken by the user to a cloud server. This upload is performed via an internet connection.

[0650] Step 3:

[0651] The server receives the uploaded image and sends it to an AI image analysis module. There, machine learning algorithms are used to recognize the food items in the image and identify their names.

[0652] Step 4:

[0653] The server records identified food ingredient information in a database. For each ingredient, the expiration date previously registered by the user is also stored in the database.

[0654] Step 5:

[0655] The server periodically scans the database to check the expiration dates of ingredients. It extracts ingredients that are nearing their expiration date and creates a specific list.

[0656] Step 6:

[0657] The server searches the recipe database based on the expiration date list. It selects the optimal recipe by considering factors such as ingredient combinations, user preferences, and cooking time.

[0658] Step 7:

[0659] The server generates a notification message containing the selected recipe. This notification includes the name of the ingredient that is nearing its expiration date and details of the recipe that uses that ingredient.

[0660] Step 8:

[0661] The server sends the notification message to the user's device and delivers it as a push notification.

[0662] Step 9:

[0663] The device displays the received notification to the user. Here, the user can review the contents of the suggested recipe.

[0664] Step 10:

[0665] The user checks the notification and cooks according to the provided recipe. After cooking is complete, they update the information on the ingredients consumed from their device to keep the database information up to date.

[0666] (Example 1)

[0667] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0668] Inadequate inventory management and expiration date tracking of essential goods in storage areas can lead to unnecessary waste and excess inventory. Furthermore, a lack of efficient means to consume essential goods nearing their expiration date contributes to food waste. Solving these problems, streamlining the management of essential goods, and minimizing waste are crucial.

[0669] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0670] In this invention, the server includes means for acquiring images of daily necessities in a storage location using a camera, means for analyzing the acquired images using artificial intelligence technology to identify the daily necessities, and means for registering information about the identified daily necessities in a storage device. This makes it possible to automatically grasp the inventory status of daily necessities and propose an appropriate consumption plan based on their expiration dates.

[0671] "Photography device" refers to hardware used to acquire images of storage locations, including those containing essential goods.

[0672] "Daily necessities" refer to consumable items needed in daily life.

[0673] "Artificial intelligence technology" refers to advanced algorithms and models used by computers to analyze images and data, specifically deep learning models.

[0674] A "memory device" refers to a data storage means for registering and retaining information about identified essential goods and their expiration dates.

[0675] "Expiration date" refers to information indicating the period during which essential goods can be consumed.

[0676] "Cooking instructions" refer to information that includes specific consumption methods and cooking procedures, based on essential goods nearing their expiration date.

[0677] "Notification" refers to messages or alerts used by a server to transmit relevant information to a user.

[0678] A "generative artificial intelligence model" refers to a machine learning model that learns patterns and features from data in order to perform a specific task.

[0679] A "short message notification" refers to a simple message format that instantly conveys the minimum necessary information to the user.

[0680] This invention is a system that supports users in managing and efficiently consuming their daily necessities. The system aims to minimize waste of daily necessities by primarily acquiring and analyzing images of these necessities and providing recommended cooking instructions based on that information.

[0681] First, users use their smartphones or dedicated cameras to photograph essential household items in storage areas such as refrigerators and pantries. The device then acquires high-resolution image data and sends it to a cloud server.

[0682] Upon receiving the acquired images, the server uses artificial intelligence technology for image analysis to identify the food items. TensorFlow and PyTorch are used as machine learning frameworks, with deep learning models supporting the identification of everyday necessities. This allows for the identification of specific types of food items.

[0683] The server stores the identification results in storage and manages them along with the expiration date information for each essential item. Database management systems such as MySQL and PostgreSQL are used for this information management. Based on this data, the server identifies essential items that are nearing their expiration date and generates optimal cooking instructions by comparing the recorded data with the recipe database.

[0684] The selected cooking instructions are notified from the server to the terminal. The terminal displays the notification to the user in the form of a pop-up or alert, offering options tailored to the user's lifestyle. For example, a prompt such as "Please suggest the best recipe based on the expiration dates of chicken, spinach, and carrots" is input into the AI ​​model, and based on the received information, a specific cooking method is presented.

[0685] The user consumes daily necessities according to the cooking instructions received, and then updates the information about the ingredients actually consumed via their terminal. The updated information is sent back to the server, and the database is kept up-to-date.

[0686] Through this series of processes, users can efficiently manage their daily necessities and reduce food waste.

[0687] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0688] Step 1:

[0689] Users photograph essential household items within the storage area using their smartphones or dedicated cameras. Image data from the camera device is obtained as input. This image needs to be clear, with adjustments to brightness and focus, for later analysis.

[0690] Step 2:

[0691] The device uploads the captured image to a server in the cloud. The input is an image file, and the output is the image data stored in cloud storage. During this process, data is transmitted appropriately based on the data transfer speed and security protocol (e.g., HTTPS).

[0692] Step 3:

[0693] The server receives uploaded images and analyzes them using a generative AI model. The input is image data from cloud storage. This AI model (e.g., a TensorFlow model) identifies the types and characteristics of everyday necessities within the images. The output is category information for the identified everyday necessities.

[0694] Step 4:

[0695] The server registers the identification results in a storage device (database) and manages the expiration dates of each essential item. The inputs are information on the identified essential items and expiration date information provided by the user. SQL queries are used to insert the information into the database, and the output is an updated database.

[0696] Step 5:

[0697] The server identifies essential goods nearing their expiration date based on information in the database. The input is the entire database data. Expiration dates are calculated using date calculations, and the output is a list of essential goods nearing their expiration date.

[0698] Step 6:

[0699] The server searches a recipe database based on the expiration dates of essential goods and selects the most suitable cooking instructions. The input consists of a list of essential goods nearing their expiration date and the recipe database. Considering the combination of ingredients and cooking procedures, the server outputs the selected cooking instructions.

[0700] Step 7:

[0701] The server sends a notification message containing the generated cooking instructions to the terminal. The input is selected cooking instruction information. A network protocol is used for transmission to the terminal, and the output is the user terminal receiving the notification.

[0702] Step 8:

[0703] The device displays received notifications to the user. These notifications include cooking instructions and information about essential goods nearing their expiration date. The input is the notification message, and the output is displayed on the user interface.

[0704] Step 9:

[0705] The user consumes daily necessities according to the suggested cooking instructions and updates their consumption status via a terminal after consumption. The user's consumption history is used as input, and the updated consumption data is sent back to the server as output. The server reflects this data in its storage device, keeping the database up to date.

[0706] (Application Example 1)

[0707] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0708] In modern households, managing food stored in refrigeration systems is crucial, but there is a lack of systems that effectively manage expiration dates and efficiently utilize food. Furthermore, manual food management is time-consuming and contributes to food waste. Therefore, there is a need for automated household appliances to properly manage food and automatically suggest cooking plans based on expiration dates.

[0709] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0710] In this invention, the server includes means for acquiring images of food ingredients using image acquisition means, means for analyzing the acquired images to identify the food ingredients, and means for registering information about the identified food ingredients in a database. This makes it possible to efficiently manage food ingredients in a cooling device and automate the suggestion of optimal recipes that take expiration dates into consideration.

[0711] "Image acquisition means" refers to a means for taking images of food stored in a cooling device and acquiring the digital data thereof.

[0712] "Means for identifying food ingredients" refers to methods for analyzing acquired image data and automatically identifying food ingredients.

[0713] "Methods for registering in a database" refers to methods for recording information about identified ingredients and storing it in a database for searching and updating as needed.

[0714] "Means for managing expiration dates" refers to a means of monitoring the expiration dates of registered food items and sending notifications when the expiration date is approaching.

[0715] "Methods for suggesting recipes" refer to methods that search for cooking methods based on ingredients nearing their expiration date and present the optimal cooking procedure.

[0716] "Means of notifying the user" refers to methods of displaying the suggested recipe information on the user's device or the screen of a home appliance, or communicating it via voice.

[0717] "Household automated machinery" refers to automated devices used in a home environment, specifically for managing ingredients and assisting with cooking.

[0718] A "voice support device" is a device that provides guidance and instructions to the user via voice, and is used to assist with cooking procedures.

[0719] The system for realizing this invention includes a household automated machine, a cloud server, and a user's portable terminal. At the heart of the system is a program for photographing and managing food items inside a cooling device. The household automated machine periodically acquires images of the conditions inside the cooling device using a camera. These images are transmitted via a network to a server in the cloud.

[0720] The server analyzes received images using image analysis software and a generative AI model to identify the type of food ingredient. The identified food ingredient information is stored in a database, and the expiration date is also recorded. The database has a function to manage the expiration dates of registered food ingredients, and as the expiration date approaches, it effectively manages it using expiration date management methods.

[0721] When ingredients nearing their expiration date are detected, the server searches a recipe database to find the optimal cooking method and selects a cooking method through a system that suggests recipes. The selected recipe information is notified to the user via a notification system, appearing on the user's mobile device or the display of a home-use automated appliance. It is also possible to guide the user directly through the cooking method using a voice assistance device.

[0722] For example, if a user enters the prompt "Please tell me a recommended recipe for tonight's dinner," the system will provide a suitable recipe based on the information about the ingredients in the cooling unit. This system reduces food waste and enables more efficient cooking.

[0723] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0724] Step 1:

[0725] The home-use automated machine uses a built-in camera to photograph food items inside the cooling unit. The input is visual information from inside the cooling unit, and the output is digital image data. This image data is transmitted to a cloud server via a network.

[0726] Step 2:

[0727] The server inputs the received image data into the AI ​​image analysis system. Using a generative AI model, the system analyzes the image data and identifies the food items within the image. The input is the transmitted image data, and the output is information about the identified food items. Processing this information prepares it for registration in the database.

[0728] Step 3:

[0729] The server records identified food information and its associated expiration date in a database. Input is the identified food information and expiration date data, while output is the food information stored in the database. This allows for efficient management of the food's condition.

[0730] Step 4:

[0731] The system periodically checks the expiration date information in the database to identify ingredients nearing their expiration date. The input consists of ingredient and expiration date information stored in the database. The identified ingredient information is then prepared for use in the next step.

[0732] Step 5:

[0733] The server searches a recipe database based on ingredients nearing their expiration date. The input is information about ingredients nearing their expiration date, and the output is the optimal recipe information using these ingredients. This process includes recipe selection that takes into account the type and quantity of ingredients, the user's available cooking time, and other factors.

[0734] Step 6:

[0735] The system notifies the user's device or home-use automated appliances of the found recipe information. The input is the selected recipe information, and the output is a visual or audio notification to the user. Home-use automated appliances can use voice assistance devices to provide voice guidance for the cooking procedure.

[0736] Step 7:

[0737] The user cooks according to the suggested recipe. Once cooking is complete, the user reports information about the ingredients consumed to the server via their terminal. The input is the information about the ingredients consumed entered by the user after cooking, and the output is the latest database information. This update ensures that the ingredient list is always accurate and up-to-date.

[0738] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0739] This invention is a system for efficiently managing food stored in a user's refrigerator and consuming the food before its expiration date. This system begins by recognizing food items using images taken by a terminal. The terminal takes pictures of the food items in the refrigerator and uploads the images to a server. The server uses image analysis technology to identify each food item and registers it in a database.

[0740] Furthermore, the present invention is characterized by its incorporation of an emotion engine. The user's terminal or server uses cameras and sensors to analyze the user's emotions from their facial expressions and voice, and uses this information to determine the user's current emotional state. This emotional information is used in the recipe suggestion process to reflect the user's cooking motivation and preferences. For example, if the user is feeling stressed, it is possible to suggest a recipe that can be prepared quickly and easily.

[0741] Furthermore, the server stores the user's emotional history and uses this to customize future recipe suggestions based on past emotional patterns. In this way, the system can provide personalized recipes that match the user's preferences and emotional state. For example, if the system determines that the user is in a relaxed state, it might suggest a dish that can be enjoyed over a longer period of time.

[0742] The combination of these elements allows users to receive recipes best suited to their emotional state, effectively contributing to the reduction of food waste. The system also supports users in cooking at the optimal time by providing timely notifications based on the expiration dates of ingredients.

[0743] The following describes the processing flow.

[0744] Step 1:

[0745] The user takes a picture of the food in the refrigerator with their device. The device adjusts the brightness to the appropriate level to ensure the image is clear while taking the picture.

[0746] Step 2:

[0747] The user uploads the captured image from their device to the server. The device instantly transmits the image via the internet connection.

[0748] Step 3:

[0749] The server uses AI image recognition technology to analyze the received images. Here, it identifies the types of ingredients and extracts their names.

[0750] Step 4:

[0751] The server registers the recognized food ingredient information in the database. The database also stores the expiration date of each ingredient.

[0752] Step 5:

[0753] To recognize the user's emotions, the device uses its built-in camera and microphone to capture facial expressions and voice. This prepares the device for analyzing the user's emotional state.

[0754] Step 6:

[0755] The server uses the latest emotion recognition algorithms to determine the user's emotions based on the emotional information sent from the terminal. This allows it to evaluate the user's current mood and state.

[0756] Step 7:

[0757] The server identifies ingredients nearing their expiration date and selects an appropriate recipe, taking emotional information into account. For example, if the user is tired, it will choose an easy-to-make recipe.

[0758] Step 8:

[0759] The server provides optimal suggestions to the user based on the emotions associated with the selected recipe. These suggestions are emotionally sensitive and customized to enhance the user experience.

[0760] Step 9:

[0761] The server sends the final recipe suggestion as a notification to the user's device. A list of required ingredients is provided along with detailed recipe information.

[0762] Step 10:

[0763] The device displays received notifications to the user. The user cooks according to the suggested recipe and updates the consumed ingredients from the device to record them in the database.

[0764] (Example 2)

[0765] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0766] To address the inefficiencies of food management and food waste in households, it is necessary to accurately understand the condition of individual ingredients and consume them effectively before their expiration date. Furthermore, a challenge is the decline in cooking motivation due to the lack of personalized suggestions that take into account the user's emotional state.

[0767] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0768] In this invention, the server includes means for capturing an image of an object with an image acquisition device, means for analyzing the acquired image to identify the object, and means for registering information about the identified object in a storage device. As a result, the user is offered processing methods based on objects nearing their expiration date, enabling consumption at the optimal time. Furthermore, by detecting the user's emotions and selecting a processing method based on that information, it becomes possible to offer suggestions tailored to individual preferences and circumstances.

[0769] An "image acquisition device" is a device that has the function of capturing images of objects and recording them as digital data.

[0770] "Means for identifying objects" refers to technologies that analyze captured image data to identify the type and characteristics of an object.

[0771] A "memory device" is a device or system that stores information about an identified object and allows it to be retrieved as needed.

[0772] "Shelf life" refers to information indicating the period during which an object is considered usable while maintaining its quality.

[0773] "Real-time notification" is a function that provides information to users immediately based on pre-set conditions.

[0774] "Means of detecting emotions" refers to technologies that analyze a user's physiological and emotional state using cameras, sensors, and other means.

[0775] "Means for suggesting processing methods" refers to a function that, based on pre-registered information and the current situation, indicates to the user the optimal way to use an object.

[0776] This invention is a system that streamlines object management in users' homes and ensures appropriate consumption within the expiration date. This system provides highly personalized suggestions by utilizing an image acquisition device, object recognition technology, a storage device, and user emotion analysis technology.

[0777] The device's role is to acquire images of objects inside the refrigerator taken by the user. The images are uploaded to the server in real time, and image analysis software such as the Google Cloud Vision API is used to identify the type and number of objects.

[0778] The server registers the analysis results in storage and manages the expiration date for each object. This information serves as foundational data for proposing individually optimized processing methods, taking into account the user's emotional state. User emotional analysis is performed using data acquired through the terminal's camera and microphone, and is carried out using Microsoft Azure's emotion analysis API, among others. Cooking methods and ingredient consumption methods are selected according to the user's emotional state.

[0779] For example, when a user takes a picture, the prompt might say, "Please take a picture of the food. Analysis will begin," and during emotion analysis, the prompt might say, "Please show your face to the camera." In this way, users are automatically supported in optimizing the management and consumption of their food ingredients.

[0780] For example, when a user is feeling stressed, the server can suggest a dish that is easy to prepare. Conversely, if the user is relaxed, it can suggest a dish that takes longer to prepare. This allows for a service tailored to the user's individual state and preferences, minimizing food waste.

[0781] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0782] Step 1:

[0783] The user takes an image of an object inside the refrigerator using the device's camera. The image data is saved on the device as input. The device then sends this image data to the server. In this step, the user takes an image according to the prompt message "Please take a picture of the food. Analysis will begin."

[0784] Step 2:

[0785] The server receives image data sent from the terminal. Using the received image data as input, it identifies objects using the Google Cloud Vision API and extracts their types and characteristics. The output is data containing the types and numbers of objects. The server then registers this identification data in its storage device.

[0786] Step 3:

[0787] The server manages the expiration dates of each object based on the registered object data. It retrieves expiration date data from the storage device as input and identifies objects that are nearing their expiration date. A list of objects nearing their expiration date is generated as output. The server then uses this list to prepare notifications to the user.

[0788] Step 4:

[0789] The device acquires the user's emotional state through its camera or microphone. Based on the acquired audio and image data, the server analyzes the user's emotions using Microsoft Azure's Sentiment Analysis API. Data indicating the emotional state is generated as output.

[0790] Step 5:

[0791] The server integrates the object's expiration date with the user's emotional state and uses a generative AI model to select the optimal processing method. The input is expiration date information and emotional data, and the output is a proposed recipe or processing method. The server then prepares to notify the terminal of this information. In this step, if the user's emotional state is stress, a simple recipe is selected.

[0792] Step 6:

[0793] The terminal notifies the user of the processing instructions received from the server. The user then checks the suggested information in response to the prompt "See recommended recipes." Specifically, an interface is provided that allows the user to check the recipes and begin cooking.

[0794] (Application Example 2)

[0795] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0796] Modern households are required to efficiently manage ingredients and consume them without waste amidst busy lifestyles. However, it is difficult for consumers to keep track of what they have in their refrigerators and find appropriate recipes, leading to food waste. Furthermore, the lack of personalized suggestions tailored to the user's emotional state makes it difficult to maintain motivation for cooking. To solve these problems, a system is needed that can simultaneously manage ingredients and provide personalized support based on the user's emotions.

[0797] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0798] In this invention, the server includes means for acquiring images of ingredients using image acquisition means, means for acquiring the user's facial expressions or voice using sensors and analyzing their emotional state, and means for formulating a recipe suitable for the user based on ingredients nearing their expiration date. This enables not only efficient management of ingredients in the refrigerator but also personalized recipe suggestions tailored to the user's emotional state.

[0799] "Image acquisition means" refers to technology that uses cameras or sensors to acquire images of food items inside a refrigerator and transmit them to a system.

[0800] "Food ingredient identification means" refers to a technology that analyzes acquired images and individually recognizes food ingredients that can be stored or cooked.

[0801] A "database registration method" is a technology that records information about identified food ingredients in a database within the system to prepare for future data retrieval.

[0802] A "shelf-life management system" is a system function that tracks the expiration dates of registered food items and sends notifications when the expiration date is approaching.

[0803] A "recipe suggestion method" is a technology that generates and suggests recipes to facilitate cooking based on the expiration date of ingredients and the user's emotional state.

[0804] "Emotional analysis methods" refer to technologies that use sensors to capture a user's facial expressions and voice, analyze them, and determine their current emotional state.

[0805] "Push notification" refers to electronic notification technology used to inform users in real time about information such as the expiration date of ingredients or selected recipes.

[0806] A system for carrying out this invention includes a user terminal, a server, and sensors for performing sentiment analysis.

[0807] The user's device uses its camera to capture images of food items in the refrigerator and uploads them to a server. The server uses image analysis software such as OpenCV to identify the food items and registers the information in a database. This database records the type of food item and its expiration date. The system also has a function to notify the user via push notification when the expiration date of the food items is approaching.

[0808] Furthermore, emotion analysis is performed using software such as Amazon Rekognition, which captures the user's facial expressions or voice using sensors. This emotion data is sent to a server, where the user's emotional state is determined.

[0809] Using generative AI models such as ChatGPT, the server generates recipes considering ingredient information and the user's emotional state. These recipes are provided as feedback tailored to the user's individual state. For example, if the user is tired, the system suggests an easy-to-make dish. Conversational recipe suggestions are also possible, improving the user experience.

[0810] For example, if a user living alone enters the prompt "I have eggs, butter, and milk" via their smartphone, the system can suggest a recipe such as "Since you seem to want to relax, why not try making an omelet?"

[0811] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0812] Step 1:

[0813] The user's device takes pictures of the food items inside the refrigerator using its camera and sends the image data to the server. The input is the images of the food items, and the output is the image data sent to the server. This step includes the user using the device to check the contents of the refrigerator.

[0814] Step 2:

[0815] The server analyzes the received image data using OpenCV to identify the type and quantity of ingredients. The input is image data sent from the user's terminal, and the output is the identified ingredient information. At this time, the information obtained through image analysis is recorded in a database.

[0816] Step 3:

[0817] After ingredient information is registered in the database, the server tracks and manages the expiration date and sends a push notification to the user as the expiration date approaches. The input is the ingredient information registered in the database, and the output is the push notification. By receiving this notification, users can plan how to use the ingredients before they expire.

[0818] Step 4:

[0819] The user's device or attached sensors capture the user's facial expressions and voice, and transmit this data to the server. The input is data about the user's emotional state, and the output is the analyzed emotional state. This step includes actions to acquire the emotional data.

[0820] Step 5:

[0821] The server uses Amazon Rekognition to analyze emotional data and determine the user's emotional state. The input is facial or voice data sent by the user, and the output is the result of the determination of the user's emotional state. This information is used for recipe selection in the next step.

[0822] Step 6:

[0823] The server uses a generative AI model to generate recipes based on the user's emotional state and ingredient information. The input is the emotional state and identified ingredient information, and the output is a individually customized recipe. This step involves the AI ​​formulating a recipe suitable for the user.

[0824] Step 7:

[0825] The user's device notifies the user of recipes received from the server and suggests them as cooking options. The input is recipe information sent from the server, and the output is the recipe notified to the user. Upon receiving this notification, the user can then plan their next meal.

[0826] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0827] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0828] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0829] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0830] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0831] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0832] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0833] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0834] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0835] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0836] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0837] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0838] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0839] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0840] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0841] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0842] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0843] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0844] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0845] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0846] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0847] The following is further disclosed regarding the embodiments described above.

[0848] (Claim 1)

[0849] A means of acquiring images of food ingredients using an image acquisition method,

[0850] A means of identifying ingredients by analyzing acquired images,

[0851] A means of registering information about identified ingredients in a database,

[0852] A means of managing the expiration dates of registered food ingredients,

[0853] A method for suggesting recipes based on ingredients nearing their expiration date,

[0854] A means of notifying the user of the suggested recipe information,

[0855] A system that includes this.

[0856] (Claim 2)

[0857] The system according to claim 1, which uses a deep learning model for image analysis of food ingredients.

[0858] (Claim 3)

[0859] The system according to claim 1, which provides information to the user using push notifications when the expiration date is approaching.

[0860] "Example 1"

[0861] (Claim 1)

[0862] A means of acquiring images of essential goods in a storage area using a camera,

[0863] The acquired images are analyzed using artificial intelligence technology to identify essential household items,

[0864] A means for registering information on identified daily necessities in a storage device,

[0865] A means of managing the expiration dates of registered essential goods,

[0866] A method for suggesting cooking instructions based on daily necessities that are nearing their expiration date,

[0867] A means of notifying the user of the proposed cooking instructions,

[0868] A means of updating the contents of a storage device in accordance with the user's consumption behavior,

[0869] A system that includes this.

[0870] (Claim 2)

[0871] The system according to claim 1, which uses a generative artificial intelligence model for image analysis of daily necessities.

[0872] (Claim 3)

[0873] The system according to claim 1, which provides information to the user using a short message notification when the expiration date is approaching.

[0874] "Application Example 1"

[0875] (Claim 1)

[0876] A means of acquiring images of food ingredients using an image acquisition method,

[0877] A means of identifying ingredients by analyzing acquired images,

[0878] A means of registering information about identified ingredients in a database,

[0879] A means of managing the expiration dates of registered food ingredients,

[0880] A method for suggesting recipes based on ingredients nearing their expiration date,

[0881] A means of notifying the user of the suggested recipe information,

[0882] A means of periodically photographing food items inside a cooling device using a household automated machine and executing each stage of processing,

[0883] A means of guiding the user through cooking procedures using a voice support device,

[0884] A system that includes this.

[0885] (Claim 2)

[0886] The system according to claim 1, which uses a deep learning model for image analysis of food ingredients.

[0887] (Claim 3)

[0888] The system according to claim 1, which provides information to the user using push notifications when the expiration date is approaching.

[0889] "Example 2 of combining an emotion engine"

[0890] (Claim 1)

[0891] A means for capturing an image of an object with an image acquisition device,

[0892] A means for analyzing acquired images to identify objects,

[0893] Means for registering information about an identified object in a storage device,

[0894] A means for managing the storage period of registered objects,

[0895] A means of proposing a processing method based on objects whose expiration date is approaching,

[0896] A means for detecting the user's emotions and selecting a processing method based on that information,

[0897] A means of notifying the user of the proposed processing method information,

[0898] A system that includes this.

[0899] (Claim 2)

[0900] The system according to claim 1, which uses a generative AI model in the image analysis of an object.

[0901] (Claim 3)

[0902] The system according to claim 1, which provides information to the user using real-time notifications when the expiration date is approaching.

[0903] "Application example 2 when combining with an emotional engine"

[0904] (Claim 1)

[0905] A means of acquiring images of food ingredients using an image acquisition method,

[0906] A means of identifying ingredients by analyzing acquired images,

[0907] A means of registering information about identified ingredients in a database,

[0908] A means of managing the expiration dates of registered food ingredients,

[0909] A method for suggesting recipes based on ingredients nearing their expiration date,

[0910] A means of notifying the user of the suggested recipe information,

[0911] A means of acquiring the user's facial expressions or voice using sensors and analyzing their emotional state,

[0912] A method for formulating recipes suitable for users based on the results of sentiment analysis,

[0913] A system that includes this.

[0914] (Claim 2)

[0915] The system according to claim 1, which uses a deep learning model for image analysis of food ingredients and analysis of emotional states.

[0916] (Claim 3)

[0917] The system according to claim 1, which provides information to the user via push notification when the expiration date is approaching, and further suggests recipes according to the user's emotional state. [Explanation of symbols]

[0918] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring images of food ingredients using an image acquisition method, A means of identifying ingredients by analyzing acquired images, A means of registering information about identified ingredients in a database, A means of managing the expiration dates of registered food ingredients, A method for suggesting recipes based on ingredients nearing their expiration date, A means of notifying the user of the suggested recipe information, A means of periodically photographing food items inside a cooling device using a household automated machine and executing each stage of processing, A means of guiding the user through cooking procedures using a voice support device, A system that includes this.

2. The system according to claim 1, which uses a deep learning model for image analysis of food ingredients.

3. The system according to claim 1, which provides information to the user using push notifications when the expiration date is approaching.

Images