System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving an image of an inside of a refrigerator taken by a user; means for analyzing the received image to generate an ingredient list; means for generating a cooking recipe based on the generated ingredient list; and means for providing the generated cooking recipe to the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Food waste in the home has become a major problem in modern society. This is mainly due to people throwing away food in their refrigerators without using it properly. Many households lack effective ways to avoid food waste, resulting in a large amount of food going to waste. Furthermore, people often don't have time to think up new cooking ideas in their busy daily lives, which results in food going unused and spoiling. This invention aims to maximize the use of food in the refrigerator and reduce food waste. [Means for solving the problem]

[0005] The present invention is a system that includes a means for receiving images of the inside of a refrigerator taken by a user, a means for analyzing the received images to generate an ingredient list, a means for generating a cooking recipe based on the generated ingredient list, and a means for providing the generated cooking recipe to the user.The user sends images of the inside of the refrigerator to an external image analysis API to obtain an ingredient list, and further generates a cooking recipe that takes nutritional balance into consideration based on this ingredient list and provides it to the user, thereby making efficient use of ingredients in the refrigerator and effectively reducing food waste.

[0006] "User" refers to an individual who uses this system to check ingredients in the refrigerator and obtain cooking recipes.

[0007] "Images inside the refrigerator" refers to photographic data taken by a user after opening the refrigerator door, showing the food inside.

[0008] "Means for receiving" refers to a device such as a server or terminal that has the function of receiving images of the inside of a refrigerator that a user uploads to the system.

[0009] The "means for analyzing and generating an ingredient list" refers to the technical means for analyzing the received image and listing the ingredients contained in the image as text data.

[0010] "Means for generating cooking recipes" refers to algorithms or generative AI models for creating appropriate cooking recipes based on the generated ingredient list.

[0011] The "means for providing" refers to a device such as a server or terminal that has the function of displaying the generated cooking recipe to the user.

[0012] "External image analysis API" refers to an API that provides an external platform or service that the server uses to analyze image data and generate an ingredient list.

[0013] "Cooking recipes that take nutritional balance into consideration" refers to recipes that include cooking steps and ingredients designed with consideration for the nutritional value and health aspects of the ingredients. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] The present invention relates to a system that uses image analysis technology to automatically generate an ingredient list by taking a picture of the contents of a refrigerator and providing the system with original cooking recipes based on that list. This system provides a means to efficiently use ingredients in the refrigerator without wasting them.

[0036] System Overview

[0037] The system consists of the following main components:

[0038] 1. Image receiving means

[0039] It provides an interface for users to upload images of the inside of their refrigerator.

[0040] The server stores the images received from the user and prepares them for analysis.

[0041] 2. Image analysis methods

[0042] The server analyzes the received image and creates a list of ingredients. Here, an external image analysis API is used to recognize ingredients in the image and convert them into text data.

[0043] 3. Recipe Generation Method

[0044] The server uses a generative AI model based on the ingredient list to generate a cooking recipe that takes nutritional balance into consideration.

[0045] The generated recipe includes detailed information such as ingredients, steps, and cooking times.

[0046] 4. Means of providing information

[0047] The server has an interface for providing the generated recipe information to the user.

[0048] Users can receive recipe information through devices such as smartphones and PCs.

[0049] Specific processing content of the program

[0050] User Actions

[0051] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door so that the food inside is clearly visible, and then pressing the shutter button.

[0052] Upload the captured image to the system's application using the app's "Upload Image" function.

[0053] Server Processing

[0054] The server receives the uploaded image and immediately saves it in a specified directory.

[0055] The saved image is sent to an external image analysis API to generate an ingredient list, and the JSON response returned from the API is analyzed to extract the ingredient list.

[0056] For example: "Carrots, onions, tomatoes, chicken" would be included in the list.

[0057] The generated ingredient list is sent to a generative AI model to generate recipes that take nutritional balance into account.

[0058] Processing generative AI models

[0059] A generative AI model creates creative cooking recipes based on a provided list of ingredients, including ingredients, steps, cooking times, and nutritional information.

[0060] For example, recipes for "Tomato and Chicken Stew" and "Carrot and Onion Salad" are generated.

[0061] Server information provision

[0062] The server provides the generated recipe to the user, and the user interface displays detailed information about the recipe to the user.

[0063] Users can view the provided recipes and actually cook them.

[0064] Specific examples

[0065] Suppose a user takes a photo of the contents of their refrigerator and uploads the image to the system. The server receives the image and uses an image analysis API to generate a list of ingredients: "carrots, onions, tomatoes, and chicken." The server then sends this list to a generative AI model, which generates recipes for "tomato and chicken stew" and "carrot and onion salad." Finally, the server provides these recipes to the user, who then begins cooking according to the recipes.

[0066] In this way, this system allows users to make effective use of the ingredients in their refrigerator and reduce food waste.

[0067] The processing flow will be explained below.

[0068] Step 1:

[0069] A user takes a picture of the contents of the refrigerator with their smartphone. The user launches the smartphone's camera app, opens the refrigerator door, and presses the shutter button after making sure the food inside is clearly visible.

[0070] Step 2:

[0071] Users can upload images they have taken to the system's application by tapping the "Upload Image" button on the application screen, selecting an image from the album, and sending it.

[0072] Step 3:

[0073] The server receives the uploaded image. The server listens to the HTTP request and saves the received image data in a specified directory.

[0074] Step 4:

[0075] The server sends the received image to an external image analysis API, which then sends the image file to an API endpoint and executes the image analysis request.

[0076] Step 5:

[0077] The server receives a JSON response from the image analysis API. The response contains text data of the analyzed ingredients, such as a list of ingredients such as "carrot, onion, tomato, and chicken."

[0078] Step 6:

[0079] The server retrieves the generated ingredient list and sends it to the generative AI model. The server then sends a request including the ingredient list to the API endpoint of the generative AI model.

[0080] Step 7:

[0081] A generative AI model generates a cooking recipe based on an ingredient list. The model combines ingredients and generates multiple recipe ideas, taking into account nutritional balance and cooking procedures. For example, it generates recipes such as "Tomato and Chicken Stew" and "Carrot and Onion Salad."

[0082] Step 8:

[0083] The server receives the recipe returned from the generative AI model. The server receives the response from the recipe generation API and analyzes the response data.

[0084] Step 9:

[0085] The server displays the generated recipe to the user, converts the received recipe data into a format for display on the user interface, and displays detailed recipe information on the user's smartphone screen.

[0086] Step 10:

[0087] Users can select the dish they want to make from the displayed recipes by tapping on their smartphone screen.

[0088] Step 11:

[0089] The app will then create a recipe based on the user's chosen recipe, and the user will follow the instructions displayed on the app to select the ingredients they need and begin cooking.

[0090] Example 1

[0091] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0092] In modern society, there is a problem that purchased ingredients are often forgotten in the refrigerator and wasted. Furthermore, many users find it difficult to easily plan and execute nutritionally balanced meals at home amid busy lifestyles. This has led to increased food waste and health concerns. The purpose of this invention is to solve these problems by providing a system that effectively utilizes ingredients in the refrigerator and efficiently provides nutritionally balanced meals.

[0093] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0094] In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for saving the received images, means for calling an external image analysis API to analyze the saved images and generate an ingredient list, means for using a generative AI model to generate cooking recipes that take nutritional balance into consideration based on the ingredient list obtained through the analysis, and means for providing the generated cooking recipes through a user interface, thereby enabling users to efficiently plan and execute nutritionally balanced meals without wasting ingredients in the refrigerator.

[0095] "User" refers to an individual who uses the system to take pictures of the inside of a refrigerator and upload them to the system.

[0096] "Server" refers to the central computing device that receives and processes images uploaded by users.

[0097] "Image receiving means" refers to a function or device that allows the server to receive images taken by the user.

[0098] "Storage means" refers to a function or device for storing received images in a predetermined location.

[0099] "Image analysis API" refers to an external application program interface that analyzes received images and recognizes and lists ingredients within the images.

[0100] "Ingredient list" refers to a list of ingredients in an image obtained as a result of analysis by the image analysis API.

[0101] "Generative AI model" refers to an artificial intelligence model used to generate creative cooking recipes based on an ingredient list.

[0102] "Recipe generation means" refers to a function or device for generating a cooking recipe that takes nutritional balance into consideration based on an ingredient list.

[0103] "User Interface" refers to an interactive screen or application that provides the generated cooking recipe to the user.

[0104] This invention relates to a system that uses image analysis technology to automatically generate an ingredient list by taking a picture of the contents of a refrigerator and providing the system with original cooking recipes based on that list. This system provides a means to efficiently use ingredients in the refrigerator without wasting them.

[0105] Key Components of the System

[0106] 1. Image receiving means

[0107] It provides an interface for users to upload images of the inside of their refrigerator.

[0108] The server stores the images received from the user and prepares them for analysis.

[0109] 2. Image analysis methods

[0110] The server analyzes the received image and creates a list of ingredients. Here, an external image analysis API is used to recognize ingredients in the image and convert them into text data.

[0111] 3. Recipe Generation Method

[0112] The server uses a generative AI model based on the ingredient list to generate a cooking recipe that takes nutritional balance into consideration.

[0113] The generated recipe includes detailed information such as ingredients, steps, and cooking times.

[0114] 4. Means of providing information

[0115] The server has an interface for providing the generated recipe information to the user.

[0116] Users can receive recipe information through devices such as smartphones and PCs.

[0117] Specific processing content of the program

[0118] 1. User Actions

[0119] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door so that the food inside is clearly visible, and then pressing the shutter button.

[0120] Upload the captured image to the system's application using the app's "Upload Image" function.

[0121] 2. Server Processing

[0122] The server receives the uploaded image and saves it in a specified directory.

[0123] The saved image is sent to an external image analysis API to generate an ingredient list, and the JSON response returned from the API is analyzed to extract the ingredient list.

[0124] For example: "Carrots, onions, tomatoes, chicken" would be included in the list.

[0125] The generated ingredient list is sent to a generative AI model to generate recipes that take nutritional balance into account.

[0126] 3. Processing of generative AI models

[0127] A generative AI model creates creative cooking recipes based on a provided list of ingredients, including ingredients, steps, cooking times, and nutritional information.

[0128] For example, recipes for "Tomato and Chicken Stew" and "Carrot and Onion Salad" are generated.

[0129] 4. Server Information

[0130] The server provides the generated recipe to the user, and the user interface displays detailed information about the recipe to the user.

[0131] Users can view the provided recipes and actually cook them.

[0132] Specific examples

[0133] Suppose a user takes a photo of the contents of their refrigerator and uploads the image to the system. The server receives the image and uses an image analysis API to generate a list of ingredients: "carrots, onions, tomatoes, and chicken." The server then sends this list to a generative AI model, which generates recipes for "tomato and chicken stew" and "carrot and onion salad." Finally, the server provides these recipes to the user, who then begins cooking according to the recipes.

[0134] Prompt Sentence Examples

[0135] An example of a prompt to send to the image analysis API is:

[0136] {

[0137] "action": "parse_image",

[0138] "image_url": "YOUR_IMAGE_URL",

[0139] "response_format": "json"

[0140] }

[0141] Here's an example prompt to send to a generative AI model:

[0142] Recipe category to create: Cooking

[0143] Ingredients: carrots, onions, tomatoes, chicken

[0144] Generate two recipes, taking nutritional balance into consideration.

[0145] In this way, the present invention allows users to make better use of food in their refrigerators and reduce food waste.

[0146] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0147] Step 1:

[0148] User image capture and upload

[0149] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door and pressing the shutter button so that the food inside is clearly visible.

[0150] The captured image is uploaded to the system's application. The user opens the app, taps the "Upload Image" button, selects an image, and presses the "Send" button.

[0151] Input: Image of the contents of the refrigerator taken with a smartphone

[0152] Output: Images uploaded to the system

[0153] Step 2:

[0154] Receiving and saving images on the server

[0155] The server receives the uploaded image. The image data is sent to the server via an HTTP request.

[0156] Save the received images in a specified directory (e.g., / images / uploads).

[0157] Log the save directory path to ensure the image has been saved successfully.

[0158] Input: Image data uploaded by the user

[0159] Output: Images saved in the specified directory on the server

[0160] Step 3:

[0161] Calling the server's image analysis API

[0162] The server sends the stored image to an external image analysis API using an HTTP POST request, sending the image's binary data and URL to the API.

[0163] The image analysis API analyzes the image and returns a list of ingredients in JSON format.

[0164] The server parses the JSON response and extracts the list of ingredients in the image.

[0165] Input: Path to the image stored on the server

[0166] Output: Generated list of ingredients (e.g. carrot, onion, tomato, chicken)

[0167] Step 4:

[0168] Server recipe generation

[0169] Based on the ingredient list, the server sends a prompt to the generative AI model.

[0170] Recipe category to create: Cooking

[0171] Ingredients: carrots, onions, tomatoes, chicken

[0172] Generate two recipes, taking nutritional balance into consideration.

[0173] The generative AI model generates a recipe and returns it to the server, including ingredients, steps, cooking time, and nutritional information.

[0174] Input: Generated ingredient list and prompt statement

[0175] Output: Generated cooking recipe

[0176] Step 5:

[0177] Server recipe provision

[0178] The server displays the generated recipe in the user interface, allowing the user to view detailed recipe information on the app's "View Recipe" screen.

[0179] It allows users to browse recipes and provides instructions for actually cooking the food.

[0180] Input: Generated cooking recipe

[0181] Output: Recipe information displayed in the user interface

[0182] Through these steps, the system enables users to effectively utilize the ingredients in their refrigerator and provide efficient, nutritionally balanced meals.

[0183] (Application example 1)

[0184] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0185] Modern households are expected to efficiently utilize the ingredients in their refrigerators, but this often fails. Food waste is a problem due to ineffective use of ingredients. Many people are looking for new recipes and cooking videos when it comes to home cooking, but finding the right content can be a hassle. Furthermore, there is a lack of real-time interactive tools to improve cooking skills at home. Providing an effective system to solve these problems is a challenge.

[0186] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0187] In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for analyzing the received images to generate an ingredient list, means for generating cooking videos and articles related to cooking recipes based on the generated ingredient list, means for providing the user with cooking videos and articles related to the generated cooking recipes, and means for providing a virtual kitchen environment in which the user can perform cooking simulations in real time. This not only enables efficient use of ingredients in the refrigerator, but also makes it possible to provide the user with new recipes and cooking videos to help improve their cooking skills at home.

[0188] A "user" is someone who uses this system to manage ingredients in their refrigerator and obtain cooking recipes and related content.

[0189] "Photographing" refers to the act of taking a photo of the food in the refrigerator using a smartphone or other image capture device.

[0190] "Images inside the refrigerator" are photographic data of the inside of the refrigerator taken by the user, showing the state of the ingredients.

[0191] "Analysis" is the process by which the received image is processed by software to identify and list elements within the image.

[0192] The "ingredient list" is a list of names and information of ingredients extracted from the analyzed images of the inside of the refrigerator.

[0193] A "cooking recipe" is a document that contains specific cooking steps and a list of necessary ingredients, generated based on an ingredient list.

[0194] "Cooking videos" are video content that visually show cooking procedures and can be viewed by users.

[0195] An "article" is a text-based explanation related to cooking methods and ingredients.

[0196] A "virtual kitchen environment" is a virtual space that simulates a real-world kitchen and allows users to interactively experience cooking procedures.

[0197] "Real-time cooking simulation" refers to a user experiencing a simulation in which they react instantly within a virtual kitchen environment.

[0198] This invention is a system that efficiently utilizes ingredients in a refrigerator and provides users with cooking recipes and related content. This system is implemented mainly using a smartphone and a server.

[0199] 1. User Action

[0200] The user takes a picture of the food in the refrigerator using their smartphone. After taking the picture, the user uploads the picture to the server using the system's dedicated application. At this stage, the "Image Upload" function is used.

[0201] 2. Server processing

[0202] The server saves the received images in a specified directory. Then, it sends the saved images to an external image analysis API (e.g., Google Cloud Vision API) to generate an ingredient list. It parses the JSON response returned by the API and extracts the ingredient list. Specifically, the following prompt sentence is used:

[0203] Example prompt:

[0204] "Generate a list of ingredients in an image. Analyze the image and list the names of the ingredients."

[0205] 3. Recipe and content generation

[0206] The server generates a recipe based on the generated ingredient list using a generative AI model (e.g., OpenAI's GPT-4), taking nutritional balance into consideration. The generated recipe includes detailed information such as ingredients, steps, and cooking time. Specifically, the following prompt is used:

[0207] Example prompt:

[0208] "Create a delicious recipe using the following ingredients: chicken, onion, carrot, and tomato."

[0209] Additionally, to provide you with related cooking videos and articles, we use cooking video services (e.g., YouTube API, Vimeo API) to search for and provide related content to you.

[0210] 4. Virtual Kitchen Environment

[0211] The server provides a 3D virtual environment using platforms such as Unreal Engine and Unity, allowing users to simulate real-time cooking in a virtual kitchen environment, allowing them to improve their cooking skills by recreating real-life cooking procedures in a virtual environment.

[0212] Specific examples

[0213] A user takes a photo of the contents of their refrigerator with their smartphone and uploads the image to the system's application. The server receives the image and uses the Google Cloud Vision API to generate an ingredient list: "carrots, onions, tomatoes, and chicken." Next, GPT-4 is used to generate recipes: "Tomato and Chicken Stew" and "Carrot and Onion Salad." Additionally, cooking videos for these recipes are provided using the YouTube API. Users can browse these recipes and videos and experience cooking simulations in a virtual kitchen environment using an HMD (e.g., Oculus Rift).

[0214] This system allows users to efficiently use ingredients in their refrigerators and easily acquire and try out new recipes and cooking methods, which is expected to not only reduce food waste but also improve the cooking experience at home.

[0215] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0216] Step 1:

[0217] The user takes a picture of the inside of the refrigerator with their smartphone and uploads the picture to the system's application. The input is the image taken by the user, and the output is the upload of the image data to the server. Specifically, the user takes a picture of the food in the refrigerator using the camera app on their smartphone, and then uses the "image upload" function in the app to send the picture to the server.

[0218] Step 2:

[0219] The server saves the images it receives in a specified directory. The input is the uploaded image data, and the output is an image file saved on the server. Specifically, the server receives an HTTP request and saves the sent image data in a specific folder.

[0220] Step 3:

[0221] The server sends the saved image to an external image analysis API to generate an ingredient list. The input is the saved image file, and the output is the ingredient list. Specifically, the server sends the image data to an analysis service such as Google Cloud Vision API, analyzes the JSON response returned from the API, and lists the ingredients in the image.

[0222] Step 4:

[0223] The server generates a cooking recipe using a generative AI model based on the generated ingredient list. The input is the ingredient list, and the output is the generated cooking recipe. Specifically, the server creates a prompt sentence like the one below based on the ingredient list and sends it to a generative AI model such as OpenAI's GPT-4:

[0224] Example prompt:

[0225] "Create a delicious recipe using the following ingredients: chicken, onion, carrot, and tomato."

[0226] The generative AI model generates a recipe based on these prompts, and the server receives the results.

[0227] Step 5:

[0228] The server generates related cooking videos and articles. The input is a cooking recipe, and the output is related videos and articles. Specifically, the server searches for cooking videos related to the recipe using the YouTube API or Vimeo API, and provides the retrieved video links and articles to the user.

[0229] Step 6:

[0230] The server provides the generated recipe and related cooking videos and articles to the user. The input is the generated recipe and related content, and the output is the display on the user's device. Specifically, the server sends data in HTML or JSON format so that it can be displayed on the user's smartphone or PC browser.

[0231] Step 7:

[0232] It provides a virtual kitchen environment where users can perform cooking simulations in real time. The input is a cooking recipe and virtual kitchen environment data, and the output is a virtual experience on the user's HMD. Specifically, the server uses Unreal Engine or Unity to build the virtual kitchen, and the user wears the HMD to perform the cooking simulation in real time.

[0233] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0234] This invention combines a system that uses image analysis technology to automatically generate an ingredient list based on a user's image of the contents of their refrigerator, and then provides original cooking recipes based on that list with an emotion engine that recognizes the user's emotions, thereby enabling more personalized recipe suggestions. This system makes it possible to efficiently use ingredients in the refrigerator according to the user's condition, without wasting any ingredients.

[0235] System Overview

[0236] The system consists of the following main components:

[0237] 1. Image receiving means

[0238] It provides an interface for users to upload images of the inside of their refrigerator.

[0239] The server stores the images received from the user and prepares them for analysis.

[0240] 2. Image analysis methods

[0241] The server analyzes the received image and creates a list of ingredients. Here, an external image analysis API is used to recognize ingredients in the image and convert them into text data.

[0242] 3. Recipe Generation Method

[0243] The server uses a generative AI model based on the ingredient list to generate a cooking recipe that takes nutritional balance into consideration.

[0244] The generated recipe includes detailed information such as ingredients, steps, and cooking times.

[0245] 4. Emotion Engine

[0246] An engine that recognizes and analyzes user emotions.

[0247] For example, the user's facial expressions and voice can be analyzed to identify their emotional state (e.g., joy, sadness, stress).

[0248] 5. Means of providing information

[0249] The server has an interface for providing the generated recipe information to the user.

[0250] Users can receive recipe information through devices such as smartphones and PCs.

[0251] Specific processing content of the program

[0252] User Actions

[0253] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door so that the food inside is clearly visible, and then pressing the shutter button.

[0254] Upload the captured image to the system's application using the app's "Upload Image" function.

[0255] Server Processing

[0256] The server receives the uploaded image and immediately saves it in a specified directory.

[0257] The saved image is sent to an external image analysis API to generate an ingredient list, and the JSON response returned from the API is analyzed to extract the ingredient list.

[0258] For example: "Carrots, onions, tomatoes, chicken" would be included in the list.

[0259] The generated ingredient list is sent to a generative AI model to generate recipes that take nutritional balance into account.

[0260] Emotion engine processing

[0261] The emotion engine recognizes the user's emotional state and uses facial recognition cameras and voice analysis to identify emotions when a user accesses the app.

[0262] For example, it can detect "joy" from a user's facial expression.

[0263] The server uses the emotional data obtained from the emotion engine and sends it to the generative AI model as additional information, which generates a recipe that is optimal for the user's current emotional state.

[0264] Processing generative AI models

[0265] A generative AI model creates creative cooking recipes based on a provided list of ingredients and user sentiment data, including ingredients, steps, cooking times, and nutritional information.

[0266] For example, recipes such as "Tomato and chicken stew" and "Carrot and onion salad" are generated, but the more suitable recipe is selected based on the user's emotions.

[0267] Server information provision

[0268] The server provides the generated recipe to the user, and the user interface displays detailed information about the recipe to the user.

[0269] Users can view the provided recipes and actually cook them.

[0270] Specific examples

[0271] Suppose a user takes a photo of the contents of their refrigerator and uploads the image to the system. The server receives this image and uses an image analysis API to generate an ingredient list such as "carrots, onions, tomatoes, and chicken." The server then uses an emotion engine to recognize the user's emotions before sending the ingredient list to the generative AI model. For example, if the emotion engine detects a "stressed state" from the user's facial expression, the generative AI model will prioritize generating recipes that have a relaxing effect on the user. In this case, a recipe such as "soup to make you feel calm" would be selected. Finally, the server provides this recipe to the user, who then begins cooking according to the recipe.

[0272] In this way, this system not only helps users make effective use of the ingredients in their refrigerator, but also provides a more personalized dining experience by suggesting recipes that match the user's emotional state.

[0273] The processing flow will be explained below.

[0274] Step 1:

[0275] A user takes a picture of the contents of the refrigerator with their smartphone. The user launches the smartphone's camera app, opens the refrigerator door, and presses the shutter button after making sure the food inside is clearly visible.

[0276] Step 2:

[0277] Users can upload images they have taken to the system's application by tapping the "Upload Image" button on the application screen, selecting an image from the album, and sending it.

[0278] Step 3:

[0279] The server receives the uploaded image. The server listens to the HTTP request and saves the received image data in a specified directory.

[0280] Step 4:

[0281] The server sends the received image to an external image analysis API, which then sends the image file to an API endpoint and executes the image analysis request.

[0282] Step 5:

[0283] The server receives a JSON response from the image analysis API. The response contains text data of the analyzed ingredients, such as a list of ingredients such as "carrot, onion, tomato, and chicken."

[0284] Step 6:

[0285] The server retrieves the generated ingredient list and sends it to the generative AI model. The server then sends a request including the ingredient list to the API endpoint of the generative AI model.

[0286] Step 7:

[0287] The device recognizes the user's emotional state. The device uses a camera and microphone to analyze the user's facial expressions and voice, and the emotion engine analyzes that data to identify the user's emotional state.

[0288] Step 8:

[0289] The device transmits the emotion data obtained from the emotion engine to the server, which uses this data as information for creating recipes.

[0290] Step 9:

[0291] A generative AI model generates a cooking recipe based on the ingredient list and emotional data. The generative AI model runs an algorithm to generate a cooking recipe that best suits the user's emotional state. For example, if the user is in a "stressed state," the model will prioritize suggestions that have a relaxing effect.

[0292] Step 10:

[0293] The server provides the generated recipe to the user, converts the received recipe data into a format for display on the user interface, and displays detailed recipe information on the user's smartphone screen.

[0294] Step 11:

[0295] Users can select the dish they want to make from the displayed recipes by tapping on their smartphone screen.

[0296] Step 12:

[0297] The app will then create a recipe based on the user's chosen recipe, and the user will follow the instructions displayed on the app to select the ingredients they need and begin cooking.

[0298] Example 2

[0299] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0300] Currently, there are systems that efficiently suggest cooking recipes using ingredients in the refrigerator, but they do not suggest personalized recipes based on the user's emotional state. As a result, there is a problem in that they cannot provide recipes that match the user's emotional state and preferences.

[0301] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0302] In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for sending the received images to an external image analysis API to generate an ingredient list, means for generating a cooking recipe using a generative AI model based on the ingredient list, means for recognizing the user's emotional state and selecting a personalized cooking recipe based on that, and means for providing the generated cooking recipe to the user, thereby enabling efficient and personalized cooking recipe suggestions tailored to the user's emotional state.

[0303] "User" refers to an individual who uses this system to take and upload images of the inside of a refrigerator.

[0304] "Means for receiving" refers to a device or program that has the function of transmitting images taken by a user to a server and receiving the image data.

[0305] "Image analysis API" refers to a program interface provided by an external service provider for extracting specific information from image data.

[0306] "Ingredient list" refers to text data in list format that shows the names and quantities of ingredients in the refrigerator, extracted using an image analysis API.

[0307] A "generative AI model" refers to an artificial intelligence algorithm that generates new information or content based on provided data.

[0308] A "cooking recipe" refers to detailed information that shows how to make a dish and the steps involved, generated based on a list of ingredients.

[0309] "Emotional state" refers to data that indicates a user's current emotional or psychological state.

[0310] "Personalized cooking recipes" refer to cooking recipes that are optimized based on the user's emotional state and individual preferences.

[0311] "Means for providing" refers to an interface or device for showing the generated cooking recipe to the user.

[0312] This invention combines a system that uses image analysis technology to automatically generate an ingredient list based on a user's image of the contents of their refrigerator, and then provides original cooking recipes based on that list with an emotion engine that recognizes the user's emotions, thereby enabling more personalized recipe suggestions. This system makes it possible to efficiently use ingredients in the refrigerator according to the user's condition, without wasting any ingredients.

[0313] The main components of the system include:

[0314] 1. Image receiving means

[0315] Provide an interface for users to upload images of the inside of their refrigerator. For example, users can take a picture of the contents of their refrigerator with their smartphone and use the "upload image" function within the app to send the image to the server.

[0316] 2. Image analysis methods

[0317] The server analyzes the received image and generates an ingredient list. This analysis uses an external image analysis API to recognize ingredients in the image and convert them into text data. For example, an ingredient list such as "carrot, onion, tomato, chicken" is extracted.

[0318] 3. Recipe Generation Method

[0319] The server generates a nutritionally balanced cooking recipe based on the generated ingredient list, using a generative AI model to create a recipe that includes detailed information such as ingredients, steps, and cooking time.

[0320] 4. Emotion recognition means

[0321] The emotion engine recognizes the user's emotional state. Specifically, when a user accesses the app, it uses a facial recognition camera and voice analysis to identify emotions. For example, it can detect "joy" or "stress" from the user's facial expressions.

[0322] 5. Means of providing information

[0323] The server has an interface to provide the generated recipes to users, who can receive the recipe information via their smartphones, PCs, or other devices.

[0324] Here is an example of this system:

[0325] 1. The user takes a picture of the contents of their refrigerator and uploads the image to the system using the "upload image" function within the app.

[0326] 2. The server receives this image and sends the saved image to an external image analysis API, which generates a list of ingredients, for example, "carrot, onion, tomato, chicken."

[0327] 3. The server sends the ingredient list to the generative AI model to generate a cooking recipe.

[0328] 4. The emotion engine recognizes the user’s emotional state. For example, if it detects a “stressed state,” the generative AI model will prioritize recipes that will have a relaxing effect on the user. In this case, a recipe such as “soup to make you feel calm” will be selected.

[0329] 5. Finally, the server provides this recipe to the user, who then starts cooking according to the recipe.

[0330] Prompt Sentence Examples

[0331] "Please upload a picture of the ingredients in your refrigerator."

[0332] "Generating the best recipe for your current emotional state. Please wait a moment."

[0333] "The following recipe was generated: 'Comforting Soup'."

[0334] Check out this recipe and start cooking.

[0335] In this way, this system not only allows users to make effective use of the ingredients in their refrigerator, but also provides a more personalized dining experience by suggesting recipes that match the user's emotional state.

[0336] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0337] Step 1:

[0338] The user takes a picture of the inside of the refrigerator with their smartphone. The user opens the refrigerator door to clearly see the food inside and presses the shutter button. The image is then saved on the smartphone.

[0339] Input: Actual ingredients in the fridge

[0340] Output: Food images stored on the smartphone

[0341] Step 2:

[0342] The user opens the system application and uploads the captured image. Using the "Upload Image" function in the application, the user selects the saved image file and sends it to the server.

[0343] Input: Food images stored on your smartphone

[0344] Output: Food images uploaded to the server

[0345] Step 3:

[0346] The server receives the uploaded image and immediately saves it in a specified directory.

[0347] Input: User-submitted food images

[0348] Output: Food images saved in a specified directory on the server

[0349] Step 4:

[0350] The server sends the stored image to an external image analysis API and requests analysis. It sends the image data using an HTTP request and receives a JSON-formatted response from the API.

[0351] Input: Food images stored on the server

[0352] Output: A list of ingredients in JSON format returned by the API

[0353] Step 5:

[0354] The server parses the JSON response returned from the API and extracts the list of ingredients. Specifically, it extracts the ingredient names from the JSON data and converts them into a text list.

[0355] Input: JSON formatted list of ingredients returned from the API

[0356] Output: Ingredient list in text format

[0357] Step 6:

[0358] The server sends the generated ingredient list to the generative AI model and requests it to generate a recipe. The server then sends the ingredient list to the generative AI model as input data and waits for the recipe to be generated.

[0359] Input: Text list of ingredients

[0360] Output: Input data sent to the generative AI model

[0361] Step 7:

[0362] The server activates the emotion engine to recognize the user's emotional state. When a user accesses the app, the camera and microphone are used to capture facial expressions and voice, and emotion recognition algorithms are applied.

[0363] Input: Facial expressions and voice data captured through a camera and microphone

[0364] Output: Recognized emotional state of the user

[0365] Step 8:

[0366] The server sends the emotion data obtained from the emotion engine to the generative AI model, which then uses the emotion data as additional information to generate recipes.

[0367] Input: Perceived emotional state of the user

[0368] Output: Emotion data sent to the generative AI model

[0369] Step 9:

[0370] The generative AI model creates creative cooking recipes based on the provided ingredient list and emotional data. Specifically, the algorithm inputs the ingredient list and emotional data to generate a recipe that includes ingredients, steps, cooking time, and nutritional information.

[0371] Input: Text list of ingredients and emotion data

[0372] Output: Generated cooking recipe

[0373] Step 10:

[0374] The server provides the generated recipe to the user, and the recipe details are displayed through an app or web interface.

[0375] Input: Generated cooking recipe

[0376] Output: The cooking recipe displayed to the user

[0377] Step 11:

[0378] Users actually cook according to the provided recipe, preparing ingredients and following the steps on their smartphone or PC screen as a reference.

[0379] Input: A cooking recipe shown to the user

[0380] Output: The actual cooked food

[0381] (Application example 2)

[0382] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0383] Conventional systems that analyze images of the inside of a refrigerator to generate an ingredient list and provide cooking recipes suggest uniform recipes regardless of the user's state, which means they have the problem of not being able to suggest appropriate recipes that reflect the user's current emotions or preferences.In particular, when users shop in physical stores, there are no product suggestions or guidance that reflect their emotional state, making it difficult to provide an efficient and satisfying shopping experience.

[0384] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for analyzing the received images and generating an ingredient list, means for generating a cooking recipe based on the generated ingredient list, means for recognizing the user's emotions, means for generating a personalized cooking recipe based on the user's emotion data, and means for providing the generated cooking recipe to the user. This makes it possible to suggest ingredients and recipes that are optimal for the user's emotional state.

[0385] The "means for receiving images of the inside of the refrigerator taken by the user" is an interface for sending images of the inside of the refrigerator taken by the user using a smart device or the like to a server or cloud service.

[0386] The "means for analyzing received images and generating an ingredient list" is a process for identifying ingredients present in the refrigerator from the received image data using image analysis technology and creating a list of them.

[0387] The "means for generating a cooking recipe based on the generated ingredient list" refers to an algorithm or system for automatically generating a cooking recipe based on the ingredient information obtained by analysis.

[0388] "Means for recognizing user emotions" refers to hardware and software that analyzes the user's emotional state from their facial expressions and voice, and identifies emotions such as joy, sadness, and stress.

[0389] The "means for generating personalized cooking recipes based on user emotional data" is a process for generating cooking recipes that are optimally adjusted for each individual user based on the recognized emotional state of the user.

[0390] The "means for providing the generated cooking recipe to the user" refers to a communication interface or application for displaying or notifying the cooking recipe information generated by the server on the user's device.

[0391] The present invention is a system that allows a user to take a picture of the inside of a refrigerator, analyzes the picture to generate an ingredient list, and recognizes the user's emotional state to provide personalized cooking recipes. The system consists of the following main components:

[0392] 1. User Interface:

[0393] Users use devices such as smartphones or tablets to take pictures of the inside of the refrigerator and upload them to the application, which then sends the uploaded images to the system's server.

[0394] 2. Image analysis system:

[0395] The server analyzes the received image of the refrigerator interior using image recognition technology to generate an ingredient list. Specifically, it uses an external image analysis API to identify ingredients in the image and create a text list.

[0396] 3. Emotion Recognition System:

[0397] For emotion recognition, users input facial expressions and voice using their smart devices. The server analyzes these input data and uses an emotion recognition engine to identify the user's emotional state (e.g., joy, sadness, stress).

[0398] 4. Recipe Generation System:

[0399] The server uses a generative AI model to generate cooking recipes based on the generated ingredient list and emotional data. The cooking recipes are personalized, taking into account nutritional balance and the user's emotional state.

[0400] 5. Information provision system:

[0401] Finally, the server provides the generated cooking recipe to the user's device, where the user can access, view, and actually cook the recipe.

[0402] Hardware and software used

[0403] Smart devices (smartphones, tablets, etc.): Used by users to take pictures of the inside of the refrigerator and upload them to the application.

[0404] Server: Integrates and manages the image analysis system, emotion recognition system, recipe generation system, and information provision system.

[0405] Image analysis API: Uses an external API to analyze images of the inside of the refrigerator and identify ingredients.

[0406] Emotion recognition engine: An engine for identifying the user's emotional state from their facial expressions and voice.

[0407] Generative AI model: Generates personalized cooking recipes based on ingredient lists and emotional data.

[0408] Specific examples

[0409] For example, a user can use their smartphone to take a picture of the inside of their refrigerator and upload it to the system. The server receives this image and uses an image analysis API to generate an ingredient list such as "carrots, onions, tomatoes, and chicken." The server then uses the smart device's camera and microphone to recognize the user's emotional state and identify a "stressed state." Based on this information, the server uses a generative AI model to generate a recipe for a "calming soup" that will help the user relax and provides it to the user's smartphone. The user can then cook according to this recipe.

[0410] Prompt Sentence Examples

[0411] "When a user visits a physical store and starts shopping using the smart glasses, the system captures the user's face and recognizes their emotions. Depending on their emotions, the system will suggest the following product list:

[0412] 1. Happiness → Healthy foods, desserts

[0413] 2. Sadness → Chocolate, comfort food

[0414] 3. Stress → Herbal tea, relaxation items

[0415] This system allows users to effectively utilize ingredients in their refrigerators, receive personalized recipes based on their emotional state, and improve their shopping experience in brick-and-mortar stores.

[0416] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0417] Step 1:

[0418] A user takes a picture of the inside of the refrigerator using a smartphone or tablet and uploads the picture to the system through a dedicated application. The input is the image of the inside of the refrigerator, and the output is an image file sent to the server.

[0419] Step 2:

[0420] The server saves the image received from the user in a predetermined save directory. In this step, the image file sent from the user is received as input, and the image file is saved in the save directory as output.

[0421] Step 3:

[0422] The server sends the saved image to an external image analysis API and obtains the ingredient list as the analysis result. Here, the saved image file is used as input and the ingredient list in JSON format is obtained as output. Specifically, the ingredients in the image are identified and each is listed as text data.

[0423] Step 4:

[0424] The server analyzes the acquired ingredient list and sends it to the generative AI model. Here, the ingredient list is used as input, and data is prepared to be sent to the generative AI model as output.

[0425] Step 5:

[0426] The user inputs facial expressions and voice data into the system using the camera and microphone of the smart device. The input is the user's facial expressions and voice data, and the output is an audio file or image data sent to the server.

[0427] Step 6:

[0428] The server sends facial expression and voice data to an emotion recognition engine to identify the user's emotional state. Here, audio files and image data are used as input, and emotion recognition results are obtained as output. Specifically, emotional states such as "joy," "sadness," and "stress" are identified.

[0429] Step 7:

[0430] The server combines the emotion recognition results and the ingredient list and sends them to the generative AI model. Here, the emotional state data and ingredient list are used as input, and the data sent to the generative AI model as output is prepared.

[0431] Step 8:

[0432] A generative AI model generates personalized cooking recipes based on ingredient lists and emotional data. Here, the input is an ingredient list and emotional state data, and the output is a personalized cooking recipe, taking into account the ingredients, steps, time, and nutritional balance of the recipe.

[0433] Step 9:

[0434] The server provides the generated personalized cooking recipe to the user's device, and uses the generated cooking recipe as input to prepare recipe information to be displayed on the user's smartphone or tablet as output.

[0435] Step 10:

[0436] The user accesses the provided cooking recipe, browses it, and actually cooks the food. Here, the provided cooking recipe is used as input, and the completed dish is obtained as output. In concrete terms, the user cooks the food according to the recipe steps.

[0437] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0438] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0439] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0440] [Second embodiment]

[0441] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0442] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0443] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0444] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0445] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0446] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0447] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0448] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0449] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0450] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0451] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0452] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0453] The present invention relates to a system that uses image analysis technology to automatically generate an ingredient list by taking a picture of the contents of a refrigerator and providing the system with original cooking recipes based on that list. This system provides a means to efficiently use ingredients in the refrigerator without wasting them.

[0454] System Overview

[0455] The system consists of the following main components:

[0456] 1. Image receiving means

[0457] It provides an interface for users to upload images of the inside of their refrigerator.

[0458] The server stores the images received from the user and prepares them for analysis.

[0459] 2. Image analysis methods

[0460] The server analyzes the received image and creates a list of ingredients. Here, an external image analysis API is used to recognize ingredients in the image and convert them into text data.

[0461] 3. Recipe Generation Method

[0462] The server uses a generative AI model based on the ingredient list to generate a cooking recipe that takes nutritional balance into consideration.

[0463] The generated recipe includes detailed information such as ingredients, steps, and cooking times.

[0464] 4. Means of providing information

[0465] The server has an interface for providing the generated recipe information to the user.

[0466] Users can receive recipe information through devices such as smartphones and PCs.

[0467] Specific processing content of the program

[0468] User Actions

[0469] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door so that the food inside is clearly visible, and then pressing the shutter button.

[0470] Upload the captured image to the system's application using the app's "Upload Image" function.

[0471] Server Processing

[0472] The server receives the uploaded image and immediately saves it in a specified directory.

[0473] The saved image is sent to an external image analysis API to generate an ingredient list, and the JSON response returned from the API is analyzed to extract the ingredient list.

[0474] For example: "Carrots, onions, tomatoes, chicken" would be included in the list.

[0475] The generated ingredient list is sent to a generative AI model to generate recipes that take nutritional balance into account.

[0476] Processing generative AI models

[0477] A generative AI model creates creative cooking recipes based on a provided list of ingredients, including ingredients, steps, cooking times, and nutritional information.

[0478] For example, recipes for "Tomato and Chicken Stew" and "Carrot and Onion Salad" are generated.

[0479] Server information provision

[0480] The server provides the generated recipe to the user, and the user interface displays detailed information about the recipe to the user.

[0481] Users can view the provided recipes and actually cook them.

[0482] Specific examples

[0483] Suppose a user takes a photo of the contents of their refrigerator and uploads the image to the system. The server receives the image and uses an image analysis API to generate a list of ingredients: "carrots, onions, tomatoes, and chicken." The server then sends this list to a generative AI model, which generates recipes for "tomato and chicken stew" and "carrot and onion salad." Finally, the server provides these recipes to the user, who then begins cooking according to the recipes.

[0484] In this way, this system allows users to make effective use of the ingredients in their refrigerator and reduce food waste.

[0485] The processing flow will be explained below.

[0486] Step 1:

[0487] A user takes a picture of the contents of the refrigerator with their smartphone. The user launches the smartphone's camera app, opens the refrigerator door, and presses the shutter button after making sure the food inside is clearly visible.

[0488] Step 2:

[0489] Users can upload images they have taken to the system's application by tapping the "Upload Image" button on the application screen, selecting an image from the album, and sending it.

[0490] Step 3:

[0491] The server receives the uploaded image. The server listens to the HTTP request and saves the received image data in a specified directory.

[0492] Step 4:

[0493] The server sends the received image to an external image analysis API, which then sends the image file to an API endpoint and executes the image analysis request.

[0494] Step 5:

[0495] The server receives a JSON response from the image analysis API. The response contains text data of the analyzed ingredients, such as a list of ingredients such as "carrot, onion, tomato, and chicken."

[0496] Step 6:

[0497] The server retrieves the generated ingredient list and sends it to the generative AI model. The server then sends a request including the ingredient list to the API endpoint of the generative AI model.

[0498] Step 7:

[0499] A generative AI model generates a cooking recipe based on an ingredient list. The model combines ingredients and generates multiple recipe ideas, taking into account nutritional balance and cooking procedures. For example, it generates recipes such as "Tomato and Chicken Stew" and "Carrot and Onion Salad."

[0500] Step 8:

[0501] The server receives the recipe returned from the generative AI model. The server receives the response from the recipe generation API and analyzes the response data.

[0502] Step 9:

[0503] The server displays the generated recipe to the user, converts the received recipe data into a format for display on the user interface, and displays detailed recipe information on the user's smartphone screen.

[0504] Step 10:

[0505] Users can select the dish they want to make from the displayed recipes by tapping on their smartphone screen.

[0506] Step 11:

[0507] The app will then create a recipe based on the user's chosen recipe, and the user will follow the instructions displayed on the app to select the ingredients they need and begin cooking.

[0508] Example 1

[0509] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0510] In modern society, there is a problem that purchased ingredients are often forgotten in the refrigerator and wasted. Furthermore, many users find it difficult to easily plan and execute nutritionally balanced meals at home amid busy lifestyles. This has led to increased food waste and health concerns. The purpose of this invention is to solve these problems by providing a system that effectively utilizes ingredients in the refrigerator and efficiently provides nutritionally balanced meals.

[0511] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0512] In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for saving the received images, means for calling an external image analysis API to analyze the saved images and generate an ingredient list, means for using a generative AI model to generate cooking recipes that take nutritional balance into consideration based on the ingredient list obtained through the analysis, and means for providing the generated cooking recipes through a user interface, thereby enabling users to efficiently plan and execute nutritionally balanced meals without wasting ingredients in the refrigerator.

[0513] "User" refers to an individual who uses the system to take pictures of the inside of a refrigerator and upload them to the system.

[0514] "Server" refers to the central computing device that receives and processes images uploaded by users.

[0515] "Image receiving means" refers to a function or device that allows the server to receive images taken by the user.

[0516] "Storage means" refers to a function or device for storing received images in a predetermined location.

[0517] "Image analysis API" refers to an external application program interface that analyzes received images and recognizes and lists ingredients within the images.

[0518] "Ingredient list" refers to a list of ingredients in an image obtained as a result of analysis by the image analysis API.

[0519] "Generative AI model" refers to an artificial intelligence model used to generate creative cooking recipes based on an ingredient list.

[0520] "Recipe generation means" refers to a function or device for generating a cooking recipe that takes nutritional balance into consideration based on an ingredient list.

[0521] "User Interface" refers to an interactive screen or application that provides the generated cooking recipe to the user.

[0522] This invention relates to a system that uses image analysis technology to automatically generate an ingredient list by taking a picture of the contents of a refrigerator and providing the system with original cooking recipes based on that list. This system provides a means to efficiently use ingredients in the refrigerator without wasting them.

[0523] Key Components of the System

[0524] 1. Image receiving means

[0525] It provides an interface for users to upload images of the inside of their refrigerator.

[0526] The server stores the images received from the user and prepares them for analysis.

[0527] 2. Image analysis methods

[0528] The server analyzes the received image and creates a list of ingredients. Here, an external image analysis API is used to recognize ingredients in the image and convert them into text data.

[0529] 3. Recipe Generation Method

[0530] The server uses a generative AI model based on the ingredient list to generate a cooking recipe that takes nutritional balance into consideration.

[0531] The generated recipe includes detailed information such as ingredients, steps, and cooking times.

[0532] 4. Means of providing information

[0533] The server has an interface for providing the generated recipe information to the user.

[0534] Users can receive recipe information through devices such as smartphones and PCs.

[0535] Specific processing content of the program

[0536] 1. User Actions

[0537] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door so that the food inside is clearly visible, and then pressing the shutter button.

[0538] Upload the captured image to the system's application using the app's "Upload Image" function.

[0539] 2. Server Processing

[0540] The server receives the uploaded image and saves it in a specified directory.

[0541] The saved image is sent to an external image analysis API to generate an ingredient list, and the JSON response returned from the API is analyzed to extract the ingredient list.

[0542] For example: "Carrots, onions, tomatoes, chicken" would be included in the list.

[0543] The generated ingredient list is sent to a generative AI model to generate recipes that take nutritional balance into account.

[0544] 3. Processing of generative AI models

[0545] A generative AI model creates creative cooking recipes based on a provided list of ingredients, including ingredients, steps, cooking times, and nutritional information.

[0546] For example, recipes for "Tomato and Chicken Stew" and "Carrot and Onion Salad" are generated.

[0547] 4. Server Information

[0548] The server provides the generated recipe to the user, and the user interface displays detailed information about the recipe to the user.

[0549] Users can view the provided recipes and actually cook them.

[0550] Specific examples

[0551] Suppose a user takes a photo of the contents of their refrigerator and uploads the image to the system. The server receives the image and uses an image analysis API to generate a list of ingredients: "carrots, onions, tomatoes, and chicken." The server then sends this list to a generative AI model, which generates recipes for "tomato and chicken stew" and "carrot and onion salad." Finally, the server provides these recipes to the user, who then begins cooking according to the recipes.

[0552] Prompt Sentence Examples

[0553] An example of a prompt to send to the image analysis API is:

[0554] {

[0555] "action": "parse_image",

[0556] "image_url": "YOUR_IMAGE_URL",

[0557] "response_format": "json"

[0558] }

[0559] Here's an example prompt to send to a generative AI model:

[0560] Recipe category to create: Cooking

[0561] Ingredients: carrots, onions, tomatoes, chicken

[0562] Generate two recipes, taking nutritional balance into consideration.

[0563] In this way, the present invention allows users to make better use of food in their refrigerators and reduce food waste.

[0564] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0565] Step 1:

[0566] User image capture and upload

[0567] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door and pressing the shutter button so that the food inside is clearly visible.

[0568] The captured image is uploaded to the system's application. The user opens the app, taps the "Upload Image" button, selects an image, and presses the "Send" button.

[0569] Input: Image of the contents of the refrigerator taken with a smartphone

[0570] Output: Images uploaded to the system

[0571] Step 2:

[0572] Receiving and saving images on the server

[0573] The server receives the uploaded image. The image data is sent to the server via an HTTP request.

[0574] Save the received images in a specified directory (e.g., / images / uploads).

[0575] Log the save directory path to ensure the image has been saved successfully.

[0576] Input: Image data uploaded by the user

[0577] Output: Images saved in the specified directory on the server

[0578] Step 3:

[0579] Calling the server's image analysis API

[0580] The server sends the stored image to an external image analysis API using an HTTP POST request, sending the image's binary data and URL to the API.

[0581] The image analysis API analyzes the image and returns a list of ingredients in JSON format.

[0582] The server parses the JSON response and extracts the list of ingredients in the image.

[0583] Input: Path to the image stored on the server

[0584] Output: Generated list of ingredients (e.g. carrot, onion, tomato, chicken)

[0585] Step 4:

[0586] Server recipe generation

[0587] Based on the ingredient list, the server sends a prompt to the generative AI model.

[0588] Recipe category to create: Cooking

[0589] Ingredients: carrots, onions, tomatoes, chicken

[0590] Generate two recipes, taking nutritional balance into consideration.

[0591] The generative AI model generates a recipe and returns it to the server, including ingredients, steps, cooking time, and nutritional information.

[0592] Input: Generated ingredient list and prompt statement

[0593] Output: Generated cooking recipe

[0594] Step 5:

[0595] Server recipe provision

[0596] The server displays the generated recipe in the user interface, allowing the user to view detailed recipe information on the app's "View Recipe" screen.

[0597] It allows users to browse recipes and provides instructions for actually cooking the food.

[0598] Input: Generated cooking recipe

[0599] Output: Recipe information displayed in the user interface

[0600] Through these steps, the system enables users to effectively utilize the ingredients in their refrigerator and provide efficient, nutritionally balanced meals.

[0601] (Application example 1)

[0602] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0603] Modern households are expected to efficiently utilize the ingredients in their refrigerators, but this often fails. Food waste is a problem due to ineffective use of ingredients. Many people are looking for new recipes and cooking videos when it comes to home cooking, but finding the right content can be a hassle. Furthermore, there is a lack of real-time interactive tools to improve cooking skills at home. Providing an effective system to solve these problems is a challenge.

[0604] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0605] In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for analyzing the received images to generate an ingredient list, means for generating cooking videos and articles related to cooking recipes based on the generated ingredient list, means for providing the user with cooking videos and articles related to the generated cooking recipes, and means for providing a virtual kitchen environment in which the user can perform cooking simulations in real time. This not only enables efficient use of ingredients in the refrigerator, but also makes it possible to provide the user with new recipes and cooking videos to help improve their cooking skills at home.

[0606] A "user" is someone who uses this system to manage ingredients in their refrigerator and obtain cooking recipes and related content.

[0607] "Photographing" refers to the act of taking a photo of the food in the refrigerator using a smartphone or other image capture device.

[0608] "Images inside the refrigerator" are photographic data of the inside of the refrigerator taken by the user, showing the state of the ingredients.

[0609] "Analysis" is the process by which the received image is processed by software to identify and list elements within the image.

[0610] The "ingredient list" is a list of names and information of ingredients extracted from the analyzed images of the inside of the refrigerator.

[0611] A "cooking recipe" is a document that contains specific cooking steps and a list of necessary ingredients, generated based on an ingredient list.

[0612] "Cooking videos" are video content that visually show cooking procedures and can be viewed by users.

[0613] An "article" is a text-based explanation related to cooking methods and ingredients.

[0614] A "virtual kitchen environment" is a virtual space that simulates a real-world kitchen and allows users to interactively experience cooking procedures.

[0615] "Real-time cooking simulation" refers to a user experiencing a simulation in which they react instantly within a virtual kitchen environment.

[0616] This invention is a system that efficiently utilizes ingredients in a refrigerator and provides users with cooking recipes and related content. This system is implemented mainly using a smartphone and a server.

[0617] 1. User Action

[0618] The user takes a picture of the food in the refrigerator using their smartphone. After taking the picture, the user uploads the picture to the server using the system's dedicated application. At this stage, the "Image Upload" function is used.

[0619] 2. Server processing

[0620] The server saves the received images in a specified directory. Then, it sends the saved images to an external image analysis API (e.g., Google Cloud Vision API) to generate an ingredient list. It parses the JSON response returned by the API and extracts the ingredient list. Specifically, the following prompt sentence is used:

[0621] Example prompt:

[0622] "Generate a list of ingredients in an image. Analyze the image and list the names of the ingredients."

[0623] 3. Recipe and content generation

[0624] The server generates a recipe based on the generated ingredient list using a generative AI model (e.g., OpenAI's GPT-4), taking nutritional balance into consideration. The generated recipe includes detailed information such as ingredients, steps, and cooking time. Specifically, the following prompt is used:

[0625] Example prompt:

[0626] "Create a delicious recipe using the following ingredients: chicken, onion, carrot, and tomato."

[0627] Additionally, to provide you with related cooking videos and articles, we use cooking video services (e.g., YouTube API, Vimeo API) to search for and provide related content to you.

[0628] 4. Virtual Kitchen Environment

[0629] The server provides a 3D virtual environment using platforms such as Unreal Engine and Unity, allowing users to simulate real-time cooking in a virtual kitchen environment, allowing them to improve their cooking skills by recreating real-life cooking procedures in a virtual environment.

[0630] Specific examples

[0631] A user takes a photo of the contents of their refrigerator with their smartphone and uploads the image to the system's application. The server receives the image and uses the Google Cloud Vision API to generate an ingredient list: "carrots, onions, tomatoes, and chicken." Next, GPT-4 is used to generate recipes: "Tomato and Chicken Stew" and "Carrot and Onion Salad." Additionally, cooking videos for these recipes are provided using the YouTube API. Users can browse these recipes and videos and experience cooking simulations in a virtual kitchen environment using an HMD (e.g., Oculus Rift).

[0632] This system allows users to efficiently use ingredients in their refrigerators and easily acquire and try out new recipes and cooking methods, which is expected to not only reduce food waste but also improve the cooking experience at home.

[0633] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0634] Step 1:

[0635] The user takes a picture of the inside of the refrigerator with their smartphone and uploads the picture to the system's application. The input is the image taken by the user, and the output is the upload of the image data to the server. Specifically, the user takes a picture of the food in the refrigerator using the camera app on their smartphone, and then uses the "image upload" function in the app to send the picture to the server.

[0636] Step 2:

[0637] The server saves the images it receives in a specified directory. The input is the uploaded image data, and the output is an image file saved on the server. Specifically, the server receives an HTTP request and saves the sent image data in a specific folder.

[0638] Step 3:

[0639] The server sends the saved image to an external image analysis API to generate an ingredient list. The input is the saved image file, and the output is the ingredient list. Specifically, the server sends the image data to an analysis service such as Google Cloud Vision API, analyzes the JSON response returned from the API, and lists the ingredients in the image.

[0640] Step 4:

[0641] The server generates a cooking recipe using a generative AI model based on the generated ingredient list. The input is the ingredient list, and the output is the generated cooking recipe. Specifically, the server creates a prompt sentence like the one below based on the ingredient list and sends it to a generative AI model such as OpenAI's GPT-4:

[0642] Example prompt:

[0643] "Create a delicious recipe using the following ingredients: chicken, onion, carrot, and tomato."

[0644] The generative AI model generates a recipe based on these prompts, and the server receives the results.

[0645] Step 5:

[0646] The server generates related cooking videos and articles. The input is a cooking recipe, and the output is related videos and articles. Specifically, the server searches for cooking videos related to the recipe using the YouTube API or Vimeo API, and provides the retrieved video links and articles to the user.

[0647] Step 6:

[0648] The server provides the generated recipe and related cooking videos and articles to the user. The input is the generated recipe and related content, and the output is the display on the user's device. Specifically, the server sends data in HTML or JSON format so that it can be displayed on the user's smartphone or PC browser.

[0649] Step 7:

[0650] It provides a virtual kitchen environment where users can perform cooking simulations in real time. The input is a cooking recipe and virtual kitchen environment data, and the output is a virtual experience on the user's HMD. Specifically, the server uses Unreal Engine or Unity to build the virtual kitchen, and the user wears the HMD to perform the cooking simulation in real time.

[0651] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0652] This invention combines a system that uses image analysis technology to automatically generate an ingredient list based on a user's image of the contents of their refrigerator, and then provides original cooking recipes based on that list with an emotion engine that recognizes the user's emotions, thereby enabling more personalized recipe suggestions. This system makes it possible to efficiently use ingredients in the refrigerator according to the user's condition, without wasting any ingredients.

[0653] System Overview

[0654] The system consists of the following main components:

[0655] 1. Image receiving means

[0656] It provides an interface for users to upload images of the inside of their refrigerator.

[0657] The server stores the images received from the user and prepares them for analysis.

[0658] 2. Image analysis methods

[0659] The server analyzes the received image and creates a list of ingredients. Here, an external image analysis API is used to recognize ingredients in the image and convert them into text data.

[0660] 3. Recipe Generation Method

[0661] The server uses a generative AI model based on the ingredient list to generate a cooking recipe that takes nutritional balance into consideration.

[0662] The generated recipe includes detailed information such as ingredients, steps, and cooking times.

[0663] 4. Emotion Engine

[0664] An engine that recognizes and analyzes user emotions.

[0665] For example, the user's facial expressions and voice can be analyzed to identify their emotional state (e.g., joy, sadness, stress).

[0666] 5. Means of providing information

[0667] The server has an interface for providing the generated recipe information to the user.

[0668] Users can receive recipe information through devices such as smartphones and PCs.

[0669] Specific processing content of the program

[0670] User Actions

[0671] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door so that the food inside is clearly visible, and then pressing the shutter button.

[0672] Upload the captured image to the system's application using the app's "Upload Image" function.

[0673] Server Processing

[0674] The server receives the uploaded image and immediately saves it in a specified directory.

[0675] The saved image is sent to an external image analysis API to generate an ingredient list, and the JSON response returned from the API is analyzed to extract the ingredient list.

[0676] For example: "Carrots, onions, tomatoes, chicken" would be included in the list.

[0677] The generated ingredient list is sent to a generative AI model to generate recipes that take nutritional balance into account.

[0678] Emotion engine processing

[0679] The emotion engine recognizes the user's emotional state and uses facial recognition cameras and voice analysis to identify emotions when a user accesses the app.

[0680] For example, it can detect "joy" from a user's facial expression.

[0681] The server uses the emotional data obtained from the emotion engine and sends it to the generative AI model as additional information, which generates a recipe that is optimal for the user's current emotional state.

[0682] Processing generative AI models

[0683] A generative AI model creates creative cooking recipes based on a provided list of ingredients and user sentiment data, including ingredients, steps, cooking times, and nutritional information.

[0684] For example, recipes such as "Tomato and chicken stew" and "Carrot and onion salad" are generated, but the more suitable recipe is selected based on the user's emotions.

[0685] Server information provision

[0686] The server provides the generated recipe to the user, and the user interface displays detailed information about the recipe to the user.

[0687] Users can view the provided recipes and actually cook them.

[0688] Specific examples

[0689] Suppose a user takes a photo of the contents of their refrigerator and uploads the image to the system. The server receives this image and uses an image analysis API to generate an ingredient list such as "carrots, onions, tomatoes, and chicken." The server then uses an emotion engine to recognize the user's emotions before sending the ingredient list to the generative AI model. For example, if the emotion engine detects a "stressed state" from the user's facial expression, the generative AI model will prioritize generating recipes that have a relaxing effect on the user. In this case, a recipe such as "soup to make you feel calm" would be selected. Finally, the server provides this recipe to the user, who then begins cooking according to the recipe.

[0690] In this way, this system not only helps users make effective use of the ingredients in their refrigerator, but also provides a more personalized dining experience by suggesting recipes that match the user's emotional state.

[0691] The processing flow will be explained below.

[0692] Step 1:

[0693] A user takes a picture of the contents of the refrigerator with their smartphone. The user launches the smartphone's camera app, opens the refrigerator door, and presses the shutter button after making sure the food inside is clearly visible.

[0694] Step 2:

[0695] Users can upload images they have taken to the system's application by tapping the "Upload Image" button on the application screen, selecting an image from the album, and sending it.

[0696] Step 3:

[0697] The server receives the uploaded image. The server listens to the HTTP request and saves the received image data in a specified directory.

[0698] Step 4:

[0699] The server sends the received image to an external image analysis API, which then sends the image file to an API endpoint and executes the image analysis request.

[0700] Step 5:

[0701] The server receives a JSON response from the image analysis API. The response contains text data of the analyzed ingredients, such as a list of ingredients such as "carrot, onion, tomato, and chicken."

[0702] Step 6:

[0703] The server retrieves the generated ingredient list and sends it to the generative AI model. The server then sends a request including the ingredient list to the API endpoint of the generative AI model.

[0704] Step 7:

[0705] The device recognizes the user's emotional state. The device uses a camera and microphone to analyze the user's facial expressions and voice, and the emotion engine analyzes that data to identify the user's emotional state.

[0706] Step 8:

[0707] The device transmits the emotion data obtained from the emotion engine to the server, which uses this data as information for creating recipes.

[0708] Step 9:

[0709] A generative AI model generates a cooking recipe based on the ingredient list and emotional data. The generative AI model runs an algorithm to generate a cooking recipe that best suits the user's emotional state. For example, if the user is in a "stressed state," the model will prioritize suggestions that have a relaxing effect.

[0710] Step 10:

[0711] The server provides the generated recipe to the user, converts the received recipe data into a format for display on the user interface, and displays detailed recipe information on the user's smartphone screen.

[0712] Step 11:

[0713] Users can select the dish they want to make from the displayed recipes by tapping on their smartphone screen.

[0714] Step 12:

[0715] The app will then create a recipe based on the user's chosen recipe, and the user will follow the instructions displayed on the app to select the ingredients they need and begin cooking.

[0716] Example 2

[0717] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0718] Currently, there are systems that efficiently suggest cooking recipes using ingredients in the refrigerator, but they do not suggest personalized recipes based on the user's emotional state. As a result, there is a problem in that they cannot provide recipes that match the user's emotional state and preferences.

[0719] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0720] In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for sending the received images to an external image analysis API to generate an ingredient list, means for generating a cooking recipe using a generative AI model based on the ingredient list, means for recognizing the user's emotional state and selecting a personalized cooking recipe based on that, and means for providing the generated cooking recipe to the user, thereby enabling efficient and personalized cooking recipe suggestions tailored to the user's emotional state.

[0721] "User" refers to an individual who uses this system to take and upload images of the inside of a refrigerator.

[0722] "Means for receiving" refers to a device or program that has the function of transmitting images taken by a user to a server and receiving the image data.

[0723] "Image analysis API" refers to a program interface provided by an external service provider for extracting specific information from image data.

[0724] "Ingredient list" refers to text data in list format that shows the names and quantities of ingredients in the refrigerator, extracted using an image analysis API.

[0725] A "generative AI model" refers to an artificial intelligence algorithm that generates new information or content based on provided data.

[0726] A "cooking recipe" refers to detailed information that shows how to make a dish and the steps involved, generated based on a list of ingredients.

[0727] "Emotional state" refers to data that indicates a user's current emotional or psychological state.

[0728] "Personalized cooking recipes" refer to cooking recipes that are optimized based on the user's emotional state and individual preferences.

[0729] "Means for providing" refers to an interface or device for showing the generated cooking recipe to the user.

[0730] This invention combines a system that uses image analysis technology to automatically generate an ingredient list based on a user's image of the contents of their refrigerator, and then provides original cooking recipes based on that list with an emotion engine that recognizes the user's emotions, thereby enabling more personalized recipe suggestions. This system makes it possible to efficiently use ingredients in the refrigerator according to the user's condition, without wasting any ingredients.

[0731] The main components of the system include:

[0732] 1. Image receiving means

[0733] Provide an interface for users to upload images of the inside of their refrigerator. For example, users can take a picture of the contents of their refrigerator with their smartphone and use the "upload image" function within the app to send the image to the server.

[0734] 2. Image analysis methods

[0735] The server analyzes the received image and generates an ingredient list. This analysis uses an external image analysis API to recognize ingredients in the image and convert them into text data. For example, an ingredient list such as "carrot, onion, tomato, chicken" is extracted.

[0736] 3. Recipe Generation Method

[0737] The server generates a nutritionally balanced cooking recipe based on the generated ingredient list, using a generative AI model to create a recipe that includes detailed information such as ingredients, steps, and cooking time.

[0738] 4. Emotion recognition means

[0739] The emotion engine recognizes the user's emotional state. Specifically, when a user accesses the app, it uses a facial recognition camera and voice analysis to identify emotions. For example, it can detect "joy" or "stress" from the user's facial expressions.

[0740] 5. Means of providing information

[0741] The server has an interface to provide the generated recipes to users, who can receive the recipe information via their smartphones, PCs, or other devices.

[0742] Here is an example of this system:

[0743] 1. The user takes a picture of the contents of their refrigerator and uploads the image to the system using the "upload image" function within the app.

[0744] 2. The server receives this image and sends the saved image to an external image analysis API, which generates a list of ingredients, for example, "carrot, onion, tomato, chicken."

[0745] 3. The server sends the ingredient list to the generative AI model to generate a cooking recipe.

[0746] 4. The emotion engine recognizes the user’s emotional state. For example, if it detects a “stressed state,” the generative AI model will prioritize recipes that will have a relaxing effect on the user. In this case, a recipe such as “soup to make you feel calm” will be selected.

[0747] 5. Finally, the server provides this recipe to the user, who then starts cooking according to the recipe.

[0748] Prompt Sentence Examples

[0749] "Please upload a picture of the ingredients in your refrigerator."

[0750] "Generating the best recipe for your current emotional state. Please wait a moment."

[0751] "The following recipe was generated: 'Comforting Soup'."

[0752] Check out this recipe and start cooking.

[0753] In this way, this system not only allows users to make effective use of the ingredients in their refrigerator, but also provides a more personalized dining experience by suggesting recipes that match the user's emotional state.

[0754] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0755] Step 1:

[0756] The user takes a picture of the inside of the refrigerator with their smartphone. The user opens the refrigerator door to clearly see the food inside and presses the shutter button. The image is then saved on the smartphone.

[0757] Input: Actual ingredients in the fridge

[0758] Output: Food images stored on the smartphone

[0759] Step 2:

[0760] The user opens the system application and uploads the captured image. Using the "Upload Image" function in the application, the user selects the saved image file and sends it to the server.

[0761] Input: Food images stored on your smartphone

[0762] Output: Food images uploaded to the server

[0763] Step 3:

[0764] The server receives the uploaded image and immediately saves it in a specified directory.

[0765] Input: User-submitted food images

[0766] Output: Food images saved in a specified directory on the server

[0767] Step 4:

[0768] The server sends the stored image to an external image analysis API and requests analysis. It sends the image data using an HTTP request and receives a JSON-formatted response from the API.

[0769] Input: Food images stored on the server

[0770] Output: A list of ingredients in JSON format returned by the API

[0771] Step 5:

[0772] The server parses the JSON response returned from the API and extracts the list of ingredients. Specifically, it extracts the ingredient names from the JSON data and converts them into a text list.

[0773] Input: JSON formatted list of ingredients returned from the API

[0774] Output: Ingredient list in text format

[0775] Step 6:

[0776] The server sends the generated ingredient list to the generative AI model and requests it to generate a recipe. The server then sends the ingredient list to the generative AI model as input data and waits for the recipe to be generated.

[0777] Input: Text list of ingredients

[0778] Output: Input data sent to the generative AI model

[0779] Step 7:

[0780] The server activates the emotion engine to recognize the user's emotional state. When a user accesses the app, the camera and microphone are used to capture facial expressions and voice, and emotion recognition algorithms are applied.

[0781] Input: Facial expressions and voice data captured through a camera and microphone

[0782] Output: Recognized emotional state of the user

[0783] Step 8:

[0784] The server sends the emotion data obtained from the emotion engine to the generative AI model, which then uses the emotion data as additional information to generate recipes.

[0785] Input: Perceived emotional state of the user

[0786] Output: Emotion data sent to the generative AI model

[0787] Step 9:

[0788] The generative AI model creates creative cooking recipes based on the provided ingredient list and emotional data. Specifically, the algorithm inputs the ingredient list and emotional data to generate a recipe that includes ingredients, steps, cooking time, and nutritional information.

[0789] Input: Text list of ingredients and emotion data

[0790] Output: Generated cooking recipe

[0791] Step 10:

[0792] The server provides the generated recipe to the user, and the recipe details are displayed through an app or web interface.

[0793] Input: Generated cooking recipe

[0794] Output: The cooking recipe displayed to the user

[0795] Step 11:

[0796] Users actually cook according to the provided recipe, preparing ingredients and following the steps on their smartphone or PC screen as a reference.

[0797] Input: A cooking recipe shown to the user

[0798] Output: The actual cooked food

[0799] (Application example 2)

[0800] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0801] Conventional systems that analyze images of the inside of a refrigerator to generate an ingredient list and provide cooking recipes suggest uniform recipes regardless of the user's state, which means they have the problem of not being able to suggest appropriate recipes that reflect the user's current emotions or preferences.In particular, when users shop in physical stores, there are no product suggestions or guidance that reflect their emotional state, making it difficult to provide an efficient and satisfying shopping experience.

[0802] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for analyzing the received images and generating an ingredient list, means for generating a cooking recipe based on the generated ingredient list, means for recognizing the user's emotions, means for generating a personalized cooking recipe based on the user's emotion data, and means for providing the generated cooking recipe to the user. This makes it possible to suggest ingredients and recipes that are optimal for the user's emotional state.

[0803] The "means for receiving images of the inside of the refrigerator taken by the user" is an interface for sending images of the inside of the refrigerator taken by the user using a smart device or the like to a server or cloud service.

[0804] The "means for analyzing received images and generating an ingredient list" is a process for identifying ingredients present in the refrigerator from the received image data using image analysis technology and creating a list of them.

[0805] The "means for generating a cooking recipe based on the generated ingredient list" refers to an algorithm or system for automatically generating a cooking recipe based on the ingredient information obtained by analysis.

[0806] "Means for recognizing user emotions" refers to hardware and software that analyzes the user's emotional state from their facial expressions and voice, and identifies emotions such as joy, sadness, and stress.

[0807] The "means for generating personalized cooking recipes based on user emotional data" is a process for generating cooking recipes that are optimally adjusted for each individual user based on the recognized emotional state of the user.

[0808] The "means for providing the generated cooking recipe to the user" refers to a communication interface or application for displaying or notifying the cooking recipe information generated by the server on the user's device.

[0809] The present invention is a system that allows a user to take a picture of the inside of a refrigerator, analyzes the picture to generate an ingredient list, and recognizes the user's emotional state to provide personalized cooking recipes. The system consists of the following main components:

[0810] 1. User Interface:

[0811] Users use devices such as smartphones or tablets to take pictures of the inside of the refrigerator and upload them to the application, which then sends the uploaded images to the system's server.

[0812] 2. Image analysis system:

[0813] The server analyzes the received image of the refrigerator interior using image recognition technology to generate an ingredient list. Specifically, it uses an external image analysis API to identify ingredients in the image and create a text list.

[0814] 3. Emotion Recognition System:

[0815] For emotion recognition, users input facial expressions and voice using their smart devices. The server analyzes these input data and uses an emotion recognition engine to identify the user's emotional state (e.g., joy, sadness, stress).

[0816] 4. Recipe Generation System:

[0817] The server uses a generative AI model to generate cooking recipes based on the generated ingredient list and emotional data. The cooking recipes are personalized, taking into account nutritional balance and the user's emotional state.

[0818] 5. Information provision system:

[0819] Finally, the server provides the generated cooking recipe to the user's device, where the user can access, view, and actually cook the recipe.

[0820] Hardware and software used

[0821] Smart devices (smartphones, tablets, etc.): Used by users to take pictures of the inside of the refrigerator and upload them to the application.

[0822] Server: Integrates and manages the image analysis system, emotion recognition system, recipe generation system, and information provision system.

[0823] Image analysis API: Uses an external API to analyze images of the inside of the refrigerator and identify ingredients.

[0824] Emotion recognition engine: An engine for identifying the user's emotional state from their facial expressions and voice.

[0825] Generative AI model: Generates personalized cooking recipes based on ingredient lists and emotional data.

[0826] Specific examples

[0827] For example, a user can use their smartphone to take a picture of the inside of their refrigerator and upload it to the system. The server receives this image and uses an image analysis API to generate an ingredient list such as "carrots, onions, tomatoes, and chicken." The server then uses the smart device's camera and microphone to recognize the user's emotional state and identify a "stressed state." Based on this information, the server uses a generative AI model to generate a recipe for a "calming soup" that will help the user relax and provides it to the user's smartphone. The user can then cook according to this recipe.

[0828] Prompt Sentence Examples

[0829] "When a user visits a physical store and starts shopping using the smart glasses, the system captures the user's face and recognizes their emotions. Depending on their emotions, the system will suggest the following product list:

[0830] 1. Happiness → Healthy foods, desserts

[0831] 2. Sadness → Chocolate, comfort food

[0832] 3. Stress → Herbal tea, relaxation items

[0833] This system allows users to effectively utilize ingredients in their refrigerators, receive personalized recipes based on their emotional state, and improve their shopping experience in brick-and-mortar stores.

[0834] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0835] Step 1:

[0836] A user takes a picture of the inside of the refrigerator using a smartphone or tablet and uploads the picture to the system through a dedicated application. The input is the image of the inside of the refrigerator, and the output is an image file sent to the server.

[0837] Step 2:

[0838] The server saves the image received from the user in a predetermined save directory. In this step, the image file sent from the user is received as input, and the image file is saved in the save directory as output.

[0839] Step 3:

[0840] The server sends the saved image to an external image analysis API and obtains the ingredient list as the analysis result. Here, the saved image file is used as input and the ingredient list in JSON format is obtained as output. Specifically, the ingredients in the image are identified and each is listed as text data.

[0841] Step 4:

[0842] The server analyzes the acquired ingredient list and sends it to the generative AI model. Here, the ingredient list is used as input, and data is prepared to be sent to the generative AI model as output.

[0843] Step 5:

[0844] The user inputs facial expressions and voice data into the system using the camera and microphone of the smart device. The input is the user's facial expressions and voice data, and the output is an audio file or image data sent to the server.

[0845] Step 6:

[0846] The server sends facial expression and voice data to an emotion recognition engine to identify the user's emotional state. Here, audio files and image data are used as input, and emotion recognition results are obtained as output. Specifically, emotional states such as "joy," "sadness," and "stress" are identified.

[0847] Step 7:

[0848] The server combines the emotion recognition results and the ingredient list and sends them to the generative AI model. Here, the emotional state data and ingredient list are used as input, and the data sent to the generative AI model as output is prepared.

[0849] Step 8:

[0850] A generative AI model generates personalized cooking recipes based on ingredient lists and emotional data. Here, the input is an ingredient list and emotional state data, and the output is a personalized cooking recipe, taking into account the ingredients, steps, time, and nutritional balance of the recipe.

[0851] Step 9:

[0852] The server provides the generated personalized cooking recipe to the user's device, and uses the generated cooking recipe as input to prepare recipe information to be displayed on the user's smartphone or tablet as output.

[0853] Step 10:

[0854] The user accesses the provided cooking recipe, browses it, and actually cooks the food. Here, the provided cooking recipe is used as input, and the completed dish is obtained as output. In concrete terms, the user cooks the food according to the recipe steps.

[0855] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0856] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0857] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0858] [Third embodiment]

[0859] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0860] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0861] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0862] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0863] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0864] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0865] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0866] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0867] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0868] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0869] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0870] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0871] The present invention relates to a system that uses image analysis technology to automatically generate an ingredient list by taking a picture of the contents of a refrigerator and providing the system with original cooking recipes based on that list. This system provides a means to efficiently use ingredients in the refrigerator without wasting them.

[0872] System Overview

[0873] The system consists of the following main components:

[0874] 1. Image receiving means

[0875] It provides an interface for users to upload images of the inside of their refrigerator.

[0876] The server stores the images received from the user and prepares them for analysis.

[0877] 2. Image analysis methods

[0878] The server analyzes the received image and creates a list of ingredients. Here, an external image analysis API is used to recognize ingredients in the image and convert them into text data.

[0879] 3. Recipe Generation Method

[0880] The server uses a generative AI model based on the ingredient list to generate a cooking recipe that takes nutritional balance into consideration.

[0881] The generated recipe includes detailed information such as ingredients, steps, and cooking times.

[0882] 4. Means of providing information

[0883] The server has an interface for providing the generated recipe information to the user.

[0884] Users can receive recipe information through devices such as smartphones and PCs.

[0885] Specific processing content of the program

[0886] User Actions

[0887] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door so that the food inside is clearly visible, and then pressing the shutter button.

[0888] Upload the captured image to the system's application using the app's "Upload Image" function.

[0889] Server Processing

[0890] The server receives the uploaded image and immediately saves it in a specified directory.

[0891] The saved image is sent to an external image analysis API to generate an ingredient list, and the JSON response returned from the API is analyzed to extract the ingredient list.

[0892] For example: "Carrots, onions, tomatoes, chicken" would be included in the list.

[0893] The generated ingredient list is sent to a generative AI model to generate recipes that take nutritional balance into account.

[0894] Processing generative AI models

[0895] A generative AI model creates creative cooking recipes based on a provided list of ingredients, including ingredients, steps, cooking times, and nutritional information.

[0896] For example, recipes for "Tomato and Chicken Stew" and "Carrot and Onion Salad" are generated.

[0897] Server information provision

[0898] The server provides the generated recipe to the user, and the user interface displays detailed information about the recipe to the user.

[0899] Users can view the provided recipes and actually cook them.

[0900] Specific examples

[0901] Suppose a user takes a photo of the contents of their refrigerator and uploads the image to the system. The server receives the image and uses an image analysis API to generate a list of ingredients: "carrots, onions, tomatoes, and chicken." The server then sends this list to a generative AI model, which generates recipes for "tomato and chicken stew" and "carrot and onion salad." Finally, the server provides these recipes to the user, who then begins cooking according to the recipes.

[0902] In this way, this system allows users to make effective use of the ingredients in their refrigerator and reduce food waste.

[0903] The processing flow will be explained below.

[0904] Step 1:

[0905] A user takes a picture of the contents of the refrigerator with their smartphone. The user launches the smartphone's camera app, opens the refrigerator door, and presses the shutter button after making sure the food inside is clearly visible.

[0906] Step 2:

[0907] Users can upload images they have taken to the system's application by tapping the "Upload Image" button on the application screen, selecting an image from the album, and sending it.

[0908] Step 3:

[0909] The server receives the uploaded image. The server listens to the HTTP request and saves the received image data in a specified directory.

[0910] Step 4:

[0911] The server sends the received image to an external image analysis API, which then sends the image file to an API endpoint and executes the image analysis request.

[0912] Step 5:

[0913] The server receives a JSON response from the image analysis API. The response contains text data of the analyzed ingredients, such as a list of ingredients such as "carrot, onion, tomato, and chicken."

[0914] Step 6:

[0915] The server retrieves the generated ingredient list and sends it to the generative AI model. The server then sends a request including the ingredient list to the API endpoint of the generative AI model.

[0916] Step 7:

[0917] A generative AI model generates a cooking recipe based on an ingredient list. The model combines ingredients and generates multiple recipe ideas, taking into account nutritional balance and cooking procedures. For example, it generates recipes such as "Tomato and Chicken Stew" and "Carrot and Onion Salad."

[0918] Step 8:

[0919] The server receives the recipe returned from the generative AI model. The server receives the response from the recipe generation API and analyzes the response data.

[0920] Step 9:

[0921] The server displays the generated recipe to the user, converts the received recipe data into a format for display on the user interface, and displays detailed recipe information on the user's smartphone screen.

[0922] Step 10:

[0923] Users can select the dish they want to make from the displayed recipes by tapping on their smartphone screen.

[0924] Step 11:

[0925] The app will then create a recipe based on the user's chosen recipe, and the user will follow the instructions displayed on the app to select the ingredients they need and begin cooking.

[0926] Example 1

[0927] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0928] In modern society, there is a problem that purchased ingredients are often forgotten in the refrigerator and wasted. Furthermore, many users find it difficult to easily plan and execute nutritionally balanced meals at home amid busy lifestyles. This has led to increased food waste and health concerns. The purpose of this invention is to solve these problems by providing a system that effectively utilizes ingredients in the refrigerator and efficiently provides nutritionally balanced meals.

[0929] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0930] In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for saving the received images, means for calling an external image analysis API to analyze the saved images and generate an ingredient list, means for using a generative AI model to generate cooking recipes that take nutritional balance into consideration based on the ingredient list obtained through the analysis, and means for providing the generated cooking recipes through a user interface, thereby enabling users to efficiently plan and execute nutritionally balanced meals without wasting ingredients in the refrigerator.

[0931] "User" refers to an individual who uses the system to take pictures of the inside of a refrigerator and upload them to the system.

[0932] "Server" refers to the central computing device that receives and processes images uploaded by users.

[0933] "Image receiving means" refers to a function or device that allows the server to receive images taken by the user.

[0934] "Storage means" refers to a function or device for storing received images in a predetermined location.

[0935] "Image analysis API" refers to an external application program interface that analyzes received images and recognizes and lists ingredients within the images.

[0936] "Ingredient list" refers to a list of ingredients in an image obtained as a result of analysis by the image analysis API.

[0937] "Generative AI model" refers to an artificial intelligence model used to generate creative cooking recipes based on an ingredient list.

[0938] "Recipe generation means" refers to a function or device for generating a cooking recipe that takes nutritional balance into consideration based on an ingredient list.

[0939] "User Interface" refers to an interactive screen or application that provides the generated cooking recipe to the user.

[0940] This invention relates to a system that uses image analysis technology to automatically generate an ingredient list by taking a picture of the contents of a refrigerator and providing the system with original cooking recipes based on that list. This system provides a means to efficiently use ingredients in the refrigerator without wasting them.

[0941] Key Components of the System

[0942] 1. Image receiving means

[0943] It provides an interface for users to upload images of the inside of their refrigerator.

[0944] The server stores the images received from the user and prepares them for analysis.

[0945] 2. Image analysis methods

[0946] The server analyzes the received image and creates a list of ingredients. Here, an external image analysis API is used to recognize ingredients in the image and convert them into text data.

[0947] 3. Recipe Generation Method

[0948] The server uses a generative AI model based on the ingredient list to generate a cooking recipe that takes nutritional balance into consideration.

[0949] The generated recipe includes detailed information such as ingredients, steps, and cooking times.

[0950] 4. Means of providing information

[0951] The server has an interface for providing the generated recipe information to the user.

[0952] Users can receive recipe information through devices such as smartphones and PCs.

[0953] Specific processing content of the program

[0954] 1. User Actions

[0955] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door so that the food inside is clearly visible, and then pressing the shutter button.

[0956] Upload the captured image to the system's application using the app's "Upload Image" function.

[0957] 2. Server Processing

[0958] The server receives the uploaded image and saves it in a specified directory.

[0959] The saved image is sent to an external image analysis API to generate an ingredient list, and the JSON response returned from the API is analyzed to extract the ingredient list.

[0960] For example: "Carrots, onions, tomatoes, chicken" would be included in the list.

[0961] The generated ingredient list is sent to a generative AI model to generate recipes that take nutritional balance into account.

[0962] 3. Processing of generative AI models

[0963] A generative AI model creates creative cooking recipes based on a provided list of ingredients, including ingredients, steps, cooking times, and nutritional information.

[0964] For example, recipes for "Tomato and Chicken Stew" and "Carrot and Onion Salad" are generated.

[0965] 4. Server Information

[0966] The server provides the generated recipe to the user, and the user interface displays detailed information about the recipe to the user.

[0967] Users can view the provided recipes and actually cook them.

[0968] Specific examples

[0969] Suppose a user takes a photo of the contents of their refrigerator and uploads the image to the system. The server receives the image and uses an image analysis API to generate a list of ingredients: "carrots, onions, tomatoes, and chicken." The server then sends this list to a generative AI model, which generates recipes for "tomato and chicken stew" and "carrot and onion salad." Finally, the server provides these recipes to the user, who then begins cooking according to the recipes.

[0970] Prompt Sentence Examples

[0971] An example of a prompt to send to the image analysis API is:

[0972] {

[0973] "action": "parse_image",

[0974] "image_url": "YOUR_IMAGE_URL",

[0975] "response_format": "json"

[0976] }

[0977] Here's an example prompt to send to a generative AI model:

[0978] Recipe category to create: Cooking

[0979] Ingredients: carrots, onions, tomatoes, chicken

[0980] Generate two recipes, taking nutritional balance into consideration.

[0981] In this way, the present invention allows users to make better use of food in their refrigerators and reduce food waste.

[0982] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0983] Step 1:

[0984] User image capture and upload

[0985] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door and pressing the shutter button so that the food inside is clearly visible.

[0986] The captured image is uploaded to the system's application. The user opens the app, taps the "Upload Image" button, selects an image, and presses the "Send" button.

[0987] Input: Image of the contents of the refrigerator taken with a smartphone

[0988] Output: Images uploaded to the system

[0989] Step 2:

[0990] Receiving and saving images on the server

[0991] The server receives the uploaded image. The image data is sent to the server via an HTTP request.

[0992] Save the received images in a specified directory (e.g., / images / uploads).

[0993] Log the save directory path to ensure the image has been saved successfully.

[0994] Input: Image data uploaded by the user

[0995] Output: Images saved in the specified directory on the server

[0996] Step 3:

[0997] Calling the server's image analysis API

[0998] The server sends the stored image to an external image analysis API using an HTTP POST request, sending the image's binary data and URL to the API.

[0999] The image analysis API analyzes the image and returns a list of ingredients in JSON format.

[1000] The server parses the JSON response and extracts the list of ingredients in the image.

[1001] Input: Path to the image stored on the server

[1002] Output: Generated list of ingredients (e.g. carrot, onion, tomato, chicken)

[1003] Step 4:

[1004] Server recipe generation

[1005] Based on the ingredient list, the server sends a prompt to the generative AI model.

[1006] Recipe category to create: Cooking

[1007] Ingredients: carrots, onions, tomatoes, chicken

[1008] Generate two recipes, taking nutritional balance into consideration.

[1009] The generative AI model generates a recipe and returns it to the server, including ingredients, steps, cooking time, and nutritional information.

[1010] Input: Generated ingredient list and prompt statement

[1011] Output: Generated cooking recipe

[1012] Step 5:

[1013] Server recipe provision

[1014] The server displays the generated recipe in the user interface, allowing the user to view detailed recipe information on the app's "View Recipe" screen.

[1015] It allows users to browse recipes and provides instructions for actually cooking the food.

[1016] Input: Generated cooking recipe

[1017] Output: Recipe information displayed in the user interface

[1018] Through these steps, the system enables users to effectively utilize the ingredients in their refrigerator and provide efficient, nutritionally balanced meals.

[1019] (Application example 1)

[1020] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1021] Modern households are expected to efficiently utilize the ingredients in their refrigerators, but this often fails. Food waste is a problem due to ineffective use of ingredients. Many people are looking for new recipes and cooking videos when it comes to home cooking, but finding the right content can be a hassle. Furthermore, there is a lack of real-time interactive tools to improve cooking skills at home. Providing an effective system to solve these problems is a challenge.

[1022] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1023] In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for analyzing the received images to generate an ingredient list, means for generating cooking videos and articles related to cooking recipes based on the generated ingredient list, means for providing the user with cooking videos and articles related to the generated cooking recipes, and means for providing a virtual kitchen environment in which the user can perform cooking simulations in real time. This not only enables efficient use of ingredients in the refrigerator, but also makes it possible to provide the user with new recipes and cooking videos to help improve their cooking skills at home.

[1024] A "user" is someone who uses this system to manage ingredients in their refrigerator and obtain cooking recipes and related content.

[1025] "Photographing" refers to the act of taking a photo of the food in the refrigerator using a smartphone or other image capture device.

[1026] "Images inside the refrigerator" are photographic data of the inside of the refrigerator taken by the user, showing the state of the ingredients.

[1027] "Analysis" is the process by which the received image is processed by software to identify and list elements within the image.

[1028] The "ingredient list" is a list of names and information of ingredients extracted from the analyzed images of the inside of the refrigerator.

[1029] A "cooking recipe" is a document that contains specific cooking steps and a list of necessary ingredients, generated based on an ingredient list.

[1030] "Cooking videos" are video content that visually show cooking procedures and can be viewed by users.

[1031] An "article" is a text-based explanation related to cooking methods and ingredients.

[1032] A "virtual kitchen environment" is a virtual space that simulates a real-world kitchen and allows users to interactively experience cooking procedures.

[1033] "Real-time cooking simulation" refers to a user experiencing a simulation in which they react instantly within a virtual kitchen environment.

[1034] This invention is a system that efficiently utilizes ingredients in a refrigerator and provides users with cooking recipes and related content. This system is implemented mainly using a smartphone and a server.

[1035] 1. User Action

[1036] The user takes a picture of the food in the refrigerator using their smartphone. After taking the picture, the user uploads the picture to the server using the system's dedicated application. At this stage, the "Image Upload" function is used.

[1037] 2. Server processing

[1038] The server saves the received images in a specified directory. Then, it sends the saved images to an external image analysis API (e.g., Google Cloud Vision API) to generate an ingredient list. It parses the JSON response returned by the API and extracts the ingredient list. Specifically, the following prompt sentence is used:

[1039] Example prompt:

[1040] "Generate a list of ingredients in an image. Analyze the image and list the names of the ingredients."

[1041] 3. Recipe and content generation

[1042] The server generates a recipe based on the generated ingredient list using a generative AI model (e.g., OpenAI's GPT-4), taking nutritional balance into consideration. The generated recipe includes detailed information such as ingredients, steps, and cooking time. Specifically, the following prompt is used:

[1043] Example prompt:

[1044] "Create a delicious recipe using the following ingredients: chicken, onion, carrot, and tomato."

[1045] Additionally, to provide you with related cooking videos and articles, we use cooking video services (e.g., YouTube API, Vimeo API) to search for and provide related content to you.

[1046] 4. Virtual Kitchen Environment

[1047] The server provides a 3D virtual environment using platforms such as Unreal Engine and Unity, allowing users to simulate real-time cooking in a virtual kitchen environment, allowing them to improve their cooking skills by recreating real-life cooking procedures in a virtual environment.

[1048] Specific examples

[1049] A user takes a photo of the contents of their refrigerator with their smartphone and uploads the image to the system's application. The server receives the image and uses the Google Cloud Vision API to generate an ingredient list: "carrots, onions, tomatoes, and chicken." Next, GPT-4 is used to generate recipes: "Tomato and Chicken Stew" and "Carrot and Onion Salad." Additionally, cooking videos for these recipes are provided using the YouTube API. Users can browse these recipes and videos and experience cooking simulations in a virtual kitchen environment using an HMD (e.g., Oculus Rift).

[1050] This system allows users to efficiently use ingredients in their refrigerators and easily acquire and try out new recipes and cooking methods, which is expected to not only reduce food waste but also improve the cooking experience at home.

[1051] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1052] Step 1:

[1053] The user takes a picture of the inside of the refrigerator with their smartphone and uploads the picture to the system's application. The input is the image taken by the user, and the output is the upload of the image data to the server. Specifically, the user takes a picture of the food in the refrigerator using the camera app on their smartphone, and then uses the "image upload" function in the app to send the picture to the server.

[1054] Step 2:

[1055] The server saves the images it receives in a specified directory. The input is the uploaded image data, and the output is an image file saved on the server. Specifically, the server receives an HTTP request and saves the sent image data in a specific folder.

[1056] Step 3:

[1057] The server sends the saved image to an external image analysis API to generate an ingredient list. The input is the saved image file, and the output is the ingredient list. Specifically, the server sends the image data to an analysis service such as Google Cloud Vision API, analyzes the JSON response returned from the API, and lists the ingredients in the image.

[1058] Step 4:

[1059] The server generates a cooking recipe using a generative AI model based on the generated ingredient list. The input is the ingredient list, and the output is the generated cooking recipe. Specifically, the server creates a prompt sentence like the one below based on the ingredient list and sends it to a generative AI model such as OpenAI's GPT-4:

[1060] Example prompt:

[1061] "Create a delicious recipe using the following ingredients: chicken, onion, carrot, and tomato."

[1062] The generative AI model generates a recipe based on these prompts, and the server receives the results.

[1063] Step 5:

[1064] The server generates related cooking videos and articles. The input is a cooking recipe, and the output is related videos and articles. Specifically, the server searches for cooking videos related to the recipe using the YouTube API or Vimeo API, and provides the retrieved video links and articles to the user.

[1065] Step 6:

[1066] The server provides the generated recipe and related cooking videos and articles to the user. The input is the generated recipe and related content, and the output is the display on the user's device. Specifically, the server sends data in HTML or JSON format so that it can be displayed on the user's smartphone or PC browser.

[1067] Step 7:

[1068] It provides a virtual kitchen environment where users can perform cooking simulations in real time. The input is a cooking recipe and virtual kitchen environment data, and the output is a virtual experience on the user's HMD. Specifically, the server uses Unreal Engine or Unity to build the virtual kitchen, and the user wears the HMD to perform the cooking simulation in real time.

[1069] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1070] This invention combines a system that uses image analysis technology to automatically generate an ingredient list based on a user's image of the contents of their refrigerator, and then provides original cooking recipes based on that list with an emotion engine that recognizes the user's emotions, thereby enabling more personalized recipe suggestions. This system makes it possible to efficiently use ingredients in the refrigerator according to the user's condition, without wasting any ingredients.

[1071] System Overview

[1072] The system consists of the following main components:

[1073] 1. Image receiving means

[1074] It provides an interface for users to upload images of the inside of their refrigerator.

[1075] The server stores the images received from the user and prepares them for analysis.

[1076] 2. Image analysis methods

[1077] The server analyzes the received image and creates a list of ingredients. Here, an external image analysis API is used to recognize ingredients in the image and convert them into text data.

[1078] 3. Recipe Generation Method

[1079] The server uses a generative AI model based on the ingredient list to generate a cooking recipe that takes nutritional balance into consideration.

[1080] The generated recipe includes detailed information such as ingredients, steps, and cooking times.

[1081] 4. Emotion Engine

[1082] An engine that recognizes and analyzes user emotions.

[1083] For example, the user's facial expressions and voice can be analyzed to identify their emotional state (e.g., joy, sadness, stress).

[1084] 5. Means of providing information

[1085] The server has an interface for providing the generated recipe information to the user.

[1086] Users can receive recipe information through devices such as smartphones and PCs.

[1087] Specific processing content of the program

[1088] User Actions

[1089] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door so that the food inside is clearly visible, and then pressing the shutter button.

[1090] Upload the captured image to the system's application using the app's "Upload Image" function.

[1091] Server Processing

[1092] The server receives the uploaded image and immediately saves it in a specified directory.

[1093] The saved image is sent to an external image analysis API to generate an ingredient list, and the JSON response returned from the API is analyzed to extract the ingredient list.

[1094] For example: "Carrots, onions, tomatoes, chicken" would be included in the list.

[1095] The generated ingredient list is sent to a generative AI model to generate recipes that take nutritional balance into account.

[1096] Emotion engine processing

[1097] The emotion engine recognizes the user's emotional state and uses facial recognition cameras and voice analysis to identify emotions when a user accesses the app.

[1098] For example, it can detect "joy" from a user's facial expression.

[1099] The server uses the emotional data obtained from the emotion engine and sends it to the generative AI model as additional information, which generates a recipe that is optimal for the user's current emotional state.

[1100] Processing generative AI models

[1101] A generative AI model creates creative cooking recipes based on a provided list of ingredients and user sentiment data, including ingredients, steps, cooking times, and nutritional information.

[1102] For example, recipes such as "Tomato and chicken stew" and "Carrot and onion salad" are generated, but the more suitable recipe is selected based on the user's emotions.

[1103] Server information provision

[1104] The server provides the generated recipe to the user, and the user interface displays detailed information about the recipe to the user.

[1105] Users can view the provided recipes and actually cook them.

[1106] Specific examples

[1107] Suppose a user takes a photo of the contents of their refrigerator and uploads the image to the system. The server receives this image and uses an image analysis API to generate an ingredient list such as "carrots, onions, tomatoes, and chicken." The server then uses an emotion engine to recognize the user's emotions before sending the ingredient list to the generative AI model. For example, if the emotion engine detects a "stressed state" from the user's facial expression, the generative AI model will prioritize generating recipes that have a relaxing effect on the user. In this case, a recipe such as "soup to make you feel calm" would be selected. Finally, the server provides this recipe to the user, who then begins cooking according to the recipe.

[1108] In this way, this system not only helps users make effective use of the ingredients in their refrigerator, but also provides a more personalized dining experience by suggesting recipes that match the user's emotional state.

[1109] The processing flow will be explained below.

[1110] Step 1:

[1111] A user takes a picture of the contents of the refrigerator with their smartphone. The user launches the smartphone's camera app, opens the refrigerator door, and presses the shutter button after making sure the food inside is clearly visible.

[1112] Step 2:

[1113] Users can upload images they have taken to the system's application by tapping the "Upload Image" button on the application screen, selecting an image from the album, and sending it.

[1114] Step 3:

[1115] The server receives the uploaded image. The server listens to the HTTP request and saves the received image data in a specified directory.

[1116] Step 4:

[1117] The server sends the received image to an external image analysis API, which then sends the image file to an API endpoint and executes the image analysis request.

[1118] Step 5:

[1119] The server receives a JSON response from the image analysis API. The response contains text data of the analyzed ingredients, such as a list of ingredients such as "carrot, onion, tomato, and chicken."

[1120] Step 6:

[1121] The server retrieves the generated ingredient list and sends it to the generative AI model. The server then sends a request including the ingredient list to the API endpoint of the generative AI model.

[1122] Step 7:

[1123] The device recognizes the user's emotional state. The device uses a camera and microphone to analyze the user's facial expressions and voice, and the emotion engine analyzes that data to identify the user's emotional state.

[1124] Step 8:

[1125] The device transmits the emotion data obtained from the emotion engine to the server, which uses this data as information for creating recipes.

[1126] Step 9:

[1127] A generative AI model generates a cooking recipe based on the ingredient list and emotional data. The generative AI model runs an algorithm to generate a cooking recipe that best suits the user's emotional state. For example, if the user is in a "stressed state," the model will prioritize suggestions that have a relaxing effect.

[1128] Step 10:

[1129] The server provides the generated recipe to the user, converts the received recipe data into a format for display on the user interface, and displays detailed recipe information on the user's smartphone screen.

[1130] Step 11:

[1131] Users can select the dish they want to make from the displayed recipes by tapping on their smartphone screen.

[1132] Step 12:

[1133] The app will then create a recipe based on the user's chosen recipe, and the user will follow the instructions displayed on the app to select the ingredients they need and begin cooking.

[1134] Example 2

[1135] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1136] Currently, there are systems that efficiently suggest cooking recipes using ingredients in the refrigerator, but they do not suggest personalized recipes based on the user's emotional state. As a result, there is a problem in that they cannot provide recipes that match the user's emotional state and preferences.

[1137] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1138] In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for sending the received images to an external image analysis API to generate an ingredient list, means for generating a cooking recipe using a generative AI model based on the ingredient list, means for recognizing the user's emotional state and selecting a personalized cooking recipe based on that, and means for providing the generated cooking recipe to the user, thereby enabling efficient and personalized cooking recipe suggestions tailored to the user's emotional state.

[1139] "User" refers to an individual who uses this system to take and upload images of the inside of a refrigerator.

[1140] "Means for receiving" refers to a device or program that has the function of transmitting images taken by a user to a server and receiving the image data.

[1141] "Image analysis API" refers to a program interface provided by an external service provider for extracting specific information from image data.

[1142] "Ingredient list" refers to text data in list format that shows the names and quantities of ingredients in the refrigerator, extracted using an image analysis API.

[1143] A "generative AI model" refers to an artificial intelligence algorithm that generates new information or content based on provided data.

[1144] A "cooking recipe" refers to detailed information that shows how to make a dish and the steps involved, generated based on a list of ingredients.

[1145] "Emotional state" refers to data that indicates a user's current emotional or psychological state.

[1146] "Personalized cooking recipes" refer to cooking recipes that are optimized based on the user's emotional state and individual preferences.

[1147] "Means for providing" refers to an interface or device for showing the generated cooking recipe to the user.

[1148] This invention combines a system that uses image analysis technology to automatically generate an ingredient list based on a user's image of the contents of their refrigerator, and then provides original cooking recipes based on that list with an emotion engine that recognizes the user's emotions, thereby enabling more personalized recipe suggestions. This system makes it possible to efficiently use ingredients in the refrigerator according to the user's condition, without wasting any ingredients.

[1149] The main components of the system include:

[1150] 1. Image receiving means

[1151] Provide an interface for users to upload images of the inside of their refrigerator. For example, users can take a picture of the contents of their refrigerator with their smartphone and use the "upload image" function within the app to send the image to the server.

[1152] 2. Image analysis methods

[1153] The server analyzes the received image and generates an ingredient list. This analysis uses an external image analysis API to recognize ingredients in the image and convert them into text data. For example, an ingredient list such as "carrot, onion, tomato, chicken" is extracted.

[1154] 3. Recipe Generation Method

[1155] The server generates a nutritionally balanced cooking recipe based on the generated ingredient list, using a generative AI model to create a recipe that includes detailed information such as ingredients, steps, and cooking time.

[1156] 4. Emotion recognition means

[1157] The emotion engine recognizes the user's emotional state. Specifically, when a user accesses the app, it uses a facial recognition camera and voice analysis to identify emotions. For example, it can detect "joy" or "stress" from the user's facial expressions.

[1158] 5. Means of providing information

[1159] The server has an interface to provide the generated recipes to users, who can receive the recipe information via their smartphones, PCs, or other devices.

[1160] Here is an example of this system:

[1161] 1. The user takes a picture of the contents of their refrigerator and uploads the image to the system using the "upload image" function within the app.

[1162] 2. The server receives this image and sends the saved image to an external image analysis API, which generates a list of ingredients, for example, "carrot, onion, tomato, chicken."

[1163] 3. The server sends the ingredient list to the generative AI model to generate a cooking recipe.

[1164] 4. The emotion engine recognizes the user’s emotional state. For example, if it detects a “stressed state,” the generative AI model will prioritize recipes that will have a relaxing effect on the user. In this case, a recipe such as “soup to make you feel calm” will be selected.

[1165] 5. Finally, the server provides this recipe to the user, who then starts cooking according to the recipe.

[1166] Prompt Sentence Examples

[1167] "Please upload a picture of the ingredients in your refrigerator."

[1168] "Generating the best recipe for your current emotional state. Please wait a moment."

[1169] "The following recipe was generated: 'Comforting Soup'."

[1170] Check out this recipe and start cooking.

[1171] In this way, this system not only allows users to make effective use of the ingredients in their refrigerator, but also provides a more personalized dining experience by suggesting recipes that match the user's emotional state.

[1172] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1173] Step 1:

[1174] The user takes a picture of the inside of the refrigerator with their smartphone. The user opens the refrigerator door to clearly see the food inside and presses the shutter button. The image is then saved on the smartphone.

[1175] Input: Actual ingredients in the fridge

[1176] Output: Food images stored on the smartphone

[1177] Step 2:

[1178] The user opens the system application and uploads the captured image. Using the "Upload Image" function in the application, the user selects the saved image file and sends it to the server.

[1179] Input: Food images stored on your smartphone

[1180] Output: Food images uploaded to the server

[1181] Step 3:

[1182] The server receives the uploaded image and immediately saves it in a specified directory.

[1183] Input: User-submitted food images

[1184] Output: Food images saved in a specified directory on the server

[1185] Step 4:

[1186] The server sends the stored image to an external image analysis API and requests analysis. It sends the image data using an HTTP request and receives a JSON-formatted response from the API.

[1187] Input: Food images stored on the server

[1188] Output: A list of ingredients in JSON format returned by the API

[1189] Step 5:

[1190] The server parses the JSON response returned from the API and extracts the list of ingredients. Specifically, it extracts the ingredient names from the JSON data and converts them into a text list.

[1191] Input: JSON formatted list of ingredients returned from the API

[1192] Output: Ingredient list in text format

[1193] Step 6:

[1194] The server sends the generated ingredient list to the generative AI model and requests it to generate a recipe. The server then sends the ingredient list to the generative AI model as input data and waits for the recipe to be generated.

[1195] Input: Text list of ingredients

[1196] Output: Input data sent to the generative AI model

[1197] Step 7:

[1198] The server activates the emotion engine to recognize the user's emotional state. When a user accesses the app, the camera and microphone are used to capture facial expressions and voice, and emotion recognition algorithms are applied.

[1199] Input: Facial expressions and voice data captured through a camera and microphone

[1200] Output: Recognized emotional state of the user

[1201] Step 8:

[1202] The server sends the emotion data obtained from the emotion engine to the generative AI model, which then uses the emotion data as additional information to generate recipes.

[1203] Input: Perceived emotional state of the user

[1204] Output: Emotion data sent to the generative AI model

[1205] Step 9:

[1206] The generative AI model creates creative cooking recipes based on the provided ingredient list and emotional data. Specifically, the algorithm inputs the ingredient list and emotional data to generate a recipe that includes ingredients, steps, cooking time, and nutritional information.

[1207] Input: Text list of ingredients and emotion data

[1208] Output: Generated cooking recipe

[1209] Step 10:

[1210] The server provides the generated recipe to the user, and the recipe details are displayed through an app or web interface.

[1211] Input: Generated cooking recipe

[1212] Output: The cooking recipe displayed to the user

[1213] Step 11:

[1214] Users actually cook according to the provided recipe, preparing ingredients and following the steps on their smartphone or PC screen as a reference.

[1215] Input: A cooking recipe shown to the user

[1216] Output: The actual cooked food

[1217] (Application example 2)

[1218] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1219] Conventional systems that analyze images of the inside of a refrigerator to generate an ingredient list and provide cooking recipes suggest uniform recipes regardless of the user's state, which means they have the problem of not being able to suggest appropriate recipes that reflect the user's current emotions or preferences.In particular, when users shop in physical stores, there are no product suggestions or guidance that reflect their emotional state, making it difficult to provide an efficient and satisfying shopping experience.

[1220] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for analyzing the received images and generating an ingredient list, means for generating a cooking recipe based on the generated ingredient list, means for recognizing the user's emotions, means for generating a personalized cooking recipe based on the user's emotion data, and means for providing the generated cooking recipe to the user. This makes it possible to suggest ingredients and recipes that are optimal for the user's emotional state.

[1221] The "means for receiving images of the inside of the refrigerator taken by the user" is an interface for sending images of the inside of the refrigerator taken by the user using a smart device or the like to a server or cloud service.

[1222] The "means for analyzing received images and generating an ingredient list" is a process for identifying ingredients present in the refrigerator from the received image data using image analysis technology and creating a list of them.

[1223] The "means for generating a cooking recipe based on the generated ingredient list" refers to an algorithm or system for automatically generating a cooking recipe based on the ingredient information obtained by analysis.

[1224] "Means for recognizing user emotions" refers to hardware and software that analyzes the user's emotional state from their facial expressions and voice, and identifies emotions such as joy, sadness, and stress.

[1225] The "means for generating personalized cooking recipes based on user emotional data" is a process for generating cooking recipes that are optimally adjusted for each individual user based on the recognized emotional state of the user.

[1226] The "means for providing the generated cooking recipe to the user" refers to a communication interface or application for displaying or notifying the cooking recipe information generated by the server on the user's device.

[1227] The present invention is a system that allows a user to take a picture of the inside of a refrigerator, analyzes the picture to generate an ingredient list, and recognizes the user's emotional state to provide personalized cooking recipes. The system consists of the following main components:

[1228] 1. User Interface:

[1229] Users use devices such as smartphones or tablets to take pictures of the inside of the refrigerator and upload them to the application, which then sends the uploaded images to the system's server.

[1230] 2. Image analysis system:

[1231] The server analyzes the received image of the refrigerator interior using image recognition technology to generate an ingredient list. Specifically, it uses an external image analysis API to identify ingredients in the image and create a text list.

[1232] 3. Emotion Recognition System:

[1233] For emotion recognition, users input facial expressions and voice using their smart devices. The server analyzes these input data and uses an emotion recognition engine to identify the user's emotional state (e.g., joy, sadness, stress).

[1234] 4. Recipe Generation System:

[1235] The server uses a generative AI model to generate cooking recipes based on the generated ingredient list and emotional data. The cooking recipes are personalized, taking into account nutritional balance and the user's emotional state.

[1236] 5. Information provision system:

[1237] Finally, the server provides the generated cooking recipe to the user's device, where the user can access, view, and actually cook the recipe.

[1238] Hardware and software used

[1239] Smart devices (smartphones, tablets, etc.): Used by users to take pictures of the inside of the refrigerator and upload them to the application.

[1240] Server: Integrates and manages the image analysis system, emotion recognition system, recipe generation system, and information provision system.

[1241] Image analysis API: Uses an external API to analyze images of the inside of the refrigerator and identify ingredients.

[1242] Emotion recognition engine: An engine for identifying the user's emotional state from their facial expressions and voice.

[1243] Generative AI model: Generates personalized cooking recipes based on ingredient lists and emotional data.

[1244] Specific examples

[1245] For example, a user can use their smartphone to take a picture of the inside of their refrigerator and upload it to the system. The server receives this image and uses an image analysis API to generate an ingredient list such as "carrots, onions, tomatoes, and chicken." The server then uses the smart device's camera and microphone to recognize the user's emotional state and identify a "stressed state." Based on this information, the server uses a generative AI model to generate a recipe for a "calming soup" that will help the user relax and provides it to the user's smartphone. The user can then cook according to this recipe.

[1246] Prompt Sentence Examples

[1247] "When a user visits a physical store and starts shopping using the smart glasses, the system captures the user's face and recognizes their emotions. Depending on their emotions, the system will suggest the following product list:

[1248] 1. Happiness → Healthy foods, desserts

[1249] 2. Sadness → Chocolate, comfort food

[1250] 3. Stress → Herbal tea, relaxation items

[1251] This system allows users to effectively utilize ingredients in their refrigerators, receive personalized recipes based on their emotional state, and improve their shopping experience in brick-and-mortar stores.

[1252] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1253] Step 1:

[1254] A user takes a picture of the inside of the refrigerator using a smartphone or tablet and uploads the picture to the system through a dedicated application. The input is the image of the inside of the refrigerator, and the output is an image file sent to the server.

[1255] Step 2:

[1256] The server saves the image received from the user in a predetermined save directory. In this step, the image file sent from the user is received as input, and the image file is saved in the save directory as output.

[1257] Step 3:

[1258] The server sends the saved image to an external image analysis API and obtains the ingredient list as the analysis result. Here, the saved image file is used as input and the ingredient list in JSON format is obtained as output. Specifically, the ingredients in the image are identified and each is listed as text data.

[1259] Step 4:

[1260] The server analyzes the acquired ingredient list and sends it to the generative AI model. Here, the ingredient list is used as input, and data is prepared to be sent to the generative AI model as output.

[1261] Step 5:

[1262] The user inputs facial expressions and voice data into the system using the camera and microphone of the smart device. The input is the user's facial expressions and voice data, and the output is an audio file or image data sent to the server.

[1263] Step 6:

[1264] The server sends facial expression and voice data to an emotion recognition engine to identify the user's emotional state. Here, audio files and image data are used as input, and emotion recognition results are obtained as output. Specifically, emotional states such as "joy," "sadness," and "stress" are identified.

[1265] Step 7:

[1266] The server combines the emotion recognition results and the ingredient list and sends them to the generative AI model. Here, the emotional state data and ingredient list are used as input, and the data sent to the generative AI model as output is prepared.

[1267] Step 8:

[1268] A generative AI model generates personalized cooking recipes based on ingredient lists and emotional data. Here, the input is an ingredient list and emotional state data, and the output is a personalized cooking recipe, taking into account the ingredients, steps, time, and nutritional balance of the recipe.

[1269] Step 9:

[1270] The server provides the generated personalized cooking recipe to the user's device, and uses the generated cooking recipe as input to prepare recipe information to be displayed on the user's smartphone or tablet as output.

[1271] Step 10:

[1272] The user accesses the provided cooking recipe, browses it, and actually cooks the food. Here, the provided cooking recipe is used as input, and the completed dish is obtained as output. In concrete terms, the user cooks the food according to the recipe steps.

[1273] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1274] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1275] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1276] [Fourth embodiment]

[1277] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1278] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1279] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1280] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1281] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1282] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1283] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1284] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1285] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1286] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1287] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1288] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1289] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1290] The present invention relates to a system that uses image analysis technology to automatically generate an ingredient list by taking a picture of the contents of a refrigerator and providing the system with original cooking recipes based on that list. This system provides a means to efficiently use ingredients in the refrigerator without wasting them.

[1291] System Overview

[1292] The system consists of the following main components:

[1293] 1. Image receiving means

[1294] It provides an interface for users to upload images of the inside of their refrigerator.

[1295] The server stores the images received from the user and prepares them for analysis.

[1296] 2. Image analysis methods

[1297] The server analyzes the received image and creates a list of ingredients. Here, an external image analysis API is used to recognize ingredients in the image and convert them into text data.

[1298] 3. Recipe Generation Method

[1299] The server uses a generative AI model based on the ingredient list to generate a cooking recipe that takes nutritional balance into consideration.

[1300] The generated recipe includes detailed information such as ingredients, steps, and cooking times.

[1301] 4. Means of providing information

[1302] The server has an interface for providing the generated recipe information to the user.

[1303] Users can receive recipe information through devices such as smartphones and PCs.

[1304] Specific processing content of the program

[1305] User Actions

[1306] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door so that the food inside is clearly visible, and then pressing the shutter button.

[1307] Upload the captured image to the system's application using the app's "Upload Image" function.

[1308] Server Processing

[1309] The server receives the uploaded image and immediately saves it in a specified directory.

[1310] The saved image is sent to an external image analysis API to generate an ingredient list, and the JSON response returned from the API is analyzed to extract the ingredient list.

[1311] For example: "Carrots, onions, tomatoes, chicken" would be included in the list.

[1312] The generated ingredient list is sent to a generative AI model to generate recipes that take nutritional balance into account.

[1313] Processing generative AI models

[1314] A generative AI model creates creative cooking recipes based on a provided list of ingredients, including ingredients, steps, cooking times, and nutritional information.

[1315] For example, recipes for "Tomato and Chicken Stew" and "Carrot and Onion Salad" are generated.

[1316] Server information provision

[1317] The server provides the generated recipe to the user, and the user interface displays detailed information about the recipe to the user.

[1318] Users can view the provided recipes and actually cook them.

[1319] Specific examples

[1320] Suppose a user takes a photo of the contents of their refrigerator and uploads the image to the system. The server receives the image and uses an image analysis API to generate a list of ingredients: "carrots, onions, tomatoes, and chicken." The server then sends this list to a generative AI model, which generates recipes for "tomato and chicken stew" and "carrot and onion salad." Finally, the server provides these recipes to the user, who then begins cooking according to the recipes.

[1321] In this way, this system allows users to make effective use of the ingredients in their refrigerator and reduce food waste.

[1322] The processing flow will be explained below.

[1323] Step 1:

[1324] A user takes a picture of the contents of the refrigerator with their smartphone. The user launches the smartphone's camera app, opens the refrigerator door, and presses the shutter button after making sure the food inside is clearly visible.

[1325] Step 2:

[1326] Users can upload images they have taken to the system's application by tapping the "Upload Image" button on the application screen, selecting an image from the album, and sending it.

[1327] Step 3:

[1328] The server receives the uploaded image. The server listens to the HTTP request and saves the received image data in a specified directory.

[1329] Step 4:

[1330] The server sends the received image to an external image analysis API, which then sends the image file to an API endpoint and executes the image analysis request.

[1331] Step 5:

[1332] The server receives a JSON response from the image analysis API. The response contains text data of the analyzed ingredients, such as a list of ingredients such as "carrot, onion, tomato, and chicken."

[1333] Step 6:

[1334] The server retrieves the generated ingredient list and sends it to the generative AI model. The server then sends a request including the ingredient list to the API endpoint of the generative AI model.

[1335] Step 7:

[1336] A generative AI model generates a cooking recipe based on an ingredient list. The model combines ingredients and generates multiple recipe ideas, taking into account nutritional balance and cooking procedures. For example, it generates recipes such as "Tomato and Chicken Stew" and "Carrot and Onion Salad."

[1337] Step 8:

[1338] The server receives the recipe returned from the generative AI model. The server receives the response from the recipe generation API and analyzes the response data.

[1339] Step 9:

[1340] The server displays the generated recipe to the user, converts the received recipe data into a format for display on the user interface, and displays detailed recipe information on the user's smartphone screen.

[1341] Step 10:

[1342] Users can select the dish they want to make from the displayed recipes by tapping on their smartphone screen.

[1343] Step 11:

[1344] The app will then create a recipe based on the user's chosen recipe, and the user will follow the instructions displayed on the app to select the ingredients they need and begin cooking.

[1345] Example 1

[1346] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1347] In modern society, there is a problem that purchased ingredients are often forgotten in the refrigerator and wasted. Furthermore, many users find it difficult to easily plan and execute nutritionally balanced meals at home amid busy lifestyles. This has led to increased food waste and health concerns. The purpose of this invention is to solve these problems by providing a system that effectively utilizes ingredients in the refrigerator and efficiently provides nutritionally balanced meals.

[1348] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1349] In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for saving the received images, means for calling an external image analysis API to analyze the saved images and generate an ingredient list, means for using a generative AI model to generate cooking recipes that take nutritional balance into consideration based on the ingredient list obtained through the analysis, and means for providing the generated cooking recipes through a user interface, thereby enabling users to efficiently plan and execute nutritionally balanced meals without wasting ingredients in the refrigerator.

[1350] "User" refers to an individual who uses the system to take pictures of the inside of a refrigerator and upload them to the system.

[1351] "Server" refers to the central computing device that receives and processes images uploaded by users.

[1352] "Image receiving means" refers to a function or device that allows the server to receive images taken by the user.

[1353] "Storage means" refers to a function or device for storing received images in a predetermined location.

[1354] "Image analysis API" refers to an external application program interface that analyzes received images and recognizes and lists ingredients within the images.

[1355] "Ingredient list" refers to a list of ingredients in an image obtained as a result of analysis by the image analysis API.

[1356] "Generative AI model" refers to an artificial intelligence model used to generate creative cooking recipes based on an ingredient list.

[1357] "Recipe generation means" refers to a function or device for generating a cooking recipe that takes nutritional balance into consideration based on an ingredient list.

[1358] "User Interface" refers to an interactive screen or application that provides the generated cooking recipe to the user.

[1359] This invention relates to a system that uses image analysis technology to automatically generate an ingredient list by taking a picture of the contents of a refrigerator and providing the system with original cooking recipes based on that list. This system provides a means to efficiently use ingredients in the refrigerator without wasting them.

[1360] Key Components of the System

[1361] 1. Image receiving means

[1362] It provides an interface for users to upload images of the inside of their refrigerator.

[1363] The server stores the images received from the user and prepares them for analysis.

[1364] 2. Image analysis methods

[1365] The server analyzes the received image and creates a list of ingredients. Here, an external image analysis API is used to recognize ingredients in the image and convert them into text data.

[1366] 3. Recipe Generation Method

[1367] The server uses a generative AI model based on the ingredient list to generate a cooking recipe that takes nutritional balance into consideration.

[1368] The generated recipe includes detailed information such as ingredients, steps, and cooking times.

[1369] 4. Means of providing information

[1370] The server has an interface for providing the generated recipe information to the user.

[1371] Users can receive recipe information through devices such as smartphones and PCs.

[1372] Specific processing content of the program

[1373] 1. User Actions

[1374] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door so that the food inside is clearly visible, and then pressing the shutter button.

[1375] Upload the captured image to the system's application using the app's "Upload Image" function.

[1376] 2. Server Processing

[1377] The server receives the uploaded image and saves it in a specified directory.

[1378] The saved image is sent to an external image analysis API to generate an ingredient list, and the JSON response returned from the API is analyzed to extract the ingredient list.

[1379] For example: "Carrots, onions, tomatoes, chicken" would be included in the list.

[1380] The generated ingredient list is sent to a generative AI model to generate recipes that take nutritional balance into account.

[1381] 3. Processing of generative AI models

[1382] A generative AI model creates creative cooking recipes based on a provided list of ingredients, including ingredients, steps, cooking times, and nutritional information.

[1383] For example, recipes for "Tomato and Chicken Stew" and "Carrot and Onion Salad" are generated.

[1384] 4. Server Information

[1385] The server provides the generated recipe to the user, and the user interface displays detailed information about the recipe to the user.

[1386] Users can view the provided recipes and actually cook them.

[1387] Specific examples

[1388] Suppose a user takes a photo of the contents of their refrigerator and uploads the image to the system. The server receives the image and uses an image analysis API to generate a list of ingredients: "carrots, onions, tomatoes, and chicken." The server then sends this list to a generative AI model, which generates recipes for "tomato and chicken stew" and "carrot and onion salad." Finally, the server provides these recipes to the user, who then begins cooking according to the recipes.

[1389] Prompt Sentence Examples

[1390] An example of a prompt to send to the image analysis API is:

[1391] {

[1392] "action": "parse_image",

[1393] "image_url": "YOUR_IMAGE_URL",

[1394] "response_format": "json"

[1395] }

[1396] Here's an example prompt to send to a generative AI model:

[1397] Recipe category to create: Cooking

[1398] Ingredients: carrots, onions, tomatoes, chicken

[1399] Generate two recipes, taking nutritional balance into consideration.

[1400] In this way, the present invention allows users to make better use of food in their refrigerators and reduce food waste.

[1401] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1402] Step 1:

[1403] User image capture and upload

[1404] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door and pressing the shutter button so that the food inside is clearly visible.

[1405] The captured image is uploaded to the system's application. The user opens the app, taps the "Upload Image" button, selects an image, and presses the "Send" button.

[1406] Input: Image of the contents of the refrigerator taken with a smartphone

[1407] Output: Images uploaded to the system

[1408] Step 2:

[1409] Receiving and saving images on the server

[1410] The server receives the uploaded image. The image data is sent to the server via an HTTP request.

[1411] Save the received images in a specified directory (e.g., / images / uploads).

[1412] Log the save directory path to ensure the image has been saved successfully.

[1413] Input: Image data uploaded by the user

[1414] Output: Images saved in the specified directory on the server

[1415] Step 3:

[1416] Calling the server's image analysis API

[1417] The server sends the stored image to an external image analysis API using an HTTP POST request, sending the image's binary data and URL to the API.

[1418] The image analysis API analyzes the image and returns a list of ingredients in JSON format.

[1419] The server parses the JSON response and extracts the list of ingredients in the image.

[1420] Input: Path to the image stored on the server

[1421] Output: Generated list of ingredients (e.g. carrot, onion, tomato, chicken)

[1422] Step 4:

[1423] Server recipe generation

[1424] Based on the ingredient list, the server sends a prompt to the generative AI model.

[1425] Recipe category to create: Cooking

[1426] Ingredients: carrots, onions, tomatoes, chicken

[1427] Generate two recipes, taking nutritional balance into consideration.

[1428] The generative AI model generates a recipe and returns it to the server, including ingredients, steps, cooking time, and nutritional information.

[1429] Input: Generated ingredient list and prompt statement

[1430] Output: Generated cooking recipe

[1431] Step 5:

[1432] Server recipe provision

[1433] The server displays the generated recipe in the user interface, allowing the user to view detailed recipe information on the app's "View Recipe" screen.

[1434] It allows users to browse recipes and provides instructions for actually cooking the food.

[1435] Input: Generated cooking recipe

[1436] Output: Recipe information displayed in the user interface

[1437] Through these steps, the system enables users to effectively utilize the ingredients in their refrigerator and provide efficient, nutritionally balanced meals.

[1438] (Application example 1)

[1439] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1440] Modern households are expected to efficiently utilize the ingredients in their refrigerators, but this often fails. Food waste is a problem due to ineffective use of ingredients. Many people are looking for new recipes and cooking videos when it comes to home cooking, but finding the right content can be a hassle. Furthermore, there is a lack of real-time interactive tools to improve cooking skills at home. Providing an effective system to solve these problems is a challenge.

[1441] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1442] In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for analyzing the received images to generate an ingredient list, means for generating cooking videos and articles related to cooking recipes based on the generated ingredient list, means for providing the user with cooking videos and articles related to the generated cooking recipes, and means for providing a virtual kitchen environment in which the user can perform cooking simulations in real time. This not only enables efficient use of ingredients in the refrigerator, but also makes it possible to provide the user with new recipes and cooking videos to help improve their cooking skills at home.

[1443] A "user" is someone who uses this system to manage ingredients in their refrigerator and obtain cooking recipes and related content.

[1444] "Photographing" refers to the act of taking a photo of the food in the refrigerator using a smartphone or other image capture device.

[1445] "Images inside the refrigerator" are photographic data of the inside of the refrigerator taken by the user, showing the state of the ingredients.

[1446] "Analysis" is the process by which the received image is processed by software to identify and list elements within the image.

[1447] The "ingredient list" is a list of names and information of ingredients extracted from the analyzed images of the inside of the refrigerator.

[1448] A "cooking recipe" is a document that contains specific cooking steps and a list of necessary ingredients, generated based on an ingredient list.

[1449] "Cooking videos" are video content that visually show cooking procedures and can be viewed by users.

[1450] An "article" is a text-based explanation related to cooking methods and ingredients.

[1451] A "virtual kitchen environment" is a virtual space that simulates a real-world kitchen and allows users to interactively experience cooking procedures.

[1452] "Real-time cooking simulation" refers to a user experiencing a simulation in which they react instantly within a virtual kitchen environment.

[1453] This invention is a system that efficiently utilizes ingredients in a refrigerator and provides users with cooking recipes and related content. This system is implemented mainly using a smartphone and a server.

[1454] 1. User Action

[1455] The user takes a picture of the food in the refrigerator using their smartphone. After taking the picture, the user uploads the picture to the server using the system's dedicated application. At this stage, the "Image Upload" function is used.

[1456] 2. Server processing

[1457] The server saves the received images in a specified directory. Then, it sends the saved images to an external image analysis API (e.g., Google Cloud Vision API) to generate an ingredient list. It parses the JSON response returned by the API and extracts the ingredient list. Specifically, the following prompt sentence is used:

[1458] Example prompt:

[1459] "Generate a list of ingredients in an image. Analyze the image and list the names of the ingredients."

[1460] 3. Recipe and content generation

[1461] The server generates a recipe based on the generated ingredient list using a generative AI model (e.g., OpenAI's GPT-4), taking nutritional balance into consideration. The generated recipe includes detailed information such as ingredients, steps, and cooking time. Specifically, the following prompt is used:

[1462] Example prompt:

[1463] "Create a delicious recipe using the following ingredients: chicken, onion, carrot, and tomato."

[1464] Additionally, to provide you with related cooking videos and articles, we use cooking video services (e.g., YouTube API, Vimeo API) to search for and provide related content to you.

[1465] 4. Virtual Kitchen Environment

[1466] The server provides a 3D virtual environment using platforms such as Unreal Engine and Unity, allowing users to simulate real-time cooking in a virtual kitchen environment, allowing them to improve their cooking skills by recreating real-life cooking procedures in a virtual environment.

[1467] Specific examples

[1468] A user takes a photo of the contents of their refrigerator with their smartphone and uploads the image to the system's application. The server receives the image and uses the Google Cloud Vision API to generate an ingredient list: "carrots, onions, tomatoes, and chicken." Next, GPT-4 is used to generate recipes: "Tomato and Chicken Stew" and "Carrot and Onion Salad." Additionally, cooking videos for these recipes are provided using the YouTube API. Users can browse these recipes and videos and experience cooking simulations in a virtual kitchen environment using an HMD (e.g., Oculus Rift).

[1469] This system allows users to efficiently use ingredients in their refrigerators and easily acquire and try out new recipes and cooking methods, which is expected to not only reduce food waste but also improve the cooking experience at home.

[1470] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1471] Step 1:

[1472] The user takes a picture of the inside of the refrigerator with their smartphone and uploads the picture to the system's application. The input is the image taken by the user, and the output is the upload of the image data to the server. Specifically, the user takes a picture of the food in the refrigerator using the camera app on their smartphone, and then uses the "image upload" function in the app to send the picture to the server.

[1473] Step 2:

[1474] The server saves the images it receives in a specified directory. The input is the uploaded image data, and the output is an image file saved on the server. Specifically, the server receives an HTTP request and saves the sent image data in a specific folder.

[1475] Step 3:

[1476] The server sends the saved image to an external image analysis API to generate an ingredient list. The input is the saved image file, and the output is the ingredient list. Specifically, the server sends the image data to an analysis service such as Google Cloud Vision API, analyzes the JSON response returned from the API, and lists the ingredients in the image.

[1477] Step 4:

[1478] The server generates a cooking recipe using a generative AI model based on the generated ingredient list. The input is the ingredient list, and the output is the generated cooking recipe. Specifically, the server creates a prompt sentence like the one below based on the ingredient list and sends it to a generative AI model such as OpenAI's GPT-4:

[1479] Example prompt:

[1480] "Create a delicious recipe using the following ingredients: chicken, onion, carrot, and tomato."

[1481] The generative AI model generates a recipe based on these prompts, and the server receives the results.

[1482] Step 5:

[1483] The server generates related cooking videos and articles. The input is a cooking recipe, and the output is related videos and articles. Specifically, the server searches for cooking videos related to the recipe using the YouTube API or Vimeo API, and provides the retrieved video links and articles to the user.

[1484] Step 6:

[1485] The server provides the generated recipe and related cooking videos and articles to the user. The input is the generated recipe and related content, and the output is the display on the user's device. Specifically, the server sends data in HTML or JSON format so that it can be displayed on the user's smartphone or PC browser.

[1486] Step 7:

[1487] It provides a virtual kitchen environment where users can perform cooking simulations in real time. The input is a cooking recipe and virtual kitchen environment data, and the output is a virtual experience on the user's HMD. Specifically, the server uses Unreal Engine or Unity to build the virtual kitchen, and the user wears the HMD to perform the cooking simulation in real time.

[1488] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1489] This invention combines a system that uses image analysis technology to automatically generate an ingredient list based on a user's image of the contents of their refrigerator, and then provides original cooking recipes based on that list with an emotion engine that recognizes the user's emotions, thereby enabling more personalized recipe suggestions. This system makes it possible to efficiently use ingredients in the refrigerator according to the user's condition, without wasting any ingredients.

[1490] System Overview

[1491] The system consists of the following main components:

[1492] 1. Image receiving means

[1493] It provides an interface for users to upload images of the inside of their refrigerator.

[1494] The server stores the images received from the user and prepares them for analysis.

[1495] 2. Image analysis methods

[1496] The server analyzes the received image and creates a list of ingredients. Here, an external image analysis API is used to recognize ingredients in the image and convert them into text data.

[1497] 3. Recipe Generation Method

[1498] The server uses a generative AI model based on the ingredient list to generate a cooking recipe that takes nutritional balance into consideration.

[1499] The generated recipe includes detailed information such as ingredients, steps, and cooking times.

[1500] 4. Emotion Engine

[1501] An engine that recognizes and analyzes user emotions.

[1502] For example, the user's facial expressions and voice can be analyzed to identify their emotional state (e.g., joy, sadness, stress).

[1503] 5. Means of providing information

[1504] The server has an interface for providing the generated recipe information to the user.

[1505] Users can receive recipe information through devices such as smartphones and PCs.

[1506] Specific processing content of the program

[1507] User Actions

[1508] The user takes a photo of the contents of the refrigerator with their smartphone, for example by opening the refrigerator door so that the food inside is clearly visible, and then pressing the shutter button.

[1509] Upload the captured image to the system's application using the app's "Upload Image" function.

[1510] Server Processing

[1511] The server receives the uploaded image and immediately saves it in a specified directory.

[1512] The saved image is sent to an external image analysis API to generate an ingredient list, and the JSON response returned from the API is analyzed to extract the ingredient list.

[1513] For example: "Carrots, onions, tomatoes, chicken" would be included in the list.

[1514] The generated ingredient list is sent to a generative AI model to generate recipes that take nutritional balance into account.

[1515] Emotion engine processing

[1516] The emotion engine recognizes the user's emotional state and uses facial recognition cameras and voice analysis to identify emotions when a user accesses the app.

[1517] For example, it can detect "joy" from a user's facial expression.

[1518] The server uses the emotional data obtained from the emotion engine and sends it to the generative AI model as additional information, which generates a recipe that is optimal for the user's current emotional state.

[1519] Processing generative AI models

[1520] A generative AI model creates creative cooking recipes based on a provided list of ingredients and user sentiment data, including ingredients, steps, cooking times, and nutritional information.

[1521] For example, recipes such as "Tomato and chicken stew" and "Carrot and onion salad" are generated, but the more suitable recipe is selected based on the user's emotions.

[1522] Server information provision

[1523] The server provides the generated recipe to the user, and the user interface displays detailed information about the recipe to the user.

[1524] Users can view the provided recipes and actually cook them.

[1525] Specific examples

[1526] Suppose a user takes a photo of the contents of their refrigerator and uploads the image to the system. The server receives this image and uses an image analysis API to generate an ingredient list such as "carrots, onions, tomatoes, and chicken." The server then uses an emotion engine to recognize the user's emotions before sending the ingredient list to the generative AI model. For example, if the emotion engine detects a "stressed state" from the user's facial expression, the generative AI model will prioritize generating recipes that have a relaxing effect on the user. In this case, a recipe such as "soup to make you feel calm" would be selected. Finally, the server provides this recipe to the user, who then begins cooking according to the recipe.

[1527] In this way, this system not only helps users make effective use of the ingredients in their refrigerator, but also provides a more personalized dining experience by suggesting recipes that match the user's emotional state.

[1528] The processing flow will be explained below.

[1529] Step 1:

[1530] A user takes a picture of the contents of the refrigerator with their smartphone. The user launches the smartphone's camera app, opens the refrigerator door, and presses the shutter button after making sure the food inside is clearly visible.

[1531] Step 2:

[1532] Users can upload images they have taken to the system's application by tapping the "Upload Image" button on the application screen, selecting an image from the album, and sending it.

[1533] Step 3:

[1534] The server receives the uploaded image. The server listens to the HTTP request and saves the received image data in a specified directory.

[1535] Step 4:

[1536] The server sends the received image to an external image analysis API, which then sends the image file to an API endpoint and executes the image analysis request.

[1537] Step 5:

[1538] The server receives a JSON response from the image analysis API. The response contains text data of the analyzed ingredients, such as a list of ingredients such as "carrot, onion, tomato, and chicken."

[1539] Step 6:

[1540] The server retrieves the generated ingredient list and sends it to the generative AI model. The server then sends a request including the ingredient list to the API endpoint of the generative AI model.

[1541] Step 7:

[1542] The device recognizes the user's emotional state. The device uses a camera and microphone to analyze the user's facial expressions and voice, and the emotion engine analyzes that data to identify the user's emotional state.

[1543] Step 8:

[1544] The device transmits the emotion data obtained from the emotion engine to the server, which uses this data as information for creating recipes.

[1545] Step 9:

[1546] A generative AI model generates a cooking recipe based on the ingredient list and emotional data. The generative AI model runs an algorithm to generate a cooking recipe that best suits the user's emotional state. For example, if the user is in a "stressed state," the model will prioritize suggestions that have a relaxing effect.

[1547] Step 10:

[1548] The server provides the generated recipe to the user, converts the received recipe data into a format for display on the user interface, and displays detailed recipe information on the user's smartphone screen.

[1549] Step 11:

[1550] Users can select the dish they want to make from the displayed recipes by tapping on their smartphone screen.

[1551] Step 12:

[1552] The app will then create a recipe based on the user's chosen recipe, and the user will follow the instructions displayed on the app to select the ingredients they need and begin cooking.

[1553] Example 2

[1554] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1555] Currently, there are systems that efficiently suggest cooking recipes using ingredients in the refrigerator, but they do not suggest personalized recipes based on the user's emotional state. As a result, there is a problem in that they cannot provide recipes that match the user's emotional state and preferences.

[1556] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1557] In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for sending the received images to an external image analysis API to generate an ingredient list, means for generating a cooking recipe using a generative AI model based on the ingredient list, means for recognizing the user's emotional state and selecting a personalized cooking recipe based on that, and means for providing the generated cooking recipe to the user, thereby enabling efficient and personalized cooking recipe suggestions tailored to the user's emotional state.

[1558] "User" refers to an individual who uses this system to take and upload images of the inside of a refrigerator.

[1559] "Means for receiving" refers to a device or program that has the function of transmitting images taken by a user to a server and receiving the image data.

[1560] "Image analysis API" refers to a program interface provided by an external service provider for extracting specific information from image data.

[1561] "Ingredient list" refers to text data in list format that shows the names and quantities of ingredients in the refrigerator, extracted using an image analysis API.

[1562] A "generative AI model" refers to an artificial intelligence algorithm that generates new information or content based on provided data.

[1563] A "cooking recipe" refers to detailed information that shows how to make a dish and the steps involved, generated based on a list of ingredients.

[1564] "Emotional state" refers to data that indicates a user's current emotional or psychological state.

[1565] "Personalized cooking recipes" refer to cooking recipes that are optimized based on the user's emotional state and individual preferences.

[1566] "Means for providing" refers to an interface or device for showing the generated cooking recipe to the user.

[1567] This invention combines a system that uses image analysis technology to automatically generate an ingredient list based on a user's image of the contents of their refrigerator, and then provides original cooking recipes based on that list with an emotion engine that recognizes the user's emotions, thereby enabling more personalized recipe suggestions. This system makes it possible to efficiently use ingredients in the refrigerator according to the user's condition, without wasting any ingredients.

[1568] The main components of the system include:

[1569] 1. Image receiving means

[1570] Provide an interface for users to upload images of the inside of their refrigerator. For example, users can take a picture of the contents of their refrigerator with their smartphone and use the "upload image" function within the app to send the image to the server.

[1571] 2. Image analysis methods

[1572] The server analyzes the received image and generates an ingredient list. This analysis uses an external image analysis API to recognize ingredients in the image and convert them into text data. For example, an ingredient list such as "carrot, onion, tomato, chicken" is extracted.

[1573] 3. Recipe Generation Method

[1574] The server generates a nutritionally balanced cooking recipe based on the generated ingredient list, using a generative AI model to create a recipe that includes detailed information such as ingredients, steps, and cooking time.

[1575] 4. Emotion recognition means

[1576] The emotion engine recognizes the user's emotional state. Specifically, when a user accesses the app, it uses a facial recognition camera and voice analysis to identify emotions. For example, it can detect "joy" or "stress" from the user's facial expressions.

[1577] 5. Means of providing information

[1578] The server has an interface to provide the generated recipes to users, who can receive the recipe information via their smartphones, PCs, or other devices.

[1579] Here is an example of this system:

[1580] 1. The user takes a picture of the contents of their refrigerator and uploads the image to the system using the "upload image" function within the app.

[1581] 2. The server receives this image and sends the saved image to an external image analysis API, which generates a list of ingredients, for example, "carrot, onion, tomato, chicken."

[1582] 3. The server sends the ingredient list to the generative AI model to generate a cooking recipe.

[1583] 4. The emotion engine recognizes the user’s emotional state. For example, if it detects a “stressed state,” the generative AI model will prioritize recipes that will have a relaxing effect on the user. In this case, a recipe such as “soup to make you feel calm” will be selected.

[1584] 5. Finally, the server provides this recipe to the user, who then starts cooking according to the recipe.

[1585] Prompt Sentence Examples

[1586] "Please upload a picture of the ingredients in your refrigerator."

[1587] "Generating the best recipe for your current emotional state. Please wait a moment."

[1588] "The following recipe was generated: 'Comforting Soup'."

[1589] Check out this recipe and start cooking.

[1590] In this way, this system not only allows users to make effective use of the ingredients in their refrigerator, but also provides a more personalized dining experience by suggesting recipes that match the user's emotional state.

[1591] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1592] Step 1:

[1593] The user takes a picture of the inside of the refrigerator with their smartphone. The user opens the refrigerator door to clearly see the food inside and presses the shutter button. The image is then saved on the smartphone.

[1594] Input: Actual ingredients in the fridge

[1595] Output: Food images stored on the smartphone

[1596] Step 2:

[1597] The user opens the system application and uploads the captured image. Using the "Upload Image" function in the application, the user selects the saved image file and sends it to the server.

[1598] Input: Food images stored on your smartphone

[1599] Output: Food images uploaded to the server

[1600] Step 3:

[1601] The server receives the uploaded image and immediately saves it in a specified directory.

[1602] Input: User-submitted food images

[1603] Output: Food images saved in a specified directory on the server

[1604] Step 4:

[1605] The server sends the stored image to an external image analysis API and requests analysis. It sends the image data using an HTTP request and receives a JSON-formatted response from the API.

[1606] Input: Food images stored on the server

[1607] Output: A list of ingredients in JSON format returned by the API

[1608] Step 5:

[1609] The server parses the JSON response returned from the API and extracts the list of ingredients. Specifically, it extracts the ingredient names from the JSON data and converts them into a text list.

[1610] Input: JSON formatted list of ingredients returned from the API

[1611] Output: Ingredient list in text format

[1612] Step 6:

[1613] The server sends the generated ingredient list to the generative AI model and requests it to generate a recipe. The server then sends the ingredient list to the generative AI model as input data and waits for the recipe to be generated.

[1614] Input: Text list of ingredients

[1615] Output: Input data sent to the generative AI model

[1616] Step 7:

[1617] The server activates the emotion engine to recognize the user's emotional state. When a user accesses the app, the camera and microphone are used to capture facial expressions and voice, and emotion recognition algorithms are applied.

[1618] Input: Facial expressions and voice data captured through a camera and microphone

[1619] Output: Recognized emotional state of the user

[1620] Step 8:

[1621] The server sends the emotion data obtained from the emotion engine to the generative AI model, which then uses the emotion data as additional information to generate recipes.

[1622] Input: Perceived emotional state of the user

[1623] Output: Emotion data sent to the generative AI model

[1624] Step 9:

[1625] The generative AI model creates creative cooking recipes based on the provided ingredient list and emotional data. Specifically, the algorithm inputs the ingredient list and emotional data to generate a recipe that includes ingredients, steps, cooking time, and nutritional information.

[1626] Input: Text list of ingredients and emotion data

[1627] Output: Generated cooking recipe

[1628] Step 10:

[1629] The server provides the generated recipe to the user, and the recipe details are displayed through an app or web interface.

[1630] Input: Generated cooking recipe

[1631] Output: The cooking recipe displayed to the user

[1632] Step 11:

[1633] Users actually cook according to the provided recipe, preparing ingredients and following the steps on their smartphone or PC screen as a reference.

[1634] Input: A cooking recipe shown to the user

[1635] Output: The actual cooked food

[1636] (Application example 2)

[1637] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1638] Conventional systems that analyze images of the inside of a refrigerator to generate an ingredient list and provide cooking recipes suggest uniform recipes regardless of the user's state, which means they have the problem of not being able to suggest appropriate recipes that reflect the user's current emotions or preferences.In particular, when users shop in physical stores, there are no product suggestions or guidance that reflect their emotional state, making it difficult to provide an efficient and satisfying shopping experience.

[1639] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for receiving images of the inside of the refrigerator taken by the user, means for analyzing the received images and generating an ingredient list, means for generating a cooking recipe based on the generated ingredient list, means for recognizing the user's emotions, means for generating a personalized cooking recipe based on the user's emotion data, and means for providing the generated cooking recipe to the user. This makes it possible to suggest ingredients and recipes that are optimal for the user's emotional state.

[1640] The "means for receiving images of the inside of the refrigerator taken by the user" is an interface for sending images of the inside of the refrigerator taken by the user using a smart device or the like to a server or cloud service.

[1641] The "means for analyzing received images and generating an ingredient list" is a process for identifying ingredients present in the refrigerator from the received image data using image analysis technology and creating a list of them.

[1642] The "means for generating a cooking recipe based on the generated ingredient list" refers to an algorithm or system for automatically generating a cooking recipe based on the ingredient information obtained by analysis.

[1643] "Means for recognizing user emotions" refers to hardware and software that analyzes the user's emotional state from their facial expressions and voice, and identifies emotions such as joy, sadness, and stress.

[1644] The "means for generating personalized cooking recipes based on user emotional data" is a process for generating cooking recipes that are optimally adjusted for each individual user based on the recognized emotional state of the user.

[1645] The "means for providing the generated cooking recipe to the user" refers to a communication interface or application for displaying or notifying the cooking recipe information generated by the server on the user's device.

[1646] The present invention is a system that allows a user to take a picture of the inside of a refrigerator, analyzes the picture to generate an ingredient list, and recognizes the user's emotional state to provide personalized cooking recipes. The system consists of the following main components:

[1647] 1. User Interface:

[1648] Users use devices such as smartphones or tablets to take pictures of the inside of the refrigerator and upload them to the application, which then sends the uploaded images to the system's server.

[1649] 2. Image analysis system:

[1650] The server analyzes the received image of the refrigerator interior using image recognition technology to generate an ingredient list. Specifically, it uses an external image analysis API to identify ingredients in the image and create a text list.

[1651] 3. Emotion Recognition System:

[1652] For emotion recognition, users input facial expressions and voice using their smart devices. The server analyzes these input data and uses an emotion recognition engine to identify the user's emotional state (e.g., joy, sadness, stress).

[1653] 4. Recipe Generation System:

[1654] The server uses a generative AI model to generate cooking recipes based on the generated ingredient list and emotional data. The cooking recipes are personalized, taking into account nutritional balance and the user's emotional state.

[1655] 5. Information provision system:

[1656] Finally, the server provides the generated cooking recipe to the user's device, where the user can access, view, and actually cook the recipe.

[1657] Hardware and software used

[1658] Smart devices (smartphones, tablets, etc.): Used by users to take pictures of the inside of the refrigerator and upload them to the application.

[1659] Server: Integrates and manages the image analysis system, emotion recognition system, recipe generation system, and information provision system.

[1660] Image analysis API: Uses an external API to analyze images of the inside of the refrigerator and identify ingredients.

[1661] Emotion recognition engine: An engine for identifying the user's emotional state from their facial expressions and voice.

[1662] Generative AI model: Generates personalized cooking recipes based on ingredient lists and emotional data.

[1663] Specific examples

[1664] For example, a user can use their smartphone to take a picture of the inside of their refrigerator and upload it to the system. The server receives this image and uses an image analysis API to generate an ingredient list such as "carrots, onions, tomatoes, and chicken." The server then uses the smart device's camera and microphone to recognize the user's emotional state and identify a "stressed state." Based on this information, the server uses a generative AI model to generate a recipe for a "calming soup" that will help the user relax and provides it to the user's smartphone. The user can then cook according to this recipe.

[1665] Prompt Sentence Examples

[1666] "When a user visits a physical store and starts shopping using the smart glasses, the system captures the user's face and recognizes their emotions. Depending on their emotions, the system will suggest the following product list:

[1667] 1. Happiness → Healthy foods, desserts

[1668] 2. Sadness → Chocolate, comfort food

[1669] 3. Stress → Herbal tea, relaxation items

[1670] This system allows users to effectively utilize ingredients in their refrigerators, receive personalized recipes based on their emotional state, and improve their shopping experience in brick-and-mortar stores.

[1671] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1672] Step 1:

[1673] A user takes a picture of the inside of the refrigerator using a smartphone or tablet and uploads the picture to the system through a dedicated application. The input is the image of the inside of the refrigerator, and the output is an image file sent to the server.

[1674] Step 2:

[1675] The server saves the image received from the user in a predetermined save directory. In this step, the image file sent from the user is received as input, and the image file is saved in the save directory as output.

[1676] Step 3:

[1677] The server sends the saved image to an external image analysis API and obtains the ingredient list as the analysis result. Here, the saved image file is used as input and the ingredient list in JSON format is obtained as output. Specifically, the ingredients in the image are identified and each is listed as text data.

[1678] Step 4:

[1679] The server analyzes the acquired ingredient list and sends it to the generative AI model. Here, the ingredient list is used as input, and data is prepared to be sent to the generative AI model as output.

[1680] Step 5:

[1681] The user inputs facial expressions and voice data into the system using the camera and microphone of the smart device. The input is the user's facial expressions and voice data, and the output is an audio file or image data sent to the server.

[1682] Step 6:

[1683] The server sends facial expression and voice data to an emotion recognition engine to identify the user's emotional state. Here, audio files and image data are used as input, and emotion recognition results are obtained as output. Specifically, emotional states such as "joy," "sadness," and "stress" are identified.

[1684] Step 7:

[1685] The server combines the emotion recognition results and the ingredient list and sends them to the generative AI model. Here, the emotional state data and ingredient list are used as input, and the data sent to the generative AI model as output is prepared.

[1686] Step 8:

[1687] A generative AI model generates personalized cooking recipes based on ingredient lists and emotional data. Here, the input is an ingredient list and emotional state data, and the output is a personalized cooking recipe, taking into account the ingredients, steps, time, and nutritional balance of the recipe.

[1688] Step 9:

[1689] The server provides the generated personalized cooking recipe to the user's device, and uses the generated cooking recipe as input to prepare recipe information to be displayed on the user's smartphone or tablet as output.

[1690] Step 10:

[1691] The user accesses the provided cooking recipe, browses it, and actually cooks the food. Here, the provided cooking recipe is used as input, and the completed dish is obtained as output. In concrete terms, the user cooks the food according to the recipe steps.

[1692] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1693] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1694] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1695] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1696] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1697] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1698] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1699] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1700] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1701] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1702] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1703] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1704] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1705] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1706] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1707] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1708] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1709] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1710] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1711] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1712] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1713] The following is further disclosed regarding the above embodiment.

[1714] (Claim 1)

[1715] means for receiving an image of the inside of the refrigerator taken by a user;

[1716] means for analyzing the received image and generating an ingredient list;

[1717] A means for generating a cooking recipe based on the generated ingredient list;

[1718] A means for providing the generated cooking recipe to a user;

[1719] A system including:

[1720] (Claim 2)

[1721] The system of claim 1, wherein the user sends images of the inside of the refrigerator to an external image analysis API to obtain a list of ingredients.

[1722] (Claim 3)

[1723] The system according to claim 1, wherein the cooking recipes provided to the user are generated taking nutritional balance into consideration.

[1724] "Example 1"

[1725] (Claim 1)

[1726] means for receiving an image of the inside of the refrigerator taken by a user;

[1727] means for storing the received images;

[1728] A means for calling an external image analysis API to analyze the stored images and generate an ingredient list;

[1729] A means for using a generative AI model to generate nutritionally balanced cooking recipes based on the ingredients list resulting from the analysis; and

[1730] a means for providing the generated cooking recipe through a user interface;

[1731] A system including:

[1732] (Claim 2)

[1733] The system of claim 1, wherein the stored image is sent to an external image analysis API to obtain an ingredient list.

[1734] (Claim 3)

[1735] The system according to claim 1, wherein the cooking recipe generated based on the ingredient list takes nutritional balance into consideration.

[1736] "Application Example 1"

[1737] (Claim 1)

[1738] means for receiving an image of the inside of the refrigerator taken by a user;

[1739] means for analyzing the received image and generating an ingredient list;

[1740] A means for generating cooking videos and articles related to cooking recipes based on the generated ingredient list;

[1741] A means for providing the user with cooking videos and articles related to the generated cooking recipes;

[1742] A virtual kitchen environment is provided, allowing users to simulate cooking in real time;

[1743] A system including:

[1744] (Claim 2)

[1745] The system of claim 1, wherein the user sends images of the inside of the refrigerator to an external image analysis API to obtain a list of ingredients.

[1746] (Claim 3)

[1747] The system according to claim 1, wherein the cooking recipes provided to the user are generated taking nutritional balance into consideration.

[1748] "Example 2: Combining Emotion Engines"

[1749] (Claim 1)

[1750] means for receiving an image of the inside of the refrigerator taken by a user;

[1751] A means for sending the received images to an external image analysis API to generate an ingredient list;

[1752] A means for generating a cooking recipe using a generative AI model based on an ingredient list;

[1753] a means for recognizing a user's emotional state and selecting personalized cooking recipes based thereon;

[1754] A means for providing the generated cooking recipe to a user;

[1755] A system including:

[1756] (Claim 2)

[1757] The system of claim 1, wherein the user sends images of the inside of the refrigerator to an external image analysis API to obtain a list of ingredients.

[1758] (Claim 3)

[1759] 10. The system of claim 1, wherein the cooking recipes provided to the user are personalized taking into account the emotional state of the user.

[1760] "Application example 2 when combining emotion engines"

[1761] (Claim 1)

[1762] means for receiving an image of the inside of the refrigerator taken by a user;

[1763] means for analyzing the received image and generating an ingredient list;

[1764] A means for generating a cooking recipe based on the generated ingredient list;

[1765] a means of recognizing a user's emotions;

[1766] A means for generating personalized cooking recipes based on user emotion data;

[1767] A means for providing the generated cooking recipe to a user;

[1768] A system including:

[1769] (Claim 2)

[1770] The system of claim 1, wherein the user sends images of the inside of the refrigerator to an external image analysis API to obtain a list of ingredients.

[1771] (Claim 3)

[1772] The system according to claim 1, wherein the cooking recipes provided to the user are generated taking into consideration nutritional balance and the emotional state of the user. [Explanation of symbols]

[1773] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for receiving an image of the inside of the refrigerator taken by a user; means for analyzing the received image and generating an ingredient list; A means for generating a cooking recipe based on the generated ingredient list; A means for providing the generated cooking recipe to a user; A system including:

2. The system according to claim 1, wherein the user transmits images of the inside of the refrigerator to an external image analysis API to obtain a list of ingredients.

3. The system according to claim 1, wherein the cooking recipes provided to the user are generated taking nutritional balance into consideration.

Images