System

Abstract

A system is provided.SOLUTION: A system comprising: camera means for recognizing food in a refrigerator; analysis means for analyzing image data obtained by the camera means to generate food information; search means for collecting the food information generated by the analysis means to search for an appropriate cooking recipe; display means for displaying the recipe found by the search means; and audio guide means and video guide means for assisting a user in cooking based on the recipe displayed by the display means.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] For home cooks, coming up with recipes for everyday meals is time-consuming and difficult, especially during busy periods. It's also difficult to efficiently utilize ingredients in the refrigerator, which can easily lead to waste. Furthermore, cooking processes that are unclear or complicated can be a hurdle, especially for beginners. These challenges can make cooking at home a burden, reducing productivity and enjoyment. [Means for solving the problem]

[0005] The present invention provides a system including a camera means for recognizing ingredients in a refrigerator, an analysis means for analyzing image data acquired by the camera means to generate food information, a search means for collecting the food information generated by the analysis means and searching for appropriate cooking recipes, a display means for displaying the recipes searched by the search means, and an audio guide means and a video guide means for assisting a user in cooking based on the recipe display by the display means. The analysis means has a function for optimizing recipes based on the user's past recipe selections, allergy information, and preferences, and the display means has either a touchscreen interface or a voice recognition interface and provides a means for the user to select a recipe. This allows the user to efficiently utilize ingredients in the refrigerator and enjoy cooking simply and effectively.

[0006] The "camera means" is a camera device for taking pictures of food in the refrigerator.

[0007] The "analysis means" is a device or software that analyzes the image data acquired by the camera means and generates food information.

[0008] "Food information" is information about the types and amounts of ingredients obtained from image data analyzed by the analysis means.

[0009] The "search means" is a device or software that searches a database for suitable cooking recipes based on food information.

[0010] The "display means" is a display device or user interface that displays to the user the recipes retrieved by the search means.

[0011] The "audio guide means" is a device or software that guides the user through cooking procedures by voice.

[0012] The "visual guide means" is a device or software that presents the cooking procedure to the user in visual form.

[0013] A "touchscreen interface" is a display device that allows a user to input information by touching it.

[0014] A "voice recognition interface" is a device or software that recognizes a user's voice input and performs an action based on that input. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The present invention is a system that uses a camera to recognize each food item in a refrigerator, analyzes image data based on the camera, proposes recipes to the user, and explains cooking procedures to the user. Below, the program processing of the system is explained in natural language.

[0037] System configuration

[0038] The system includes the following main components:

[0039] 1. Camera means: A camera for taking pictures of food in the refrigerator.

[0040] 2. Analysis means: A module that analyzes image data acquired by the camera means and generates food information.

[0041] 3. Search means: A server that searches for appropriate cooking recipes based on the generated food information.

[0042] 4. Display means: A refrigerator display that displays the recipes provided by the search means and allows the user to select one.

[0043] 5. Voice guidance means: A function that guides the user through cooking procedures by voice.

[0044] 6. Video guide means: A function that shows the cooking procedure to the user through video.

[0045] Processing flow

[0046] 1. Food Recognition in the Refrigerator

[0047] Terminal handling

[0048] A camera means within the refrigerator scans the interior periodically or each time the refrigerator door is closed, and acquires image data.

[0049] The acquired image data is sent to the analysis means.

[0050] The analysis means uses an object recognition algorithm to identify the photographed food items and generate an ingredient list.

[0051] Examples:

[0052] If there is cabbage, carrots, and chicken in the refrigerator, the camera will take a picture of them, and the analysis module will identify them and generate a list of ingredients such as "cabbage," "carrots," and "chicken." This list will be sent to the server.

[0053] 2. Recipe Suggestion Generation

[0054] Server Processing

[0055] The server receives the food information and queries a recipe database to retrieve recipes based on the food information.

[0056] Search results are optimized by taking into account profile data such as users' previous recipe choices, allergy information, and preferences.

[0057] The server generates a personalized recipe list and sends it to the refrigerator terminal.

[0058] Examples:

[0059] The server searches for "cabbage," "carrot," and "chicken," and based on the information that "the user likes stir-fried dishes," generates a list of recipes for "chicken and cabbage stir-fry" and "cabbage and carrot salad," and sends them to the refrigerator terminal.

[0060] 3. View and select recipes

[0061] Terminal and user handling

[0062] The terminal displays the recipe list on the display.

[0063] Users select recipes using a touchscreen or voice recognition interface.

[0064] Examples:

[0065] The refrigerator terminal's display shows "Stir-fried chicken and cabbage" and "Cabbage and carrot salad," and the user touches "Stir-fried chicken and cabbage" to select it.

[0066] 4. Cooking instructions

[0067] Terminal handling

[0068] The details of the selected recipe will be shown on the display.

[0069] The audio guide means guides the user through specific cooking procedures by voice.

[0070] A video guide means displays a video of each cooking step.

[0071] Examples:

[0072] After selecting "Stir-fried chicken and cabbage," the device will begin providing voice guidance, saying, "First, shred the cabbage," and play a video showing how to cut the cabbage accordingly. Next, it will provide voice and video guidance for each cooking step, such as, "Cut the chicken into bite-sized pieces and heat oil in a frying pan."

[0073] In this way, users can easily use the ingredients in their refrigerator and enjoy cooking effectively. This system plays a very beneficial role in reducing food waste and easing the burden of daily cooking.

[0074] The processing flow will be explained below.

[0075] Step 1:

[0076] Start scanning the inside of the refrigerator

[0077] Every time the refrigerator door is closed, the device activates the AI ​​camera and scans the inside.

[0078] Step 2:

[0079] Acquisition of image data

[0080] The device takes a photo of the inside of the refrigerator with its camera and acquires the image data.

[0081] Step 3:

[0082] Image data analysis

[0083] The device passes the acquired image data to an analysis means, which then runs a food recognition algorithm to generate food information, such as using an object detection algorithm to identify cabbage, carrots, and chicken.

[0084] Step 4:

[0085] Submitting food information

[0086] The device sends the analyzed food information to the server.

[0087] Step 5:

[0088] Recipe Search

[0089] The server searches a recipe database based on the received food information and extracts available recipes.

[0090] Step 6:

[0091] Recipe optimization

[0092] The server selects the best recipes based on the user's previous recipe choices, allergy information, and preferences. For example, if the user has a dairy allergy, dairy-free recipes will be prioritized.

[0093] Step 7:

[0094] Generate a recipe list

[0095] The server generates an optimized recipe list and transmits it to the refrigerator terminal.

[0096] Step 8:

[0097] Displaying the recipe list

[0098] The device displays the recipe list received from the server on the display. For example, "Stir-fried chicken and cabbage" or "Cabbage and carrot salad" are displayed.

[0099] Step 9:

[0100] User recipe selection

[0101] The user selects the desired recipe using the touchscreen or voice recognition interface.

[0102] Step 10:

[0103] View recipe details

[0104] The terminal displays detailed information about the selected recipe on the display.

[0105] Step 11:

[0106] Audio guide for cooking steps

[0107] The device will start a voice prompt and guide the user through each cooking step. For example, it might say, "First, shred the cabbage."

[0108] Step 12:

[0109] Video guide for cooking steps

[0110] The device will then display a visual guide for each cooking step in conjunction with the audio guide, for example showing how to cut cabbage.

[0111] Step 13:

[0112] User cooking execution

[0113] Users follow audio and video guides to cook and create delicious meals.

[0114] Through this processing step, users can efficiently utilize the ingredients in their refrigerator and enjoy cooking simply and effectively.

[0115] Example 1

[0116] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] In today's modern diet, efficiently managing the food in the refrigerator and providing recipes that utilize that food is important for reducing food waste and saving time and effort spent deciding what to cook. However, conventional refrigerators lacked these functions, forcing users to manually manage food and search for recipes. This led to problems such as food expiring and an increased burden on daily cooking.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0119] In this invention, the server includes an imaging means for recognizing the food inside, an analysis means for analyzing the image information obtained by the imaging means to generate food data, and a database search means for transferring the food data generated by the analysis means and searching for optimal cooking recipes. This makes it possible to automatically and efficiently manage the food inside the refrigerator and suggest appropriate recipes to the user.

[0120] The "imaging means" is a device for photographing the inside of the refrigerator and acquiring image information.

[0121] The "analysis means" is a device or software module that analyzes the image information acquired by the imaging means, identifies the type and amount of food, and generates food data as a result.

[0122] The "database search means" is a device or software module for searching an existing recipe database for optimal cooking recipes based on the food data generated by the analysis means.

[0123] The "display means" is a device for visually displaying to the user the recipes retrieved by the database search means.

[0124] The "voice guidance means" is a device or software module that provides audible guidance to the user on the cooking steps based on the recipe selected by the user.

[0125] The "visual guidance means" is a device or software module that visually guides the user through the cooking steps based on the recipe selected by the user.

[0126] A "tactile interface" is an interface that allows users to use touch controls to input selections and instructions such as recipes.

[0127] A "voice recognition interface" is an interface that recognizes the user's voice input and performs operations according to the instructions.

[0128] This invention relates to a system for efficiently managing food in a refrigerator and suggesting suitable cooking recipes to users. Specifically, the system includes the following main components:

[0129] 1. Imaging means: A camera for taking pictures of food in the refrigerator.

[0130] 2. Analysis means: A software module that analyzes the image information acquired by the imaging means and generates food data.

[0131] 3. Database search means: A server for searching for optimal cooking recipes based on the generated food data.

[0132] 4. Display means: refrigerator display for displaying the searched recipes.

[0133] 5. Voice guidance: A function that provides voice guidance for cooking steps based on a recipe.

[0134] 6. Video guidance: A function that provides video guidance on cooking steps based on a recipe.

[0135] 7. Haptic interface: An interface that allows the user to interact with the device by touch.

[0136] 8. Voice recognition interface: An interface that recognizes the user's voice input and performs operations.

[0137] Hardware and Software Use

[0138] The imaging means is a commercially available camera installed inside the refrigerator. Specifically, existing network cameras or smart cameras can be used. The analysis means and database search means use software running on a server. The analysis means includes a software module that implements a food recognition algorithm (e.g., YOLO, Inception). The database search means can use SQL or NoSQL databases. The audio guidance means and video guidance means use speech synthesis software and video playback software. The haptic interface and speech recognition interface use a touchscreen device and a speech recognition engine (e.g., Google Speech-to-Text), respectively.

[0139] Specific examples

[0140] For example, if you have cabbage, carrots, and chicken in your refrigerator, the system will do the following:

[0141] 1. The device uses the camera inside the refrigerator to take a picture of the inside and sends the image data to the server.

[0142] 2. The server's analysis module analyzes the image data and generates food data such as "cabbage," "carrot," and "chicken."

[0143] 3. The server searches the recipe database based on the generated food data and lists the most suitable recipes, taking into account the user's profile information (past recipe selections, allergy information, preferences).

[0144] 4. The display means displays the recipes "Chicken and Cabbage Stir-fry" and "Cabbage and Carrot Salad." The user uses the touchscreen to select "Chicken and Cabbage Stir-fry."

[0145] 5. Audio and visual guidance means guide the user through the cooking process step by step, providing specific instructions such as "First, shred the cabbage."

[0146] Prompt Sentence Examples

[0147] Below are some example prompts that the system can input to the generative AI model:

[0148] Say you have cabbage, carrots, and chicken in your fridge and your user likes stir-frying. Suggest a suitable recipe and provide step-by-step audio and visual instructions detailing each cooking step.

[0149] This system allows users to efficiently manage the food in their refrigerator, lighten the burden of daily cooking, reduce food waste, and even beginners can enjoy cooking with peace of mind.

[0150] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0151] Step 1:

[0152] Terminal handling

[0153] Input: Image data captured by a camera inside the refrigerator.

[0154] How it works: The device activates the camera periodically or whenever the refrigerator door is closed and takes a picture of the inside of the refrigerator.

[0155] Data processing: The captured image data is sent directly to the analysis means.

[0156] Output: Image data sent to analysis means.

[0157] Step 2:

[0158] Server Processing

[0159] Input: Image data sent from the device.

[0160] Operation: The server's analysis means analyzes the received image data using an object recognition algorithm (e.g., YOLO, Inception).

[0161] Data calculation: The algorithm identifies the type and location of food in the image and lists it as food data.

[0162] Output: A list of food data (e.g. "cabbage", "carrot", "chicken").

[0163] Step 3:

[0164] Server Processing

[0165] Input: The generated food data list.

[0166] Operation: The server uses the database search means to query the recipe database based on the food data list.

[0167] Data Calculation: Recipes listed as query results are optimized taking into account the user's profile information (previous recipe choices, allergies, preferences).

[0168] Output: A personalized recipe list.

[0169] Step 4:

[0170] Terminal handling

[0171] Input: Recipe list sent from the server.

[0172] Operation: A recipe list is displayed by the display means of the terminal.

[0173] Output: A visually displayed list of recipes (e.g. "Chicken and Cabbage Stir-fry", "Cabbage and Carrot Salad").

[0174] Step 5:

[0175] User Action

[0176] Input: Recipe list shown on the display.

[0177] How it works: The user selects the desired recipe using the touchscreen or voice recognition interface.

[0178] Output: The selected recipe (e.g. "Stir-fried chicken and cabbage").

[0179] Step 6:

[0180] Terminal handling

[0181] Input: The recipe selected by the user.

[0182] Operation: The audio and visual guidance means of the terminal guides the user through the cooking procedure step by step using audio and visual means.

[0183] Specific steps: "First, shred the cabbage," the voice instructions begin, and a video showing the steps for shredding the cabbage plays on the screen. Next, the instructions continue: "Cut the chicken into bite-sized pieces and heat oil in a frying pan."

[0184] Output: Audio and visual instructions for cooking procedures.

[0185] Through the above processing steps, users can efficiently manage the food in their refrigerator, easily select appropriate recipes, and proceed with cooking, thereby reducing food waste and easing the burden of cooking.

[0186] (Application example 1)

[0187] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0188] In modern life, there is a demand for effective use of food in the refrigerator and reduction of food waste. Furthermore, when shopping in a physical store, users can plan their shopping and cooking efficiently if they are provided with cooking method and recipe suggestions based on the food they are purchasing. However, previous systems have only been able to suggest recipes and provide cooking instructions based on the ingredients in the refrigerator, and have not been able to provide support for shopping in a physical store. Therefore, there is a need for a system that can seamlessly support the entire process from food selection to cooking in a physical store.

[0189] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0190] In this invention, the server includes image acquisition means for recognizing foods in the refrigerator, data analysis means for analyzing image data acquired by the image acquisition means to generate food information, information retrieval means for collecting the food information generated by the data analysis means and searching for appropriate recipes, display means for displaying the recipes retrieved by the information retrieval means, audio guidance means and video guidance means for supporting the user in cooking based on the recipe display by the display means, a visual device installed in the store for recognizing product selections by customers, product recognition means for collecting the product information recognized by the visual device, and control means for transmitting the product information generated by the product recognition means to the information retrieval means and displaying the recipes recommended by the information retrieval means on the display means. This makes it possible not only to recognize ingredients in the refrigerator but also to provide seamless support from food selection to cooking in a physical store.

[0191] An "image capture means" is a device or system for capturing an image of an object.

[0192] "Data analysis means" refers to a system or algorithm that processes and analyzes acquired image data to generate information about the target object.

[0193] The "information retrieval means" is a system for retrieving related information and content from an appropriate database based on the generated information.

[0194] "Display means" refers to a device or system for providing the searched information to the user in a visually recognizable form.

[0195] "Voice guidance means" refers to a system that provides guidance and instructions to users using voice.

[0196] A "visual guide" is a system that uses video to provide guides and instructions to users.

[0197] A "visual device" is a device or system used to recognize items selected by a customer in a physical store.

[0198] The "product recognition means" is a system for analyzing information acquired through a visual device and generating information about the recognized product.

[0199] A "control means" is a system or component that has the role of managing and coordinating the operations of various means and modules in an integrated manner.

[0200] The system according to the present invention uses a wide range of hardware and software to provide a consistent shopping and cooking assistance. The system includes the following main components:

[0201] 1. Image acquisition method

[0202] This includes cameras installed in the device's refrigerator or visual devices installed in physical stores, which take photos of food or products periodically or in response to user actions.

[0203] 2. Data analysis methods

[0204] Using a generative AI model such as TensorFlow, the captured image data is analyzed to identify the target food or product. This analysis process is performed on the server and generates the identified food information.

[0205] 3. Information search methods

[0206] The server uses the collected food information to search a recipe database for suitable recipes, optimizing the search results based on the user's previous choices, allergy information, and preferences.

[0207] 4. Display means

[0208] The recognized food information and searched recipes are displayed on visual devices such as refrigerator displays or smart glasses, which are equipped with tactile and voice input interfaces for users to select cooking methods.

[0209] 5. Audio and visual guidance means

[0210] Based on the selected recipe, the audio guide means provides audio instructions on specific cooking steps, and the visual guide means displays images of each cooking step to provide visual support to the user.

[0211] Examples:

[0212] When a user selects cabbage, carrots, and chicken at the supermarket, the visual device recognizes this and sends the information to the server. Based on this, the server searches for recipes such as "stir-fried chicken and cabbage" or "cabbage and carrot salad" and displays them on the smart glasses. If the user selects "stir-fried chicken and cabbage," the smart glasses will start a voice prompt saying, "First, shred the cabbage," and play a video showing how to cut the cabbage accordingly.

[0213] Example prompt sentence:

[0214] "Can you recommend a recipe for cabbage, carrots, and chicken?"

[0215] This system allows users to receive assistance with every step of the process, from selecting food in a physical store to cooking, not just with the ingredients in their refrigerator, reducing food waste and making cooking more efficient.

[0216] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0217] Step 1: Image acquisition

[0218] When a user opens or closes the refrigerator door, the image capture means of the device captures a photo of the food items inside the refrigerator. Also, when a user scans an item using smart glasses in a physical store, the visual device recognizes the item. Product image data from inside the refrigerator or store is acquired as input and sent to the analysis means. The output is the acquired raw image data.

[0219] Step 2: Image analysis

[0220] The server receives the image data sent from the image acquisition means and analyzes the data using a generative AI model such as TensorFlow. As input, it receives the acquired image data and identifies food or product information (e.g., "cabbage," "carrot," "chicken," etc.). The output is a list of recognized foods or products.

[0221] Step 3: Information search

[0222] The server collects food information generated by image analysis and searches a recipe database for suitable recipes based on that information. As input, it receives a list of recognized foods and identifies the best recipes based on the user's past choices, allergy information, and preferences. The output is a list of optimized recipes.

[0223] Step 4: View the recipe

[0224] The user receives the recipe information generated by the server in the refrigerator or in the brick-and-mortar store. Through the display means of the terminal, these recipes are visualized and displayed on the refrigerator display or smart glasses. As input, the searched recipe data is received and sent to the visual device. The output is a recipe list that the user can visually check.

[0225] Step 5: Recipe Selection

[0226] The user selects a preferred recipe from a displayed list of recipes. The user selects a recipe using a tactile or voice input interface. The system receives as input a visualized list of recipes and the user's selection action. The output is the specific recipe selected by the user.

[0227] Step 6: Cooking instructions

[0228] The terminal displays the details of the selected recipe and provides the user with cooking instructions using audio and video guidance means. As input, it receives detailed data of the selected recipe and plays audio and video guidance based on the instructions. As output, it provides audio and video guidance according to the cooking instructions.

[0229] In this way, by utilizing the system, users can reduce food waste and shop and cook more efficiently.

[0230] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0231] This invention combines a system that recognizes food in the refrigerator and suggests appropriate recipes to the user with an emotion engine that recognizes the user's emotions. Below, the system's program processing is explained in natural language with concrete examples.

[0232] System configuration

[0233] The system includes the following main components:

[0234] 1. Camera means: A camera for taking pictures of food in the refrigerator.

[0235] 2. Analysis means: A module that analyzes image data acquired by the camera means and generates food information.

[0236] 3. Search means: A server that searches for appropriate cooking recipes based on the generated food information.

[0237] 4. Display means: A refrigerator display that displays the recipes provided by the search means and allows the user to select one.

[0238] 5. Voice guidance means: A function that guides the user through cooking procedures by voice.

[0239] 6. Video guide means: A function that shows the cooking procedure to the user through video.

[0240] 7. Emotion engine: An engine that recognizes the user's emotions and reflects them in the system's operations.

[0241] Processing flow

[0242] 1. Food Recognition in the Refrigerator

[0243] Terminal handling

[0244] Every time the refrigerator door is closed, the device activates the AI ​​camera to scan the inside.

[0245] The image data acquired by the camera means is passed to the analysis means.

[0246] The analysis means generates food information using an object recognition algorithm, for example, identifying cabbage, carrots, and chicken.

[0247] Examples:

[0248] If there is cabbage, carrots, and chicken in the refrigerator, the camera will take a picture of them, and the analysis module will generate a list of ingredients such as "cabbage," "carrots," and "chicken." This list will be sent to the server.

[0249] 2. Recipe Suggestion Generation

[0250] Server Processing

[0251] The server queries a recipe database based on the received food information to search for available recipes.

[0252] The server selects the best recipes for you, taking into account your previous recipe choices, allergy information, and preferences. For example, if you have a dairy allergy, it will suggest dairy-free recipes.

[0253] Examples:

[0254] The server searches for recipes using "cabbage," "carrots," and "chicken," and based on the information that "the user likes stir-fried dishes," generates a list of recipes for "chicken and cabbage stir-fry" and "cabbage and carrot salad," and sends these to the refrigerator terminal.

[0255] 3. View and select recipes

[0256] Terminal and user handling

[0257] The device displays the recipe list on the display.

[0258] The user selects the desired recipe using the touchscreen or voice recognition interface.

[0259] Examples:

[0260] The refrigerator terminal's display shows "Stir-fried chicken and cabbage" and "Cabbage and carrot salad," and the user touches "Stir-fried chicken and cabbage" to select it.

[0261] 4. User Emotion Recognition

[0262] Emotion engine processing

[0263] The emotion engine analyzes the user's facial expressions, vocal tone, and physical movements to identify emotions.

[0264] The emotion engine transmits the identified emotion information to a server.

[0265] Examples:

[0266] When a user selects a recipe, the emotion engine analyzes the user's facial expressions and vocal tone to identify whether they are in a good mood, and this information is sent to the server.

[0267] 5. Cooking instructions

[0268] Terminal handling

[0269] The server optimizes recipe suggestions based on emotional information and sends them back to the device.

[0270] The terminal displays detailed information about the selected recipe on the display, and the audio guide means provides audio guidance of the cooking procedure.

[0271] A video guide means displays a video of each cooking step.

[0272] Examples:

[0273] After "Stir-fried chicken and cabbage" is selected, the device starts a voice prompt saying, "First, shred the cabbage," and plays a video showing how to cut the cabbage. Based on the results of the emotion engine, the voice guide provides instructions in a bright tone, lifting the user's spirits.

[0274] This system allows users to efficiently utilize the ingredients in their refrigerator while enjoying a personalized cooking experience that adapts to their emotions.

[0275] The processing flow will be explained below.

[0276] Step 1:

[0277] Start scanning the inside of the refrigerator

[0278] Every time the refrigerator door is closed, the device activates the AI ​​camera and scans the inside.

[0279] Step 2:

[0280] Acquisition of image data

[0281] The device takes a photo of the inside of the refrigerator with its camera and acquires the image data.

[0282] Step 3:

[0283] Image data analysis

[0284] The device passes the acquired image data to an analysis means, which then runs a food recognition algorithm to generate food information, such as using an object detection algorithm to identify cabbage, carrots, and chicken.

[0285] Step 4:

[0286] Submitting food information

[0287] The device sends the analyzed food information to the server.

[0288] Step 5:

[0289] Recipe Search

[0290] The server searches a recipe database based on the received food information and extracts available recipes.

[0291] Step 6:

[0292] Recipe optimization

[0293] The server selects the best recipes based on the user's previous recipe choices, allergy information, and preferences. For example, if the user has a dairy allergy, dairy-free recipes will be prioritized.

[0294] Step 7:

[0295] Generate a recipe list

[0296] The server generates an optimized recipe list and transmits it to the refrigerator terminal.

[0297] Step 8:

[0298] Displaying the recipe list

[0299] The device displays the recipe list received from the server on the display. For example, "Stir-fried chicken and cabbage" or "Cabbage and carrot salad" are displayed.

[0300] Step 9:

[0301] User recipe selection

[0302] The user selects the desired recipe using the touchscreen or voice recognition interface.

[0303] Step 10:

[0304] User emotion recognition

[0305] The device's emotion engine analyzes the user's facial expressions, voice tone, and physical movements to identify emotions. It also analyzes the user's facial expressions and voice tone when selecting recipes to identify emotions.

[0306] Step 11:

[0307] Sending emotional information

[0308] The terminal transmits the identified emotion information to the server.

[0309] Step 12:

[0310] Emotion-based recipe re-suggestion

[0311] The server takes emotional information into account and optimizes recipe suggestions to match the user's emotions. For example, if the user is feeling depressed, it will suggest easy and hassle-free recipes.

[0312] Step 13:

[0313] View recipe details

[0314] The terminal displays detailed information about the selected recipe on the display.

[0315] Step 14:

[0316] Audio guide for cooking steps

[0317] The device will start a voice prompt and guide the user through each cooking step. For example, it might say, "First, shred the cabbage."

[0318] Step 15:

[0319] Video guide for cooking steps

[0320] The device will then display a visual guide for each cooking step in conjunction with the audio guide, for example showing how to cut cabbage.

[0321] Step 16:

[0322] User cooking execution

[0323] Users follow audio and video guides to cook and create delicious meals.

[0324] Through this process, users can efficiently utilize the ingredients in their refrigerators, enjoy cooking simply and effectively, and optimize their cooking experience based on their emotions, providing a more personalized service.

[0325] Example 2

[0326] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0327] Conventional refrigerator systems have limited functionality for efficiently recognizing ingredients in the refrigerator and providing users with recipe suggestions based on that information. It is also difficult to provide personalized suggestions that take into account the user's emotions and individual preferences. Therefore, there is a demand for a system that allows users to easily find the perfect dish and enjoy cooking. In particular, there is a lack of functionality that recognizes the user's emotions in real time during the cooking process and provides guidance accordingly.

[0328] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0329] In this invention, the server includes an imaging means for recognizing food in the refrigerator, an analysis means for analyzing the image data obtained by the imaging means to generate ingredient information, a search means for collecting the ingredient information generated by the analysis means and searching for appropriate cooking procedures, a display means for displaying the cooking procedures searched by the search means, an audio guide means and a visual guide means for assisting the user in cooking based on the cooking procedure display by the display means, and an emotion recognition means for recognizing the user's emotions and reflecting them in the operation content. This makes it possible to effectively use ingredients in the refrigerator and provide personalized recipe suggestions and cooking support according to the user's emotions and preferences.

[0330] The "imaging means" is a device that acquires image data to identify ingredients in the refrigerator.

[0331] The "analysis means" is a device or module that analyzes the image data acquired by the imaging means and identifies the type and amount of food.

[0332] The "search means" is a device or software that searches a database for appropriate cooking procedures and recipes using the ingredient information generated by the analysis means.

[0333] The "display means" is a device for visually showing the cooking steps and recipes found by the search means to the user, and an example of this is a display.

[0334] "Audio guidance means" refers to a device or software that provides audio guidance to the user on cooking methods and procedures.

[0335] A "visual guide means" is a device or software that visually explains cooking methods and procedures to the user.

[0336] "Emotion recognition means" refers to a device or software that analyzes a user's facial expressions, voice, and physical movements to recognize their emotions.

[0337] The present invention combines a system that recognizes food in a refrigerator and suggests suitable cooking recipes to the user with technology that recognizes user emotions. The system includes the following main components: an imaging means, an analysis means, a search means, a display means, an audio guide means, a visual guide means, and an emotion recognition means.

[0338] Imaging means

[0339] The device is equipped with a camera that recognizes the food inside the refrigerator. The AI ​​camera activates every time the refrigerator door is closed and scans the ingredients inside.

[0340] Analysis means

[0341] The acquired image data is analyzed using an object recognition algorithm (e.g., YOLOv4) to generate food information. The analysis means identifies ingredients such as cabbage, carrots, and chicken, and sends the information to a server.

[0342] Search methods

[0343] The server queries the recipe database based on the received food information to find available recipes, and then selects the most suitable recipe taking into account the user's past recipe choices, allergy information, and preferences (e.g., dairy allergy).

[0344] Display means

[0345] The terminal displays the recipe list sent from the server on a display, which includes a touchscreen interface and a voice recognition interface (e.g., Google Assistant) that the user uses to select a desired recipe.

[0346] Acoustic and visual guide means

[0347] Based on the selected recipe, the audio guide module provides audio instructions for the cooking steps, and the visual guide module displays videos of each cooking step on the display. The audio guide module uses Amazon Polly, and the visual guide module plays videos using the YouTube API.

[0348] emotion recognition means

[0349] The emotion engine captures the user's facial expressions with a camera and analyzes them using the Facial Action Coding System. It also uses voice analysis technology (e.g., DeepSpeech) to analyze the tone of the voice and identify the user's emotions. The identified emotion information is sent to the server and used to optimize recipe suggestions.

[0350] Specific examples

[0351] For example, if there is cabbage, carrots, and chicken in the refrigerator, the imaging means takes a picture of them, and the analysis means generates an ingredient list of "cabbage," "carrots," and "chicken." This list is sent to the server, which searches for available recipes. The server then sends recipes such as "stir-fried chicken and cabbage" and "cabbage and carrot salad" to the terminal, which displays them on the display. When the user selects "stir-fried chicken and cabbage," the audio guide means begins guiding the user by saying, "First, shred the cabbage," and the visual guide means plays a video on how to cut the cabbage. At the same time, the emotion recognition means analyzes the user's facial expressions and voice tone to provide personalized guidance tailored to their emotions.

[0352] Prompt Sentence Examples

[0353] An example prompt for a generative AI model is:

[0354] "Please explain the process by which an AI system recognizes that there is cabbage, carrots, and chicken in the refrigerator and suggests appropriate recipes to the user. Also, please include specific actions and techniques to identify the user's emotions and provide guidance tailored to those emotions."

[0355] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0356] Step 1:

[0357] Scanning for ingredient recognition (processing on device)

[0358] Input: Status inside the refrigerator, refrigerator door opening / closing information.

[0359] How it works: Every time the refrigerator door is closed, the device detects this with a sensor and activates the AI ​​camera, which then captures images of the inside of the refrigerator and generates image data.

[0360] Output: Image data of the inside of the refrigerator.

[0361] Step 2:

[0362] Image data analysis (terminal processing)

[0363] Input: Image data of the inside of the refrigerator.

[0364] How it works: The device passes the image data to the analysis means, which uses an image analysis algorithm (e.g., YOLOv4) to identify the type and location of the food.

[0365] Output: A list of identified ingredients (e.g., cabbage, carrot, chicken).

[0366] Step 3:

[0367] Sending ingredient information and searching for recipes (server processing)

[0368] Input: Ingredient list sent from the device.

[0369] How it works: The server receives the ingredient list, queries the recipe database, and selects the best recipe, taking into account the user's previous recipe choices, allergy information, and preferences.

[0370] Output: A list of available recipes (e.g., Chicken and Cabbage Stir-fry, Cabbage and Carrot Salad).

[0371] Step 4:

[0372] Displaying recipes (terminal processing and user processing)

[0373] Input: Recipe list sent from the server.

[0374] How it works: The device displays a list of recipes on the display. The user selects the desired recipe using the touchscreen or voice recognition interface.

[0375] Output: The recipe selected by the user.

[0376] Step 5:

[0377] Emotion recognition (emotion engine processing)

[0378] Input: User facial expressions, voice tone, and physical movements.

[0379] How it works: The emotion recognition unit captures the user's facial expressions with a camera and analyzes them using the Facial Action Coding System. It also uses voice analysis technology to analyze the tone of the voice and identify the user's emotions.

[0380] Output: Identified user's sentiment information.

[0381] Step 6:

[0382] Optimization and presentation of cooking procedures (terminal processing)

[0383] Input: Selected recipe and emotion information sent from the server.

[0384] Operation: The server optimizes recipe suggestions based on emotion information and resends them to the device. The device then displays detailed cooking instructions on the screen, uses audio guidance (e.g., Amazon Polly) to provide audible instructions, and plays videos using visual guidance (e.g., YouTube API).

[0385] Output: Personalized cooking instructions and guide.

[0386] Examples:

[0387] Step 1 through Step 6:

[0388] If there is cabbage, carrots, or chicken in the refrigerator, the AI ​​camera will take a photo of it, and the device will pass the image data to the analysis means. The analysis means will identify the cabbage, carrots, and chicken, and send the ingredient list to the server. The server will query the recipe database based on the ingredient list and user information, generate a list of recipes for "chicken and cabbage stir-fry" and "cabbage and carrot salad," and send this to the device. When the user selects "chicken and cabbage stir-fry" on the touchscreen, the emotion engine will analyze the user's facial expressions and voice tone, guide them through the cooking steps in a bright tone, and play a video on how to cut cabbage on the display.

[0389] (Application example 2)

[0390] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0391] Conventional recipe suggestion systems have the ability to suggest recipes using ingredients in the refrigerator, but they are unable to flexibly suggest recipes based on the user's emotions. For example, it is difficult to suggest dishes that suit the user's mood when they are feeling depressed or excited. For this reason, there is a need for personalized recipe suggestions that take the user's psychological state into account.

[0392] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a camera means for recognizing food in the refrigerator, an analysis means for analyzing image data obtained by the camera means to generate food information, a search means for collecting the food information generated by the analysis means and searching for appropriate cooking recipes, a display means for displaying the recipes searched by the search means, and an emotion engine means for recognizing the user's emotions and reflecting them in recipe suggestions. This makes it possible to suggest recipes that take the user's emotional state into consideration.

[0393] "Camera means" refers to a photographic device used to recognize food items in the refrigerator.

[0394] The "analysis means" is a module that analyzes image data obtained by the camera means and generates food information.

[0395] The "search means" is a device or program that searches for appropriate cooking recipes based on the food information generated by the analysis means.

[0396] The "display means" is a device or interface for visually presenting the recipes retrieved by the search means to the user.

[0397] The "voice guide means" is a voice guide function that assists the user in cooking based on the recipe displayed by the display means.

[0398] The "visual guide means" is a function that presents the user with a visual guide to cooking procedures based on the recipe displayed by the display means.

[0399] The "emotion engine means" is an engine that recognizes the user's emotions and reflects them in the cooking recipe suggestions.

[0400] A "touch screen interface" is a tactile input device that allows a user to select recipes.

[0401] A "voice recognition interface" is a device or program that recognizes and inputs user voice instructions.

[0402] This invention combines a system that recognizes food in a refrigerator and suggests suitable cooking recipes to the user with an emotion engine that recognizes the user's emotions. Specific embodiments of the system will be described below.

[0403] System configuration

[0404] The system includes the following main components:

[0405] 1. Camera means: A camera for taking pictures of food in the refrigerator. This can be a camera built into a smartphone.

[0406] 2. Analysis means: A module that analyzes image data acquired by the camera means and generates food information. It uses a food recognition library.

[0407] 3. Search method: A server that searches for appropriate cooking recipes based on the generated food information. It uses the recipe search service API.

[0408] 4. Display means: An interface that displays recipes provided by the search means and allows the user to select one. An example is using the display of a smartphone.

[0409] 5. Voice guidance means: A function that guides the user through cooking procedures by voice. Use the voice guidance service.

[0410] 6. Video guide: A function that shows the cooking procedure to the user through video. It also has a video playback function.

[0411] 7. Emotion engine means: An engine that recognizes the user's emotions and reflects them in the system's operations. It uses an emotion recognition library.

[0412] System Operation

[0413] Food Recognition and Emotion Recognition

[0414] The device activates the camera when the refrigerator door is closed or based on a specific trigger, scans the interior, and photographs the food. The acquired image data is passed to an analysis means, which generates food information. A food recognition library is used to identify, for example, cabbage, carrots, chicken, etc. The device then analyzes the user's face to identify their emotions. The emotion recognition library analyzes the user's facial expressions and voice tone to generate emotional information.

[0415] Generating recipe suggestions

[0416] The server searches a recipe database based on the food information generated by the analysis means to query available recipes. It also optimizes recipes by taking into account the user's past recipe choices, allergy information, and preferences. It also takes into account emotional information to suggest recipes that suit the user's mood. For example, if the user is in high spirits, it will suggest a simple and fun recipe such as "stir-fried chicken and cabbage."

[0417] Viewing and Selecting Recipes

[0418] The provided recipes are displayed on the device's display, and the user can select them via the touchscreen or voice recognition interface. Based on the selected recipe, cooking instructions are provided through audio and visual guidance. The audio guidance is provided in a tone that corresponds to the user's emotions; for example, if the user is in a happy mood, a cheerful tone will be used to guide the user.

[0419] Specific examples

[0420] If there is cabbage, carrots, and chicken in the refrigerator, the camera takes a picture of them, and the analysis means generates a list of ingredients such as "cabbage," "carrots," and "chicken." This list is sent to the server. The server searches for recipes using "cabbage," "carrots," and "chicken," and based on the information that "the user likes stir-fries," generates a list of recipes such as "chicken and cabbage stir-fry" and "cabbage and carrot salad," and sends this to the refrigerator terminal. When the user selects a recipe, the emotion engine analyzes the user's facial expressions and voice tone to identify that they are in an excited mood. This information is sent to the server.

[0421] Prompt Sentence Examples

[0422] "The system of this invention recognizes the food in the refrigerator and suggests appropriate cooking recipes. The system has an emotion engine built in, which recognizes the user's emotions and reflects them in the recipe suggestions. If the user is smiling, it will tell us what kind of recipes should be suggested."

[0423] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0424] Step 1:

[0425] Food Recognition

[0426] Device behavior

[0427] The device activates the camera when the refrigerator door closes, scans the interior and takes pictures of food.

[0428] Input: Image data of the inside of the refrigerator acquired by a camera.

[0429] Data processing and calculation: Food information is extracted from image data using a food recognition library. This analysis generates a list of ingredients, such as "cabbage," "carrot," and "chicken."

[0430] Output: An ingredient list containing the generated food information.

[0431] Step 2:

[0432] User emotion recognition

[0433] Device behavior

[0434] The device takes a picture of the user's face with a camera and analyzes the user's emotions using an emotion recognition library.

[0435] Input: User's face image data acquired by camera.

[0436] Data processing and calculation: Using the emotion recognition library, the system analyzes facial expressions and voice tones to identify the user's emotions. Emotional information such as "excited," "neutral," and "sad" is generated.

[0437] Output: The generated emotion information.

[0438] Step 3:

[0439] Sending food and emotional information

[0440] Device behavior

[0441] The terminal transmits the generated food information and emotion information to the server.

[0442] Input: The food information and emotion information generated earlier.

[0443] Data processing and calculation: None in particular.

[0444] Output: Food information and emotion information sent to the server.

[0445] Step 4:

[0446] Recipe Search and Optimization

[0447] Server Operation

[0448] The server receives the food and emotion information and searches the recipe database based on it, optimizing the recipe taking into account past recipe choices, allergy information, and preferences.

[0449] Input: Food information, emotional information, past recipe selection data, allergy information, user preferences.

[0450] Data processing and data calculation: Execute database queries to filter the most suitable recipes from multiple options. Based on emotional information, narrow down the recipes that suit your mood.

[0451] Output: A list of suitable recipes.

[0452] Step 5:

[0453] Sending and displaying recipe lists

[0454] Server and terminal operation

[0455] The server sends the optimized recipe list to the terminal, which displays the recipe list on its display.

[0456] Input: Optimized recipe list.

[0457] Data processing and calculation: None in particular.

[0458] Output: Recipe list sent to the terminal, recipe list displayed on the display.

[0459] Step 6:

[0460] User recipe selection

[0461] User and device behavior

[0462] Users select the desired recipe using the touchscreen or voice recognition interface.

[0463] Input: User selection (touch or voice input).

[0464] Data processing / data calculation: Obtain data for the selected recipe.

[0465] Output: Selected recipe information.

[0466] Step 7:

[0467] Cooking procedure guide

[0468] Device behavior

[0469] Based on the selected recipe, the system provides audio and visual instructions for cooking, and adjusts the tone of the audio guide based on the user's emotional state.

[0470] Input: Selected recipe information, emotion information.

[0471] Data processing and calculation: Audio and visual guide generation. Optimizing the tone of audio guides based on emotional information.

[0472] Output: Audio and visual guide to cooking steps.

[0473] These steps allow users to efficiently utilize the ingredients in their refrigerator and enjoy a personalized cooking experience based on their emotions.

[0474] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0475] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0476] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0477] [Second embodiment]

[0478] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0479] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0480] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0481] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0482] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0483] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0484] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0485] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0486] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0487] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0488] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0489] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0490] The present invention is a system that uses a camera to recognize each food item in a refrigerator, analyzes image data based on the camera, proposes recipes to the user, and explains cooking procedures to the user. Below, the program processing of the system is explained in natural language.

[0491] System configuration

[0492] The system includes the following main components:

[0493] 1. Camera means: A camera for taking pictures of food in the refrigerator.

[0494] 2. Analysis means: A module that analyzes image data acquired by the camera means and generates food information.

[0495] 3. Search means: A server that searches for appropriate cooking recipes based on the generated food information.

[0496] 4. Display means: A refrigerator display that displays the recipes provided by the search means and allows the user to select one.

[0497] 5. Voice guidance means: A function that guides the user through cooking procedures by voice.

[0498] 6. Video guide means: A function that shows the cooking procedure to the user through video.

[0499] Processing flow

[0500] 1. Food Recognition in the Refrigerator

[0501] Terminal handling

[0502] A camera means within the refrigerator scans the interior periodically or each time the refrigerator door is closed, and acquires image data.

[0503] The acquired image data is sent to the analysis means.

[0504] The analysis means uses an object recognition algorithm to identify the photographed food items and generate an ingredient list.

[0505] Examples:

[0506] If there is cabbage, carrots, and chicken in the refrigerator, the camera will take a picture of them, and the analysis module will identify them and generate a list of ingredients such as "cabbage," "carrots," and "chicken." This list will be sent to the server.

[0507] 2. Recipe Suggestion Generation

[0508] Server Processing

[0509] The server receives the food information and queries a recipe database to retrieve recipes based on the food information.

[0510] Search results are optimized by taking into account profile data such as users' previous recipe choices, allergy information, and preferences.

[0511] The server generates a personalized recipe list and sends it to the refrigerator terminal.

[0512] Examples:

[0513] The server searches for "cabbage," "carrot," and "chicken," and based on the information that "the user likes stir-fried dishes," generates a list of recipes for "chicken and cabbage stir-fry" and "cabbage and carrot salad," and sends them to the refrigerator terminal.

[0514] 3. View and select recipes

[0515] Terminal and user handling

[0516] The terminal displays the recipe list on the display.

[0517] Users select recipes using a touchscreen or voice recognition interface.

[0518] Examples:

[0519] The refrigerator terminal's display shows "Stir-fried chicken and cabbage" and "Cabbage and carrot salad," and the user touches "Stir-fried chicken and cabbage" to select it.

[0520] 4. Cooking instructions

[0521] Terminal handling

[0522] The details of the selected recipe will be shown on the display.

[0523] The audio guide means guides the user through specific cooking procedures by voice.

[0524] A video guide means displays a video of each cooking step.

[0525] Examples:

[0526] After selecting "Stir-fried chicken and cabbage," the device will begin providing voice guidance, saying, "First, shred the cabbage," and play a video showing how to cut the cabbage accordingly. Next, it will provide voice and video guidance for each cooking step, such as, "Cut the chicken into bite-sized pieces and heat oil in a frying pan."

[0527] In this way, users can easily use the ingredients in their refrigerator and enjoy cooking effectively. This system plays a very beneficial role in reducing food waste and easing the burden of daily cooking.

[0528] The processing flow will be explained below.

[0529] Step 1:

[0530] Start scanning the inside of the refrigerator

[0531] Every time the refrigerator door is closed, the device activates the AI ​​camera and scans the inside.

[0532] Step 2:

[0533] Acquisition of image data

[0534] The device takes a photo of the inside of the refrigerator with its camera and acquires the image data.

[0535] Step 3:

[0536] Image data analysis

[0537] The device passes the acquired image data to an analysis means, which then runs a food recognition algorithm to generate food information, such as using an object detection algorithm to identify cabbage, carrots, and chicken.

[0538] Step 4:

[0539] Submitting food information

[0540] The device sends the analyzed food information to the server.

[0541] Step 5:

[0542] Recipe Search

[0543] The server searches a recipe database based on the received food information and extracts available recipes.

[0544] Step 6:

[0545] Recipe optimization

[0546] The server selects the best recipes based on the user's previous recipe choices, allergy information, and preferences. For example, if the user has a dairy allergy, dairy-free recipes will be prioritized.

[0547] Step 7:

[0548] Generate a recipe list

[0549] The server generates an optimized recipe list and transmits it to the refrigerator terminal.

[0550] Step 8:

[0551] Displaying the recipe list

[0552] The device displays the recipe list received from the server on the display. For example, "Stir-fried chicken and cabbage" or "Cabbage and carrot salad" are displayed.

[0553] Step 9:

[0554] User recipe selection

[0555] The user selects the desired recipe using the touchscreen or voice recognition interface.

[0556] Step 10:

[0557] View recipe details

[0558] The terminal displays detailed information about the selected recipe on the display.

[0559] Step 11:

[0560] Audio guide for cooking steps

[0561] The device will start a voice prompt and guide the user through each cooking step. For example, it might say, "First, shred the cabbage."

[0562] Step 12:

[0563] Video guide for cooking steps

[0564] The device will then display a visual guide for each cooking step in conjunction with the audio guide, for example showing how to cut cabbage.

[0565] Step 13:

[0566] User cooking execution

[0567] Users follow audio and video guides to cook and create delicious meals.

[0568] Through this processing step, users can efficiently utilize the ingredients in their refrigerator and enjoy cooking simply and effectively.

[0569] Example 1

[0570] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0571] In today's modern diet, efficiently managing the food in the refrigerator and providing recipes that utilize that food is important for reducing food waste and saving time and effort spent deciding what to cook. However, conventional refrigerators lacked these functions, forcing users to manually manage food and search for recipes. This led to problems such as food expiring and an increased burden on daily cooking.

[0572] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0573] In this invention, the server includes an imaging means for recognizing the food inside, an analysis means for analyzing the image information obtained by the imaging means to generate food data, and a database search means for transferring the food data generated by the analysis means and searching for optimal cooking recipes. This makes it possible to automatically and efficiently manage the food inside the refrigerator and suggest appropriate recipes to the user.

[0574] The "imaging means" is a device for photographing the inside of the refrigerator and acquiring image information.

[0575] The "analysis means" is a device or software module that analyzes the image information acquired by the imaging means, identifies the type and amount of food, and generates food data as a result.

[0576] The "database search means" is a device or software module for searching an existing recipe database for optimal cooking recipes based on the food data generated by the analysis means.

[0577] The "display means" is a device for visually displaying to the user the recipes retrieved by the database search means.

[0578] The "voice guidance means" is a device or software module that provides audible guidance to the user on the cooking steps based on the recipe selected by the user.

[0579] The "visual guidance means" is a device or software module that visually guides the user through the cooking steps based on the recipe selected by the user.

[0580] A "tactile interface" is an interface that allows users to use touch controls to input selections and instructions such as recipes.

[0581] A "voice recognition interface" is an interface that recognizes the user's voice input and performs operations according to the instructions.

[0582] This invention relates to a system for efficiently managing food in a refrigerator and suggesting suitable cooking recipes to users. Specifically, the system includes the following main components:

[0583] 1. Imaging means: A camera for taking pictures of food in the refrigerator.

[0584] 2. Analysis means: A software module that analyzes the image information acquired by the imaging means and generates food data.

[0585] 3. Database search means: A server for searching for optimal cooking recipes based on the generated food data.

[0586] 4. Display means: refrigerator display for displaying the searched recipes.

[0587] 5. Voice guidance: A function that provides voice guidance for cooking steps based on a recipe.

[0588] 6. Video guidance: A function that provides video guidance on cooking steps based on a recipe.

[0589] 7. Haptic interface: An interface that allows the user to interact with the device by touch.

[0590] 8. Voice recognition interface: An interface that recognizes the user's voice input and performs operations.

[0591] Hardware and Software Use

[0592] The imaging means is a commercially available camera installed inside the refrigerator. Specifically, existing network cameras or smart cameras can be used. The analysis means and database search means use software running on a server. The analysis means includes a software module that implements a food recognition algorithm (e.g., YOLO, Inception). The database search means can use SQL or NoSQL databases. The audio guidance means and video guidance means use speech synthesis software and video playback software. The haptic interface and speech recognition interface use a touchscreen device and a speech recognition engine (e.g., Google Speech-to-Text), respectively.

[0593] Specific examples

[0594] For example, if you have cabbage, carrots, and chicken in your refrigerator, the system will do the following:

[0595] 1. The device uses the camera inside the refrigerator to take a picture of the inside and sends the image data to the server.

[0596] 2. The server's analysis module analyzes the image data and generates food data such as "cabbage," "carrot," and "chicken."

[0597] 3. The server searches the recipe database based on the generated food data and lists the most suitable recipes, taking into account the user's profile information (past recipe selections, allergy information, preferences).

[0598] 4. The display means displays the recipes "Chicken and Cabbage Stir-fry" and "Cabbage and Carrot Salad." The user uses the touchscreen to select "Chicken and Cabbage Stir-fry."

[0599] 5. Audio and visual guidance means guide the user through the cooking process step by step, providing specific instructions such as "First, shred the cabbage."

[0600] Prompt Sentence Examples

[0601] Below are some example prompts that the system can input to the generative AI model:

[0602] Say you have cabbage, carrots, and chicken in your fridge and your user likes stir-frying. Suggest a suitable recipe and provide step-by-step audio and visual instructions detailing each cooking step.

[0603] This system allows users to efficiently manage the food in their refrigerator, lighten the burden of daily cooking, reduce food waste, and even beginners can enjoy cooking with peace of mind.

[0604] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0605] Step 1:

[0606] Terminal handling

[0607] Input: Image data captured by a camera inside the refrigerator.

[0608] How it works: The device activates the camera periodically or whenever the refrigerator door is closed and takes a picture of the inside of the refrigerator.

[0609] Data processing: The captured image data is sent directly to the analysis means.

[0610] Output: Image data sent to analysis means.

[0611] Step 2:

[0612] Server Processing

[0613] Input: Image data sent from the device.

[0614] Operation: The server's analysis means analyzes the received image data using an object recognition algorithm (e.g., YOLO, Inception).

[0615] Data calculation: The algorithm identifies the type and location of food in the image and lists it as food data.

[0616] Output: A list of food data (e.g. "cabbage", "carrot", "chicken").

[0617] Step 3:

[0618] Server Processing

[0619] Input: The generated food data list.

[0620] Operation: The server uses the database search means to query the recipe database based on the food data list.

[0621] Data Calculation: Recipes listed as query results are optimized taking into account the user's profile information (previous recipe choices, allergies, preferences).

[0622] Output: A personalized recipe list.

[0623] Step 4:

[0624] Terminal handling

[0625] Input: Recipe list sent from the server.

[0626] Operation: A recipe list is displayed by the display means of the terminal.

[0627] Output: A visually displayed list of recipes (e.g. "Chicken and Cabbage Stir-fry", "Cabbage and Carrot Salad").

[0628] Step 5:

[0629] User Action

[0630] Input: Recipe list shown on the display.

[0631] How it works: The user selects the desired recipe using the touchscreen or voice recognition interface.

[0632] Output: The selected recipe (e.g. "Stir-fried chicken and cabbage").

[0633] Step 6:

[0634] Terminal handling

[0635] Input: The recipe selected by the user.

[0636] Operation: The audio and visual guidance means of the terminal guides the user through the cooking procedure step by step using audio and visual means.

[0637] Specific steps: "First, shred the cabbage," the voice instructions begin, and a video showing the steps for shredding the cabbage plays on the screen. Next, the instructions continue: "Cut the chicken into bite-sized pieces and heat oil in a frying pan."

[0638] Output: Audio and visual instructions for cooking procedures.

[0639] Through the above processing steps, users can efficiently manage the food in their refrigerator, easily select appropriate recipes, and proceed with cooking, thereby reducing food waste and easing the burden of cooking.

[0640] (Application example 1)

[0641] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0642] In modern life, there is a demand for effective use of food in the refrigerator and reduction of food waste. Furthermore, when shopping in a physical store, users can plan their shopping and cooking efficiently if they are provided with cooking method and recipe suggestions based on the food they are purchasing. However, previous systems have only been able to suggest recipes and provide cooking instructions based on the ingredients in the refrigerator, and have not been able to provide support for shopping in a physical store. Therefore, there is a need for a system that can seamlessly support the entire process from food selection to cooking in a physical store.

[0643] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0644] In this invention, the server includes image acquisition means for recognizing foods in the refrigerator, data analysis means for analyzing image data acquired by the image acquisition means to generate food information, information retrieval means for collecting the food information generated by the data analysis means and searching for appropriate recipes, display means for displaying the recipes retrieved by the information retrieval means, audio guidance means and video guidance means for supporting the user in cooking based on the recipe display by the display means, a visual device installed in the store for recognizing product selections by customers, product recognition means for collecting the product information recognized by the visual device, and control means for transmitting the product information generated by the product recognition means to the information retrieval means and displaying the recipes recommended by the information retrieval means on the display means. This makes it possible not only to recognize ingredients in the refrigerator but also to provide seamless support from food selection to cooking in a physical store.

[0645] An "image capture means" is a device or system for capturing an image of an object.

[0646] "Data analysis means" refers to a system or algorithm that processes and analyzes acquired image data to generate information about the target object.

[0647] The "information retrieval means" is a system for retrieving related information and content from an appropriate database based on the generated information.

[0648] "Display means" refers to a device or system for providing the searched information to the user in a visually recognizable form.

[0649] "Voice guidance means" refers to a system that provides guidance and instructions to users using voice.

[0650] A "visual guide" is a system that uses video to provide guides and instructions to users.

[0651] A "visual device" is a device or system used to recognize items selected by a customer in a physical store.

[0652] The "product recognition means" is a system for analyzing information acquired through a visual device and generating information about the recognized product.

[0653] A "control means" is a system or component that has the role of managing and coordinating the operations of various means and modules in an integrated manner.

[0654] The system according to the present invention uses a wide range of hardware and software to provide a consistent shopping and cooking assistance. The system includes the following main components:

[0655] 1. Image acquisition method

[0656] This includes cameras installed in the device's refrigerator or visual devices installed in physical stores, which take photos of food or products periodically or in response to user actions.

[0657] 2. Data analysis methods

[0658] Using a generative AI model such as TensorFlow, the captured image data is analyzed to identify the target food or product. This analysis process is performed on the server and generates the identified food information.

[0659] 3. Information search methods

[0660] The server uses the collected food information to search a recipe database for suitable recipes, optimizing the search results based on the user's previous choices, allergy information, and preferences.

[0661] 4. Display means

[0662] The recognized food information and searched recipes are displayed on visual devices such as refrigerator displays or smart glasses, which are equipped with tactile and voice input interfaces for users to select cooking methods.

[0663] 5. Audio and visual guidance means

[0664] Based on the selected recipe, the audio guide means provides audio instructions on specific cooking steps, and the visual guide means displays images of each cooking step to provide visual support to the user.

[0665] Examples:

[0666] When a user selects cabbage, carrots, and chicken at the supermarket, the visual device recognizes this and sends the information to the server. Based on this, the server searches for recipes such as "stir-fried chicken and cabbage" or "cabbage and carrot salad" and displays them on the smart glasses. If the user selects "stir-fried chicken and cabbage," the smart glasses will start a voice prompt saying, "First, shred the cabbage," and play a video showing how to cut the cabbage accordingly.

[0667] Example prompt sentence:

[0668] "Can you recommend a recipe for cabbage, carrots, and chicken?"

[0669] This system allows users to receive assistance with every step of the process, from selecting food in a physical store to cooking, not just with the ingredients in their refrigerator, reducing food waste and making cooking more efficient.

[0670] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0671] Step 1: Image acquisition

[0672] When a user opens or closes the refrigerator door, the image capture means of the device captures a photo of the food items inside the refrigerator. Also, when a user scans an item using smart glasses in a physical store, the visual device recognizes the item. Product image data from inside the refrigerator or store is acquired as input and sent to the analysis means. The output is the acquired raw image data.

[0673] Step 2: Image analysis

[0674] The server receives the image data sent from the image acquisition means and analyzes the data using a generative AI model such as TensorFlow. As input, it receives the acquired image data and identifies food or product information (e.g., "cabbage," "carrot," "chicken," etc.). The output is a list of recognized foods or products.

[0675] Step 3: Information search

[0676] The server collects food information generated by image analysis and searches a recipe database for suitable recipes based on that information. As input, it receives a list of recognized foods and identifies the best recipes based on the user's past choices, allergy information, and preferences. The output is a list of optimized recipes.

[0677] Step 4: View the recipe

[0678] The user receives the recipe information generated by the server in the refrigerator or in the brick-and-mortar store. Through the display means of the terminal, these recipes are visualized and displayed on the refrigerator display or smart glasses. As input, the searched recipe data is received and sent to the visual device. The output is a recipe list that the user can visually check.

[0679] Step 5: Recipe Selection

[0680] The user selects a preferred recipe from a displayed list of recipes. The user selects a recipe using a tactile or voice input interface. The system receives as input a visualized list of recipes and the user's selection action. The output is the specific recipe selected by the user.

[0681] Step 6: Cooking instructions

[0682] The terminal displays the details of the selected recipe and provides the user with cooking instructions using audio and video guidance means. As input, it receives detailed data of the selected recipe and plays audio and video guidance based on the instructions. As output, it provides audio and video guidance according to the cooking instructions.

[0683] In this way, by utilizing the system, users can reduce food waste and shop and cook more efficiently.

[0684] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0685] This invention combines a system that recognizes food in the refrigerator and suggests appropriate recipes to the user with an emotion engine that recognizes the user's emotions. Below, the system's program processing is explained in natural language with concrete examples.

[0686] System configuration

[0687] The system includes the following main components:

[0688] 1. Camera means: A camera for taking pictures of food in the refrigerator.

[0689] 2. Analysis means: A module that analyzes image data acquired by the camera means and generates food information.

[0690] 3. Search means: A server that searches for appropriate cooking recipes based on the generated food information.

[0691] 4. Display means: A refrigerator display that displays the recipes provided by the search means and allows the user to select one.

[0692] 5. Voice guidance means: A function that guides the user through cooking procedures by voice.

[0693] 6. Video guide means: A function that shows the cooking procedure to the user through video.

[0694] 7. Emotion engine: An engine that recognizes the user's emotions and reflects them in the system's operations.

[0695] Processing flow

[0696] 1. Food Recognition in the Refrigerator

[0697] Terminal handling

[0698] Every time the refrigerator door is closed, the device activates the AI ​​camera to scan the inside.

[0699] The image data acquired by the camera means is passed to the analysis means.

[0700] The analysis means generates food information using an object recognition algorithm, for example, identifying cabbage, carrots, and chicken.

[0701] Examples:

[0702] If there is cabbage, carrots, and chicken in the refrigerator, the camera will take a picture of them, and the analysis module will generate a list of ingredients such as "cabbage," "carrots," and "chicken." This list will be sent to the server.

[0703] 2. Recipe Suggestion Generation

[0704] Server Processing

[0705] The server queries a recipe database based on the received food information to search for available recipes.

[0706] The server selects the best recipes for you, taking into account your previous recipe choices, allergy information, and preferences. For example, if you have a dairy allergy, it will suggest dairy-free recipes.

[0707] Examples:

[0708] The server searches for recipes using "cabbage," "carrots," and "chicken," and based on the information that "the user likes stir-fried dishes," generates a list of recipes for "chicken and cabbage stir-fry" and "cabbage and carrot salad," and sends these to the refrigerator terminal.

[0709] 3. View and select recipes

[0710] Terminal and user handling

[0711] The device displays the recipe list on the display.

[0712] The user selects the desired recipe using the touchscreen or voice recognition interface.

[0713] Examples:

[0714] The refrigerator terminal's display shows "Stir-fried chicken and cabbage" and "Cabbage and carrot salad," and the user touches "Stir-fried chicken and cabbage" to select it.

[0715] 4. User Emotion Recognition

[0716] Emotion engine processing

[0717] The emotion engine analyzes the user's facial expressions, vocal tone, and physical movements to identify emotions.

[0718] The emotion engine transmits the identified emotion information to a server.

[0719] Examples:

[0720] When a user selects a recipe, the emotion engine analyzes the user's facial expressions and vocal tone to identify whether they are in a good mood, and this information is sent to the server.

[0721] 5. Cooking instructions

[0722] Terminal handling

[0723] The server optimizes recipe suggestions based on emotional information and sends them back to the device.

[0724] The terminal displays detailed information about the selected recipe on the display, and the audio guide means provides audio guidance of the cooking procedure.

[0725] A video guide means displays a video of each cooking step.

[0726] Examples:

[0727] After "Stir-fried chicken and cabbage" is selected, the device starts a voice prompt saying, "First, shred the cabbage," and plays a video showing how to cut the cabbage. Based on the results of the emotion engine, the voice guide provides instructions in a bright tone, lifting the user's spirits.

[0728] This system allows users to efficiently utilize the ingredients in their refrigerator while enjoying a personalized cooking experience that adapts to their emotions.

[0729] The processing flow will be explained below.

[0730] Step 1:

[0731] Start scanning the inside of the refrigerator

[0732] Every time the refrigerator door is closed, the device activates the AI ​​camera and scans the inside.

[0733] Step 2:

[0734] Acquisition of image data

[0735] The device takes a photo of the inside of the refrigerator with its camera and acquires the image data.

[0736] Step 3:

[0737] Image data analysis

[0738] The device passes the acquired image data to an analysis means, which then runs a food recognition algorithm to generate food information, such as using an object detection algorithm to identify cabbage, carrots, and chicken.

[0739] Step 4:

[0740] Submitting food information

[0741] The device sends the analyzed food information to the server.

[0742] Step 5:

[0743] Recipe Search

[0744] The server searches a recipe database based on the received food information and extracts available recipes.

[0745] Step 6:

[0746] Recipe optimization

[0747] The server selects the best recipes based on the user's previous recipe choices, allergy information, and preferences. For example, if the user has a dairy allergy, dairy-free recipes will be prioritized.

[0748] Step 7:

[0749] Generate a recipe list

[0750] The server generates an optimized recipe list and transmits it to the refrigerator terminal.

[0751] Step 8:

[0752] Displaying the recipe list

[0753] The device displays the recipe list received from the server on the display. For example, "Stir-fried chicken and cabbage" or "Cabbage and carrot salad" are displayed.

[0754] Step 9:

[0755] User recipe selection

[0756] The user selects the desired recipe using the touchscreen or voice recognition interface.

[0757] Step 10:

[0758] User emotion recognition

[0759] The device's emotion engine analyzes the user's facial expressions, voice tone, and physical movements to identify emotions. It also analyzes the user's facial expressions and voice tone when selecting recipes to identify emotions.

[0760] Step 11:

[0761] Sending emotional information

[0762] The terminal transmits the identified emotion information to the server.

[0763] Step 12:

[0764] Emotion-based recipe re-suggestion

[0765] The server takes emotional information into account and optimizes recipe suggestions to match the user's emotions. For example, if the user is feeling depressed, it will suggest easy and hassle-free recipes.

[0766] Step 13:

[0767] View recipe details

[0768] The terminal displays detailed information about the selected recipe on the display.

[0769] Step 14:

[0770] Audio guide for cooking steps

[0771] The device will start a voice prompt and guide the user through each cooking step. For example, it might say, "First, shred the cabbage."

[0772] Step 15:

[0773] Video guide for cooking steps

[0774] The device will then display a visual guide for each cooking step in conjunction with the audio guide, for example showing how to cut cabbage.

[0775] Step 16:

[0776] User cooking execution

[0777] Users follow audio and video guides to cook and create delicious meals.

[0778] Through this process, users can efficiently utilize the ingredients in their refrigerators, enjoy cooking simply and effectively, and optimize their cooking experience based on their emotions, providing a more personalized service.

[0779] Example 2

[0780] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0781] Conventional refrigerator systems have limited functionality for efficiently recognizing ingredients in the refrigerator and providing users with recipe suggestions based on that information. It is also difficult to provide personalized suggestions that take into account the user's emotions and individual preferences. Therefore, there is a demand for a system that allows users to easily find the perfect dish and enjoy cooking. In particular, there is a lack of functionality that recognizes the user's emotions in real time during the cooking process and provides guidance accordingly.

[0782] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0783] In this invention, the server includes an imaging means for recognizing food in the refrigerator, an analysis means for analyzing the image data obtained by the imaging means to generate ingredient information, a search means for collecting the ingredient information generated by the analysis means and searching for appropriate cooking procedures, a display means for displaying the cooking procedures searched by the search means, an audio guide means and a visual guide means for assisting the user in cooking based on the cooking procedure display by the display means, and an emotion recognition means for recognizing the user's emotions and reflecting them in the operation content. This makes it possible to effectively use ingredients in the refrigerator and provide personalized recipe suggestions and cooking support according to the user's emotions and preferences.

[0784] The "imaging means" is a device that acquires image data to identify ingredients in the refrigerator.

[0785] The "analysis means" is a device or module that analyzes the image data acquired by the imaging means and identifies the type and amount of food.

[0786] The "search means" is a device or software that searches a database for appropriate cooking procedures and recipes using the ingredient information generated by the analysis means.

[0787] The "display means" is a device for visually showing the cooking steps and recipes found by the search means to the user, and an example of this is a display.

[0788] "Audio guidance means" refers to a device or software that provides audio guidance to the user on cooking methods and procedures.

[0789] A "visual guide means" is a device or software that visually explains cooking methods and procedures to the user.

[0790] "Emotion recognition means" refers to a device or software that analyzes a user's facial expressions, voice, and physical movements to recognize their emotions.

[0791] The present invention combines a system that recognizes food in a refrigerator and suggests suitable cooking recipes to the user with technology that recognizes user emotions. The system includes the following main components: an imaging means, an analysis means, a search means, a display means, an audio guide means, a visual guide means, and an emotion recognition means.

[0792] Imaging means

[0793] The device is equipped with a camera that recognizes the food inside the refrigerator. The AI ​​camera activates every time the refrigerator door is closed and scans the ingredients inside.

[0794] Analysis means

[0795] The acquired image data is analyzed using an object recognition algorithm (e.g., YOLOv4) to generate food information. The analysis means identifies ingredients such as cabbage, carrots, and chicken, and sends the information to a server.

[0796] Search methods

[0797] The server queries the recipe database based on the received food information to find available recipes, and then selects the most suitable recipe taking into account the user's past recipe choices, allergy information, and preferences (e.g., dairy allergy).

[0798] Display means

[0799] The terminal displays the recipe list sent from the server on a display, which includes a touchscreen interface and a voice recognition interface (e.g., Google Assistant) that the user uses to select a desired recipe.

[0800] Acoustic and visual guide means

[0801] Based on the selected recipe, the audio guide module provides audio instructions for the cooking steps, and the visual guide module displays videos of each cooking step on the display. The audio guide module uses Amazon Polly, and the visual guide module plays videos using the YouTube API.

[0802] emotion recognition means

[0803] The emotion engine captures the user's facial expressions with a camera and analyzes them using the Facial Action Coding System. It also uses voice analysis technology (e.g., DeepSpeech) to analyze the tone of the voice and identify the user's emotions. The identified emotion information is sent to the server and used to optimize recipe suggestions.

[0804] Specific examples

[0805] For example, if there is cabbage, carrots, and chicken in the refrigerator, the imaging means takes a picture of them, and the analysis means generates an ingredient list of "cabbage," "carrots," and "chicken." This list is sent to the server, which searches for available recipes. The server then sends recipes such as "stir-fried chicken and cabbage" and "cabbage and carrot salad" to the terminal, which displays them on the display. When the user selects "stir-fried chicken and cabbage," the audio guide means begins guiding the user by saying, "First, shred the cabbage," and the visual guide means plays a video on how to cut the cabbage. At the same time, the emotion recognition means analyzes the user's facial expressions and voice tone to provide personalized guidance tailored to their emotions.

[0806] Prompt Sentence Examples

[0807] An example prompt for a generative AI model is:

[0808] "Please explain the process by which an AI system recognizes that there is cabbage, carrots, and chicken in the refrigerator and suggests appropriate recipes to the user. Also, please include specific actions and techniques to identify the user's emotions and provide guidance tailored to those emotions."

[0809] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0810] Step 1:

[0811] Scanning for ingredient recognition (processing on device)

[0812] Input: Status inside the refrigerator, refrigerator door opening / closing information.

[0813] How it works: Every time the refrigerator door is closed, the device detects this with a sensor and activates the AI ​​camera, which then captures images of the inside of the refrigerator and generates image data.

[0814] Output: Image data of the inside of the refrigerator.

[0815] Step 2:

[0816] Image data analysis (terminal processing)

[0817] Input: Image data of the inside of the refrigerator.

[0818] How it works: The device passes the image data to the analysis means, which uses an image analysis algorithm (e.g., YOLOv4) to identify the type and location of the food.

[0819] Output: A list of identified ingredients (e.g., cabbage, carrot, chicken).

[0820] Step 3:

[0821] Sending ingredient information and searching for recipes (server processing)

[0822] Input: Ingredient list sent from the device.

[0823] How it works: The server receives the ingredient list, queries the recipe database, and selects the best recipe, taking into account the user's previous recipe choices, allergy information, and preferences.

[0824] Output: A list of available recipes (e.g., Chicken and Cabbage Stir-fry, Cabbage and Carrot Salad).

[0825] Step 4:

[0826] Displaying recipes (terminal processing and user processing)

[0827] Input: Recipe list sent from the server.

[0828] How it works: The device displays a list of recipes on the display. The user selects the desired recipe using the touchscreen or voice recognition interface.

[0829] Output: The recipe selected by the user.

[0830] Step 5:

[0831] Emotion recognition (emotion engine processing)

[0832] Input: User facial expressions, voice tone, and physical movements.

[0833] How it works: The emotion recognition unit captures the user's facial expressions with a camera and analyzes them using the Facial Action Coding System. It also uses voice analysis technology to analyze the tone of the voice and identify the user's emotions.

[0834] Output: Identified user's sentiment information.

[0835] Step 6:

[0836] Optimization and presentation of cooking procedures (terminal processing)

[0837] Input: Selected recipe and emotion information sent from the server.

[0838] Operation: The server optimizes recipe suggestions based on emotion information and resends them to the device. The device then displays detailed cooking instructions on the screen, uses audio guidance (e.g., Amazon Polly) to provide audible instructions, and plays videos using visual guidance (e.g., YouTube API).

[0839] Output: Personalized cooking instructions and guide.

[0840] Examples:

[0841] Step 1 through Step 6:

[0842] If there is cabbage, carrots, or chicken in the refrigerator, the AI ​​camera will take a photo of it, and the device will pass the image data to the analysis means. The analysis means will identify the cabbage, carrots, and chicken, and send the ingredient list to the server. The server will query the recipe database based on the ingredient list and user information, generate a list of recipes for "chicken and cabbage stir-fry" and "cabbage and carrot salad," and send this to the device. When the user selects "chicken and cabbage stir-fry" on the touchscreen, the emotion engine will analyze the user's facial expressions and voice tone, guide them through the cooking steps in a bright tone, and play a video on how to cut cabbage on the display.

[0843] (Application example 2)

[0844] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0845] Conventional recipe suggestion systems have the ability to suggest recipes using ingredients in the refrigerator, but they are unable to flexibly suggest recipes based on the user's emotions. For example, it is difficult to suggest dishes that suit the user's mood when they are feeling depressed or excited. For this reason, there is a need for personalized recipe suggestions that take the user's psychological state into account.

[0846] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a camera means for recognizing food in the refrigerator, an analysis means for analyzing image data obtained by the camera means to generate food information, a search means for collecting the food information generated by the analysis means and searching for appropriate cooking recipes, a display means for displaying the recipes searched by the search means, and an emotion engine means for recognizing the user's emotions and reflecting them in recipe suggestions. This makes it possible to suggest recipes that take the user's emotional state into consideration.

[0847] "Camera means" refers to a photographic device used to recognize food items in the refrigerator.

[0848] The "analysis means" is a module that analyzes image data obtained by the camera means and generates food information.

[0849] The "search means" is a device or program that searches for appropriate cooking recipes based on the food information generated by the analysis means.

[0850] The "display means" is a device or interface for visually presenting the recipes retrieved by the search means to the user.

[0851] The "voice guide means" is a voice guide function that assists the user in cooking based on the recipe displayed by the display means.

[0852] The "visual guide means" is a function that presents the user with a visual guide to cooking procedures based on the recipe displayed by the display means.

[0853] The "emotion engine means" is an engine that recognizes the user's emotions and reflects them in the cooking recipe suggestions.

[0854] A "touch screen interface" is a tactile input device that allows a user to select recipes.

[0855] A "voice recognition interface" is a device or program that recognizes and inputs user voice instructions.

[0856] This invention combines a system that recognizes food in a refrigerator and suggests suitable cooking recipes to the user with an emotion engine that recognizes the user's emotions. Specific embodiments of the system will be described below.

[0857] System configuration

[0858] The system includes the following main components:

[0859] 1. Camera means: A camera for taking pictures of food in the refrigerator. This can be a camera built into a smartphone.

[0860] 2. Analysis means: A module that analyzes image data acquired by the camera means and generates food information. It uses a food recognition library.

[0861] 3. Search method: A server that searches for appropriate cooking recipes based on the generated food information. It uses the recipe search service API.

[0862] 4. Display means: An interface that displays recipes provided by the search means and allows the user to select one. An example is using the display of a smartphone.

[0863] 5. Voice guidance means: A function that guides the user through cooking procedures by voice. Use the voice guidance service.

[0864] 6. Video guide: A function that shows the cooking procedure to the user through video. It also has a video playback function.

[0865] 7. Emotion engine means: An engine that recognizes the user's emotions and reflects them in the system's operations. It uses an emotion recognition library.

[0866] System Operation

[0867] Food Recognition and Emotion Recognition

[0868] The device activates the camera when the refrigerator door is closed or based on a specific trigger, scans the interior, and photographs the food. The acquired image data is passed to an analysis means, which generates food information. A food recognition library is used to identify, for example, cabbage, carrots, chicken, etc. The device then analyzes the user's face to identify their emotions. The emotion recognition library analyzes the user's facial expressions and voice tone to generate emotional information.

[0869] Generating recipe suggestions

[0870] The server searches a recipe database based on the food information generated by the analysis means to query available recipes. It also optimizes recipes by taking into account the user's past recipe choices, allergy information, and preferences. It also takes into account emotional information to suggest recipes that suit the user's mood. For example, if the user is in high spirits, it will suggest a simple and fun recipe such as "stir-fried chicken and cabbage."

[0871] Viewing and Selecting Recipes

[0872] The provided recipes are displayed on the device's display, and the user can select them via the touchscreen or voice recognition interface. Based on the selected recipe, cooking instructions are provided through audio and visual guidance. The audio guidance is provided in a tone that corresponds to the user's emotions; for example, if the user is in a happy mood, a cheerful tone will be used to guide the user.

[0873] Specific examples

[0874] If there is cabbage, carrots, and chicken in the refrigerator, the camera takes a picture of them, and the analysis means generates a list of ingredients such as "cabbage," "carrots," and "chicken." This list is sent to the server. The server searches for recipes using "cabbage," "carrots," and "chicken," and based on the information that "the user likes stir-fries," generates a list of recipes such as "chicken and cabbage stir-fry" and "cabbage and carrot salad," and sends this to the refrigerator terminal. When the user selects a recipe, the emotion engine analyzes the user's facial expressions and voice tone to identify that they are in an excited mood. This information is sent to the server.

[0875] Prompt Sentence Examples

[0876] "The system of this invention recognizes the food in the refrigerator and suggests appropriate cooking recipes. The system has an emotion engine built in, which recognizes the user's emotions and reflects them in the recipe suggestions. If the user is smiling, it will tell us what kind of recipes should be suggested."

[0877] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0878] Step 1:

[0879] Food Recognition

[0880] Device behavior

[0881] The device activates the camera when the refrigerator door closes, scans the interior and takes pictures of food.

[0882] Input: Image data of the inside of the refrigerator acquired by a camera.

[0883] Data processing and calculation: Food information is extracted from image data using a food recognition library. This analysis generates a list of ingredients, such as "cabbage," "carrot," and "chicken."

[0884] Output: An ingredient list containing the generated food information.

[0885] Step 2:

[0886] User emotion recognition

[0887] Device behavior

[0888] The device takes a picture of the user's face with a camera and analyzes the user's emotions using an emotion recognition library.

[0889] Input: User's face image data acquired by camera.

[0890] Data processing and calculation: Using the emotion recognition library, the system analyzes facial expressions and voice tones to identify the user's emotions. Emotional information such as "excited," "neutral," and "sad" is generated.

[0891] Output: The generated emotion information.

[0892] Step 3:

[0893] Sending food and emotional information

[0894] Device behavior

[0895] The terminal transmits the generated food information and emotion information to the server.

[0896] Input: The food information and emotion information generated earlier.

[0897] Data processing and calculation: None in particular.

[0898] Output: Food information and emotion information sent to the server.

[0899] Step 4:

[0900] Recipe Search and Optimization

[0901] Server Operation

[0902] The server receives the food and emotion information and searches the recipe database based on it, optimizing the recipe taking into account past recipe choices, allergy information, and preferences.

[0903] Input: Food information, emotional information, past recipe selection data, allergy information, user preferences.

[0904] Data processing and data calculation: Execute database queries to filter the most suitable recipes from multiple options. Based on emotional information, narrow down the recipes that suit your mood.

[0905] Output: A list of suitable recipes.

[0906] Step 5:

[0907] Sending and displaying recipe lists

[0908] Server and terminal operation

[0909] The server sends the optimized recipe list to the terminal, which displays the recipe list on its display.

[0910] Input: Optimized recipe list.

[0911] Data processing and calculation: None in particular.

[0912] Output: Recipe list sent to the terminal, recipe list displayed on the display.

[0913] Step 6:

[0914] User recipe selection

[0915] User and device behavior

[0916] Users select the desired recipe using the touchscreen or voice recognition interface.

[0917] Input: User selection (touch or voice input).

[0918] Data processing / data calculation: Obtain data for the selected recipe.

[0919] Output: Selected recipe information.

[0920] Step 7:

[0921] Cooking procedure guide

[0922] Device behavior

[0923] Based on the selected recipe, the system provides audio and visual instructions for cooking, and adjusts the tone of the audio guide based on the user's emotional state.

[0924] Input: Selected recipe information, emotion information.

[0925] Data processing and calculation: Audio and visual guide generation. Optimizing the tone of audio guides based on emotional information.

[0926] Output: Audio and visual guide to cooking steps.

[0927] These steps allow users to efficiently utilize the ingredients in their refrigerator and enjoy a personalized cooking experience based on their emotions.

[0928] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0929] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0930] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0931] [Third embodiment]

[0932] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0933] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0934] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0935] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0936] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0937] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0938] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0939] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0940] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0941] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0942] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0943] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0944] The present invention is a system that uses a camera to recognize each food item in a refrigerator, analyzes image data based on the camera, proposes recipes to the user, and explains cooking procedures to the user. Below, the program processing of the system is explained in natural language.

[0945] System configuration

[0946] The system includes the following main components:

[0947] 1. Camera means: A camera for taking pictures of food in the refrigerator.

[0948] 2. Analysis means: A module that analyzes image data acquired by the camera means and generates food information.

[0949] 3. Search means: A server that searches for appropriate cooking recipes based on the generated food information.

[0950] 4. Display means: A refrigerator display that displays the recipes provided by the search means and allows the user to select one.

[0951] 5. Voice guidance means: A function that guides the user through cooking procedures by voice.

[0952] 6. Video guide means: A function that shows the cooking procedure to the user through video.

[0953] Processing flow

[0954] 1. Food Recognition in the Refrigerator

[0955] Terminal handling

[0956] A camera means within the refrigerator scans the interior periodically or each time the refrigerator door is closed, and acquires image data.

[0957] The acquired image data is sent to the analysis means.

[0958] The analysis means uses an object recognition algorithm to identify the photographed food items and generate an ingredient list.

[0959] Examples:

[0960] If there is cabbage, carrots, and chicken in the refrigerator, the camera will take a picture of them, and the analysis module will identify them and generate a list of ingredients such as "cabbage," "carrots," and "chicken." This list will be sent to the server.

[0961] 2. Recipe Suggestion Generation

[0962] Server Processing

[0963] The server receives the food information and queries a recipe database to retrieve recipes based on the food information.

[0964] Search results are optimized by taking into account profile data such as users' previous recipe choices, allergy information, and preferences.

[0965] The server generates a personalized recipe list and sends it to the refrigerator terminal.

[0966] Examples:

[0967] The server searches for "cabbage," "carrot," and "chicken," and based on the information that "the user likes stir-fried dishes," generates a list of recipes for "chicken and cabbage stir-fry" and "cabbage and carrot salad," and sends them to the refrigerator terminal.

[0968] 3. View and select recipes

[0969] Terminal and user handling

[0970] The terminal displays the recipe list on the display.

[0971] Users select recipes using a touchscreen or voice recognition interface.

[0972] Examples:

[0973] The refrigerator terminal's display shows "Stir-fried chicken and cabbage" and "Cabbage and carrot salad," and the user touches "Stir-fried chicken and cabbage" to select it.

[0974] 4. Cooking instructions

[0975] Terminal handling

[0976] The details of the selected recipe will be shown on the display.

[0977] The audio guide means guides the user through specific cooking procedures by voice.

[0978] A video guide means displays a video of each cooking step.

[0979] Examples:

[0980] After selecting "Stir-fried chicken and cabbage," the device will begin providing voice guidance, saying, "First, shred the cabbage," and play a video showing how to cut the cabbage accordingly. Next, it will provide voice and video guidance for each cooking step, such as, "Cut the chicken into bite-sized pieces and heat oil in a frying pan."

[0981] In this way, users can easily use the ingredients in their refrigerator and enjoy cooking effectively. This system plays a very beneficial role in reducing food waste and easing the burden of daily cooking.

[0982] The processing flow will be explained below.

[0983] Step 1:

[0984] Start scanning the inside of the refrigerator

[0985] Every time the refrigerator door is closed, the device activates the AI ​​camera and scans the inside.

[0986] Step 2:

[0987] Acquisition of image data

[0988] The device takes a photo of the inside of the refrigerator with its camera and acquires the image data.

[0989] Step 3:

[0990] Image data analysis

[0991] The device passes the acquired image data to an analysis means, which then runs a food recognition algorithm to generate food information, such as using an object detection algorithm to identify cabbage, carrots, and chicken.

[0992] Step 4:

[0993] Submitting food information

[0994] The device sends the analyzed food information to the server.

[0995] Step 5:

[0996] Recipe Search

[0997] The server searches a recipe database based on the received food information and extracts available recipes.

[0998] Step 6:

[0999] Recipe optimization

[1000] The server selects the best recipes based on the user's previous recipe choices, allergy information, and preferences. For example, if the user has a dairy allergy, dairy-free recipes will be prioritized.

[1001] Step 7:

[1002] Generate a recipe list

[1003] The server generates an optimized recipe list and transmits it to the refrigerator terminal.

[1004] Step 8:

[1005] Displaying the recipe list

[1006] The device displays the recipe list received from the server on the display. For example, "Stir-fried chicken and cabbage" or "Cabbage and carrot salad" are displayed.

[1007] Step 9:

[1008] User recipe selection

[1009] The user selects the desired recipe using the touchscreen or voice recognition interface.

[1010] Step 10:

[1011] View recipe details

[1012] The terminal displays detailed information about the selected recipe on the display.

[1013] Step 11:

[1014] Audio guide for cooking steps

[1015] The device will start a voice prompt and guide the user through each cooking step. For example, it might say, "First, shred the cabbage."

[1016] Step 12:

[1017] Video guide for cooking steps

[1018] The device will then display a visual guide for each cooking step in conjunction with the audio guide, for example showing how to cut cabbage.

[1019] Step 13:

[1020] User cooking execution

[1021] Users follow audio and video guides to cook and create delicious meals.

[1022] Through this processing step, users can efficiently utilize the ingredients in their refrigerator and enjoy cooking simply and effectively.

[1023] Example 1

[1024] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1025] In today's modern diet, efficiently managing the food in the refrigerator and providing recipes that utilize that food is important for reducing food waste and saving time and effort spent deciding what to cook. However, conventional refrigerators lacked these functions, forcing users to manually manage food and search for recipes. This led to problems such as food expiring and an increased burden on daily cooking.

[1026] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1027] In this invention, the server includes an imaging means for recognizing the food inside, an analysis means for analyzing the image information obtained by the imaging means to generate food data, and a database search means for transferring the food data generated by the analysis means and searching for optimal cooking recipes. This makes it possible to automatically and efficiently manage the food inside the refrigerator and suggest appropriate recipes to the user.

[1028] The "imaging means" is a device for photographing the inside of the refrigerator and acquiring image information.

[1029] The "analysis means" is a device or software module that analyzes the image information acquired by the imaging means, identifies the type and amount of food, and generates food data as a result.

[1030] The "database search means" is a device or software module for searching an existing recipe database for optimal cooking recipes based on the food data generated by the analysis means.

[1031] The "display means" is a device for visually displaying to the user the recipes retrieved by the database search means.

[1032] The "voice guidance means" is a device or software module that provides audible guidance to the user on the cooking steps based on the recipe selected by the user.

[1033] The "visual guidance means" is a device or software module that visually guides the user through the cooking steps based on the recipe selected by the user.

[1034] A "tactile interface" is an interface that allows users to use touch controls to input selections and instructions such as recipes.

[1035] A "voice recognition interface" is an interface that recognizes the user's voice input and performs operations according to the instructions.

[1036] This invention relates to a system for efficiently managing food in a refrigerator and suggesting suitable cooking recipes to users. Specifically, the system includes the following main components:

[1037] 1. Imaging means: A camera for taking pictures of food in the refrigerator.

[1038] 2. Analysis means: A software module that analyzes the image information acquired by the imaging means and generates food data.

[1039] 3. Database search means: A server for searching for optimal cooking recipes based on the generated food data.

[1040] 4. Display means: refrigerator display for displaying the searched recipes.

[1041] 5. Voice guidance: A function that provides voice guidance for cooking steps based on a recipe.

[1042] 6. Video guidance: A function that provides video guidance on cooking steps based on a recipe.

[1043] 7. Haptic interface: An interface that allows the user to interact with the device by touch.

[1044] 8. Voice recognition interface: An interface that recognizes the user's voice input and performs operations.

[1045] Hardware and Software Use

[1046] The imaging means is a commercially available camera installed inside the refrigerator. Specifically, existing network cameras or smart cameras can be used. The analysis means and database search means use software running on a server. The analysis means includes a software module that implements a food recognition algorithm (e.g., YOLO, Inception). The database search means can use SQL or NoSQL databases. The audio guidance means and video guidance means use speech synthesis software and video playback software. The haptic interface and speech recognition interface use a touchscreen device and a speech recognition engine (e.g., Google Speech-to-Text), respectively.

[1047] Specific examples

[1048] For example, if you have cabbage, carrots, and chicken in your refrigerator, the system will do the following:

[1049] 1. The device uses the camera inside the refrigerator to take a picture of the inside and sends the image data to the server.

[1050] 2. The server's analysis module analyzes the image data and generates food data such as "cabbage," "carrot," and "chicken."

[1051] 3. The server searches the recipe database based on the generated food data and lists the most suitable recipes, taking into account the user's profile information (past recipe selections, allergy information, preferences).

[1052] 4. The display means displays the recipes "Chicken and Cabbage Stir-fry" and "Cabbage and Carrot Salad." The user uses the touchscreen to select "Chicken and Cabbage Stir-fry."

[1053] 5. Audio and visual guidance means guide the user through the cooking process step by step, providing specific instructions such as "First, shred the cabbage."

[1054] Prompt Sentence Examples

[1055] Below are some example prompts that the system can input to the generative AI model:

[1056] Say you have cabbage, carrots, and chicken in your fridge and your user likes stir-frying. Suggest a suitable recipe and provide step-by-step audio and visual instructions detailing each cooking step.

[1057] This system allows users to efficiently manage the food in their refrigerator, lighten the burden of daily cooking, reduce food waste, and even beginners can enjoy cooking with peace of mind.

[1058] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1059] Step 1:

[1060] Terminal handling

[1061] Input: Image data captured by a camera inside the refrigerator.

[1062] How it works: The device activates the camera periodically or whenever the refrigerator door is closed and takes a picture of the inside of the refrigerator.

[1063] Data processing: The captured image data is sent directly to the analysis means.

[1064] Output: Image data sent to analysis means.

[1065] Step 2:

[1066] Server Processing

[1067] Input: Image data sent from the device.

[1068] Operation: The server's analysis means analyzes the received image data using an object recognition algorithm (e.g., YOLO, Inception).

[1069] Data calculation: The algorithm identifies the type and location of food in the image and lists it as food data.

[1070] Output: A list of food data (e.g. "cabbage", "carrot", "chicken").

[1071] Step 3:

[1072] Server Processing

[1073] Input: The generated food data list.

[1074] Operation: The server uses the database search means to query the recipe database based on the food data list.

[1075] Data Calculation: Recipes listed as query results are optimized taking into account the user's profile information (previous recipe choices, allergies, preferences).

[1076] Output: A personalized recipe list.

[1077] Step 4:

[1078] Terminal handling

[1079] Input: Recipe list sent from the server.

[1080] Operation: A recipe list is displayed by the display means of the terminal.

[1081] Output: A visually displayed list of recipes (e.g. "Chicken and Cabbage Stir-fry", "Cabbage and Carrot Salad").

[1082] Step 5:

[1083] User Action

[1084] Input: Recipe list shown on the display.

[1085] How it works: The user selects the desired recipe using the touchscreen or voice recognition interface.

[1086] Output: The selected recipe (e.g. "Stir-fried chicken and cabbage").

[1087] Step 6:

[1088] Terminal handling

[1089] Input: The recipe selected by the user.

[1090] Operation: The audio and visual guidance means of the terminal guides the user through the cooking procedure step by step using audio and visual means.

[1091] Specific steps: "First, shred the cabbage," the voice instructions begin, and a video showing the steps for shredding the cabbage plays on the screen. Next, the instructions continue: "Cut the chicken into bite-sized pieces and heat oil in a frying pan."

[1092] Output: Audio and visual instructions for cooking procedures.

[1093] Through the above processing steps, users can efficiently manage the food in their refrigerator, easily select appropriate recipes, and proceed with cooking, thereby reducing food waste and easing the burden of cooking.

[1094] (Application example 1)

[1095] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1096] In modern life, there is a demand for effective use of food in the refrigerator and reduction of food waste. Furthermore, when shopping in a physical store, users can plan their shopping and cooking efficiently if they are provided with cooking method and recipe suggestions based on the food they are purchasing. However, previous systems have only been able to suggest recipes and provide cooking instructions based on the ingredients in the refrigerator, and have not been able to provide support for shopping in a physical store. Therefore, there is a need for a system that can seamlessly support the entire process from food selection to cooking in a physical store.

[1097] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1098] In this invention, the server includes image acquisition means for recognizing foods in the refrigerator, data analysis means for analyzing image data acquired by the image acquisition means to generate food information, information retrieval means for collecting the food information generated by the data analysis means and searching for appropriate recipes, display means for displaying the recipes retrieved by the information retrieval means, audio guidance means and video guidance means for supporting the user in cooking based on the recipe display by the display means, a visual device installed in the store for recognizing product selections by customers, product recognition means for collecting the product information recognized by the visual device, and control means for transmitting the product information generated by the product recognition means to the information retrieval means and displaying the recipes recommended by the information retrieval means on the display means. This makes it possible not only to recognize ingredients in the refrigerator but also to provide seamless support from food selection to cooking in a physical store.

[1099] An "image capture means" is a device or system for capturing an image of an object.

[1100] "Data analysis means" refers to a system or algorithm that processes and analyzes acquired image data to generate information about the target object.

[1101] The "information retrieval means" is a system for retrieving related information and content from an appropriate database based on the generated information.

[1102] "Display means" refers to a device or system for providing the searched information to the user in a visually recognizable form.

[1103] "Voice guidance means" refers to a system that provides guidance and instructions to users using voice.

[1104] A "visual guide" is a system that uses video to provide guides and instructions to users.

[1105] A "visual device" is a device or system used to recognize items selected by a customer in a physical store.

[1106] The "product recognition means" is a system for analyzing information acquired through a visual device and generating information about the recognized product.

[1107] A "control means" is a system or component that has the role of managing and coordinating the operations of various means and modules in an integrated manner.

[1108] The system according to the present invention uses a wide range of hardware and software to provide a consistent shopping and cooking assistance. The system includes the following main components:

[1109] 1. Image acquisition method

[1110] This includes cameras installed in the device's refrigerator or visual devices installed in physical stores, which take photos of food or products periodically or in response to user actions.

[1111] 2. Data analysis methods

[1112] Using a generative AI model such as TensorFlow, the captured image data is analyzed to identify the target food or product. This analysis process is performed on the server and generates the identified food information.

[1113] 3. Information search methods

[1114] The server uses the collected food information to search a recipe database for suitable recipes, optimizing the search results based on the user's previous choices, allergy information, and preferences.

[1115] 4. Display means

[1116] The recognized food information and searched recipes are displayed on visual devices such as refrigerator displays or smart glasses, which are equipped with tactile and voice input interfaces for users to select cooking methods.

[1117] 5. Audio and visual guidance means

[1118] Based on the selected recipe, the audio guide means provides audio instructions on specific cooking steps, and the visual guide means displays images of each cooking step to provide visual support to the user.

[1119] Examples:

[1120] When a user selects cabbage, carrots, and chicken at the supermarket, the visual device recognizes this and sends the information to the server. Based on this, the server searches for recipes such as "stir-fried chicken and cabbage" or "cabbage and carrot salad" and displays them on the smart glasses. If the user selects "stir-fried chicken and cabbage," the smart glasses will start a voice prompt saying, "First, shred the cabbage," and play a video showing how to cut the cabbage accordingly.

[1121] Example prompt sentence:

[1122] "Can you recommend a recipe for cabbage, carrots, and chicken?"

[1123] This system allows users to receive assistance with every step of the process, from selecting food in a physical store to cooking, not just with the ingredients in their refrigerator, reducing food waste and making cooking more efficient.

[1124] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1125] Step 1: Image acquisition

[1126] When a user opens or closes the refrigerator door, the image capture means of the device captures a photo of the food items inside the refrigerator. Also, when a user scans an item using smart glasses in a physical store, the visual device recognizes the item. Product image data from inside the refrigerator or store is acquired as input and sent to the analysis means. The output is the acquired raw image data.

[1127] Step 2: Image analysis

[1128] The server receives the image data sent from the image acquisition means and analyzes the data using a generative AI model such as TensorFlow. As input, it receives the acquired image data and identifies food or product information (e.g., "cabbage," "carrot," "chicken," etc.). The output is a list of recognized foods or products.

[1129] Step 3: Information search

[1130] The server collects food information generated by image analysis and searches a recipe database for suitable recipes based on that information. As input, it receives a list of recognized foods and identifies the best recipes based on the user's past choices, allergy information, and preferences. The output is a list of optimized recipes.

[1131] Step 4: View the recipe

[1132] The user receives the recipe information generated by the server in the refrigerator or in the brick-and-mortar store. Through the display means of the terminal, these recipes are visualized and displayed on the refrigerator display or smart glasses. As input, the searched recipe data is received and sent to the visual device. The output is a recipe list that the user can visually check.

[1133] Step 5: Recipe Selection

[1134] The user selects a preferred recipe from a displayed list of recipes. The user selects a recipe using a tactile or voice input interface. The system receives as input a visualized list of recipes and the user's selection action. The output is the specific recipe selected by the user.

[1135] Step 6: Cooking instructions

[1136] The terminal displays the details of the selected recipe and provides the user with cooking instructions using audio and video guidance means. As input, it receives detailed data of the selected recipe and plays audio and video guidance based on the instructions. As output, it provides audio and video guidance according to the cooking instructions.

[1137] In this way, by utilizing the system, users can reduce food waste and shop and cook more efficiently.

[1138] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1139] This invention combines a system that recognizes food in the refrigerator and suggests appropriate recipes to the user with an emotion engine that recognizes the user's emotions. Below, the system's program processing is explained in natural language with concrete examples.

[1140] System configuration

[1141] The system includes the following main components:

[1142] 1. Camera means: A camera for taking pictures of food in the refrigerator.

[1143] 2. Analysis means: A module that analyzes image data acquired by the camera means and generates food information.

[1144] 3. Search means: A server that searches for appropriate cooking recipes based on the generated food information.

[1145] 4. Display means: A refrigerator display that displays the recipes provided by the search means and allows the user to select one.

[1146] 5. Voice guidance means: A function that guides the user through cooking procedures by voice.

[1147] 6. Video guide means: A function that shows the cooking procedure to the user through video.

[1148] 7. Emotion engine: An engine that recognizes the user's emotions and reflects them in the system's operations.

[1149] Processing flow

[1150] 1. Food Recognition in the Refrigerator

[1151] Terminal handling

[1152] Every time the refrigerator door is closed, the device activates the AI ​​camera to scan the inside.

[1153] The image data acquired by the camera means is passed to the analysis means.

[1154] The analysis means generates food information using an object recognition algorithm, for example, identifying cabbage, carrots, and chicken.

[1155] Examples:

[1156] If there is cabbage, carrots, and chicken in the refrigerator, the camera will take a picture of them, and the analysis module will generate a list of ingredients such as "cabbage," "carrots," and "chicken." This list will be sent to the server.

[1157] 2. Recipe Suggestion Generation

[1158] Server Processing

[1159] The server queries a recipe database based on the received food information to search for available recipes.

[1160] The server selects the best recipes for you, taking into account your previous recipe choices, allergy information, and preferences. For example, if you have a dairy allergy, it will suggest dairy-free recipes.

[1161] Examples:

[1162] The server searches for recipes using "cabbage," "carrots," and "chicken," and based on the information that "the user likes stir-fried dishes," generates a list of recipes for "chicken and cabbage stir-fry" and "cabbage and carrot salad," and sends these to the refrigerator terminal.

[1163] 3. View and select recipes

[1164] Terminal and user handling

[1165] The device displays the recipe list on the display.

[1166] The user selects the desired recipe using the touchscreen or voice recognition interface.

[1167] Examples:

[1168] The refrigerator terminal's display shows "Stir-fried chicken and cabbage" and "Cabbage and carrot salad," and the user touches "Stir-fried chicken and cabbage" to select it.

[1169] 4. User Emotion Recognition

[1170] Emotion engine processing

[1171] The emotion engine analyzes the user's facial expressions, vocal tone, and physical movements to identify emotions.

[1172] The emotion engine transmits the identified emotion information to a server.

[1173] Examples:

[1174] When a user selects a recipe, the emotion engine analyzes the user's facial expressions and vocal tone to identify whether they are in a good mood, and this information is sent to the server.

[1175] 5. Cooking instructions

[1176] Terminal handling

[1177] The server optimizes recipe suggestions based on emotional information and sends them back to the device.

[1178] The terminal displays detailed information about the selected recipe on the display, and the audio guide means provides audio guidance of the cooking procedure.

[1179] A video guide means displays a video of each cooking step.

[1180] Examples:

[1181] After "Stir-fried chicken and cabbage" is selected, the device starts a voice prompt saying, "First, shred the cabbage," and plays a video showing how to cut the cabbage. Based on the results of the emotion engine, the voice guide provides instructions in a bright tone, lifting the user's spirits.

[1182] This system allows users to efficiently utilize the ingredients in their refrigerator while enjoying a personalized cooking experience that adapts to their emotions.

[1183] The processing flow will be explained below.

[1184] Step 1:

[1185] Start scanning the inside of the refrigerator

[1186] Every time the refrigerator door is closed, the device activates the AI ​​camera and scans the inside.

[1187] Step 2:

[1188] Acquisition of image data

[1189] The device takes a photo of the inside of the refrigerator with its camera and acquires the image data.

[1190] Step 3:

[1191] Image data analysis

[1192] The device passes the acquired image data to an analysis means, which then runs a food recognition algorithm to generate food information, such as using an object detection algorithm to identify cabbage, carrots, and chicken.

[1193] Step 4:

[1194] Submitting food information

[1195] The device sends the analyzed food information to the server.

[1196] Step 5:

[1197] Recipe Search

[1198] The server searches a recipe database based on the received food information and extracts available recipes.

[1199] Step 6:

[1200] Recipe optimization

[1201] The server selects the best recipes based on the user's previous recipe choices, allergy information, and preferences. For example, if the user has a dairy allergy, dairy-free recipes will be prioritized.

[1202] Step 7:

[1203] Generate a recipe list

[1204] The server generates an optimized recipe list and transmits it to the refrigerator terminal.

[1205] Step 8:

[1206] Displaying the recipe list

[1207] The device displays the recipe list received from the server on the display. For example, "Stir-fried chicken and cabbage" or "Cabbage and carrot salad" are displayed.

[1208] Step 9:

[1209] User recipe selection

[1210] The user selects the desired recipe using the touchscreen or voice recognition interface.

[1211] Step 10:

[1212] User emotion recognition

[1213] The device's emotion engine analyzes the user's facial expressions, voice tone, and physical movements to identify emotions. It also analyzes the user's facial expressions and voice tone when selecting recipes to identify emotions.

[1214] Step 11:

[1215] Sending emotional information

[1216] The terminal transmits the identified emotion information to the server.

[1217] Step 12:

[1218] Emotion-based recipe re-suggestion

[1219] The server takes emotional information into account and optimizes recipe suggestions to match the user's emotions. For example, if the user is feeling depressed, it will suggest easy and hassle-free recipes.

[1220] Step 13:

[1221] View recipe details

[1222] The terminal displays detailed information about the selected recipe on the display.

[1223] Step 14:

[1224] Audio guide for cooking steps

[1225] The device will start a voice prompt and guide the user through each cooking step. For example, it might say, "First, shred the cabbage."

[1226] Step 15:

[1227] Video guide for cooking steps

[1228] The device will then display a visual guide for each cooking step in conjunction with the audio guide, for example showing how to cut cabbage.

[1229] Step 16:

[1230] User cooking execution

[1231] Users follow audio and video guides to cook and create delicious meals.

[1232] Through this process, users can efficiently utilize the ingredients in their refrigerators, enjoy cooking simply and effectively, and optimize their cooking experience based on their emotions, providing a more personalized service.

[1233] Example 2

[1234] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1235] Conventional refrigerator systems have limited functionality for efficiently recognizing ingredients in the refrigerator and providing users with recipe suggestions based on that information. It is also difficult to provide personalized suggestions that take into account the user's emotions and individual preferences. Therefore, there is a demand for a system that allows users to easily find the perfect dish and enjoy cooking. In particular, there is a lack of functionality that recognizes the user's emotions in real time during the cooking process and provides guidance accordingly.

[1236] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1237] In this invention, the server includes an imaging means for recognizing food in the refrigerator, an analysis means for analyzing the image data obtained by the imaging means to generate ingredient information, a search means for collecting the ingredient information generated by the analysis means and searching for appropriate cooking procedures, a display means for displaying the cooking procedures searched by the search means, an audio guide means and a visual guide means for assisting the user in cooking based on the cooking procedure display by the display means, and an emotion recognition means for recognizing the user's emotions and reflecting them in the operation content. This makes it possible to effectively use ingredients in the refrigerator and provide personalized recipe suggestions and cooking support according to the user's emotions and preferences.

[1238] The "imaging means" is a device that acquires image data to identify ingredients in the refrigerator.

[1239] The "analysis means" is a device or module that analyzes the image data acquired by the imaging means and identifies the type and amount of food.

[1240] The "search means" is a device or software that searches a database for appropriate cooking procedures and recipes using the ingredient information generated by the analysis means.

[1241] The "display means" is a device for visually showing the cooking steps and recipes found by the search means to the user, and an example of this is a display.

[1242] "Audio guidance means" refers to a device or software that provides audio guidance to the user on cooking methods and procedures.

[1243] A "visual guide means" is a device or software that visually explains cooking methods and procedures to the user.

[1244] "Emotion recognition means" refers to a device or software that analyzes a user's facial expressions, voice, and physical movements to recognize their emotions.

[1245] The present invention combines a system that recognizes food in a refrigerator and suggests suitable cooking recipes to the user with technology that recognizes user emotions. The system includes the following main components: an imaging means, an analysis means, a search means, a display means, an audio guide means, a visual guide means, and an emotion recognition means.

[1246] Imaging means

[1247] The device is equipped with a camera that recognizes the food inside the refrigerator. The AI ​​camera activates every time the refrigerator door is closed and scans the ingredients inside.

[1248] Analysis means

[1249] The acquired image data is analyzed using an object recognition algorithm (e.g., YOLOv4) to generate food information. The analysis means identifies ingredients such as cabbage, carrots, and chicken, and sends the information to a server.

[1250] Search methods

[1251] The server queries the recipe database based on the received food information to find available recipes, and then selects the most suitable recipe taking into account the user's past recipe choices, allergy information, and preferences (e.g., dairy allergy).

[1252] Display means

[1253] The terminal displays the recipe list sent from the server on a display, which includes a touchscreen interface and a voice recognition interface (e.g., Google Assistant) that the user uses to select a desired recipe.

[1254] Acoustic and visual guide means

[1255] Based on the selected recipe, the audio guide module provides audio instructions for the cooking steps, and the visual guide module displays videos of each cooking step on the display. The audio guide module uses Amazon Polly, and the visual guide module plays videos using the YouTube API.

[1256] emotion recognition means

[1257] The emotion engine captures the user's facial expressions with a camera and analyzes them using the Facial Action Coding System. It also uses voice analysis technology (e.g., DeepSpeech) to analyze the tone of the voice and identify the user's emotions. The identified emotion information is sent to the server and used to optimize recipe suggestions.

[1258] Specific examples

[1259] For example, if there is cabbage, carrots, and chicken in the refrigerator, the imaging means takes a picture of them, and the analysis means generates an ingredient list of "cabbage," "carrots," and "chicken." This list is sent to the server, which searches for available recipes. The server then sends recipes such as "stir-fried chicken and cabbage" and "cabbage and carrot salad" to the terminal, which displays them on the display. When the user selects "stir-fried chicken and cabbage," the audio guide means begins guiding the user by saying, "First, shred the cabbage," and the visual guide means plays a video on how to cut the cabbage. At the same time, the emotion recognition means analyzes the user's facial expressions and voice tone to provide personalized guidance tailored to their emotions.

[1260] Prompt Sentence Examples

[1261] An example prompt for a generative AI model is:

[1262] "Please explain the process by which an AI system recognizes that there is cabbage, carrots, and chicken in the refrigerator and suggests appropriate recipes to the user. Also, please include specific actions and techniques to identify the user's emotions and provide guidance tailored to those emotions."

[1263] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1264] Step 1:

[1265] Scanning for ingredient recognition (processing on device)

[1266] Input: Status inside the refrigerator, refrigerator door opening / closing information.

[1267] How it works: Every time the refrigerator door is closed, the device detects this with a sensor and activates the AI ​​camera, which then captures images of the inside of the refrigerator and generates image data.

[1268] Output: Image data of the inside of the refrigerator.

[1269] Step 2:

[1270] Image data analysis (terminal processing)

[1271] Input: Image data of the inside of the refrigerator.

[1272] How it works: The device passes the image data to the analysis means, which uses an image analysis algorithm (e.g., YOLOv4) to identify the type and location of the food.

[1273] Output: A list of identified ingredients (e.g., cabbage, carrot, chicken).

[1274] Step 3:

[1275] Sending ingredient information and searching for recipes (server processing)

[1276] Input: Ingredient list sent from the device.

[1277] How it works: The server receives the ingredient list, queries the recipe database, and selects the best recipe, taking into account the user's previous recipe choices, allergy information, and preferences.

[1278] Output: A list of available recipes (e.g., Chicken and Cabbage Stir-fry, Cabbage and Carrot Salad).

[1279] Step 4:

[1280] Displaying recipes (terminal processing and user processing)

[1281] Input: Recipe list sent from the server.

[1282] How it works: The device displays a list of recipes on the display. The user selects the desired recipe using the touchscreen or voice recognition interface.

[1283] Output: The recipe selected by the user.

[1284] Step 5:

[1285] Emotion recognition (emotion engine processing)

[1286] Input: User facial expressions, voice tone, and physical movements.

[1287] How it works: The emotion recognition unit captures the user's facial expressions with a camera and analyzes them using the Facial Action Coding System. It also uses voice analysis technology to analyze the tone of the voice and identify the user's emotions.

[1288] Output: Identified user's sentiment information.

[1289] Step 6:

[1290] Optimization and presentation of cooking procedures (terminal processing)

[1291] Input: Selected recipe and emotion information sent from the server.

[1292] Operation: The server optimizes recipe suggestions based on emotion information and resends them to the device. The device then displays detailed cooking instructions on the screen, uses audio guidance (e.g., Amazon Polly) to provide audible instructions, and plays videos using visual guidance (e.g., YouTube API).

[1293] Output: Personalized cooking instructions and guide.

[1294] Examples:

[1295] Step 1 through Step 6:

[1296] If there is cabbage, carrots, or chicken in the refrigerator, the AI ​​camera will take a photo of it, and the device will pass the image data to the analysis means. The analysis means will identify the cabbage, carrots, and chicken, and send the ingredient list to the server. The server will query the recipe database based on the ingredient list and user information, generate a list of recipes for "chicken and cabbage stir-fry" and "cabbage and carrot salad," and send this to the device. When the user selects "chicken and cabbage stir-fry" on the touchscreen, the emotion engine will analyze the user's facial expressions and voice tone, guide them through the cooking steps in a bright tone, and play a video on how to cut cabbage on the display.

[1297] (Application example 2)

[1298] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1299] Conventional recipe suggestion systems have the ability to suggest recipes using ingredients in the refrigerator, but they are unable to flexibly suggest recipes based on the user's emotions. For example, it is difficult to suggest dishes that suit the user's mood when they are feeling depressed or excited. For this reason, there is a need for personalized recipe suggestions that take the user's psychological state into account.

[1300] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a camera means for recognizing food in the refrigerator, an analysis means for analyzing image data obtained by the camera means to generate food information, a search means for collecting the food information generated by the analysis means and searching for appropriate cooking recipes, a display means for displaying the recipes searched by the search means, and an emotion engine means for recognizing the user's emotions and reflecting them in recipe suggestions. This makes it possible to suggest recipes that take the user's emotional state into consideration.

[1301] "Camera means" refers to a photographic device used to recognize food items in the refrigerator.

[1302] The "analysis means" is a module that analyzes image data obtained by the camera means and generates food information.

[1303] The "search means" is a device or program that searches for appropriate cooking recipes based on the food information generated by the analysis means.

[1304] The "display means" is a device or interface for visually presenting the recipes retrieved by the search means to the user.

[1305] The "voice guide means" is a voice guide function that assists the user in cooking based on the recipe displayed by the display means.

[1306] The "visual guide means" is a function that presents the user with a visual guide to cooking procedures based on the recipe displayed by the display means.

[1307] The "emotion engine means" is an engine that recognizes the user's emotions and reflects them in the cooking recipe suggestions.

[1308] A "touch screen interface" is a tactile input device that allows a user to select recipes.

[1309] A "voice recognition interface" is a device or program that recognizes and inputs user voice instructions.

[1310] This invention combines a system that recognizes food in a refrigerator and suggests suitable cooking recipes to the user with an emotion engine that recognizes the user's emotions. Specific embodiments of the system will be described below.

[1311] System configuration

[1312] The system includes the following main components:

[1313] 1. Camera means: A camera for taking pictures of food in the refrigerator. This can be a camera built into a smartphone.

[1314] 2. Analysis means: A module that analyzes image data acquired by the camera means and generates food information. It uses a food recognition library.

[1315] 3. Search method: A server that searches for appropriate cooking recipes based on the generated food information. It uses the recipe search service API.

[1316] 4. Display means: An interface that displays recipes provided by the search means and allows the user to select one. An example is using the display of a smartphone.

[1317] 5. Voice guidance means: A function that guides the user through cooking procedures by voice. Use the voice guidance service.

[1318] 6. Video guide: A function that shows the cooking procedure to the user through video. It also has a video playback function.

[1319] 7. Emotion engine means: An engine that recognizes the user's emotions and reflects them in the system's operations. It uses an emotion recognition library.

[1320] System Operation

[1321] Food Recognition and Emotion Recognition

[1322] The device activates the camera when the refrigerator door is closed or based on a specific trigger, scans the interior, and photographs the food. The acquired image data is passed to an analysis means, which generates food information. A food recognition library is used to identify, for example, cabbage, carrots, chicken, etc. The device then analyzes the user's face to identify their emotions. The emotion recognition library analyzes the user's facial expressions and voice tone to generate emotional information.

[1323] Generating recipe suggestions

[1324] The server searches a recipe database based on the food information generated by the analysis means to query available recipes. It also optimizes recipes by taking into account the user's past recipe choices, allergy information, and preferences. It also takes into account emotional information to suggest recipes that suit the user's mood. For example, if the user is in high spirits, it will suggest a simple and fun recipe such as "stir-fried chicken and cabbage."

[1325] Viewing and Selecting Recipes

[1326] The provided recipes are displayed on the device's display, and the user can select them via the touchscreen or voice recognition interface. Based on the selected recipe, cooking instructions are provided through audio and visual guidance. The audio guidance is provided in a tone that corresponds to the user's emotions; for example, if the user is in a happy mood, a cheerful tone will be used to guide the user.

[1327] Specific examples

[1328] If there is cabbage, carrots, and chicken in the refrigerator, the camera takes a picture of them, and the analysis means generates a list of ingredients such as "cabbage," "carrots," and "chicken." This list is sent to the server. The server searches for recipes using "cabbage," "carrots," and "chicken," and based on the information that "the user likes stir-fries," generates a list of recipes such as "chicken and cabbage stir-fry" and "cabbage and carrot salad," and sends this to the refrigerator terminal. When the user selects a recipe, the emotion engine analyzes the user's facial expressions and voice tone to identify that they are in an excited mood. This information is sent to the server.

[1329] Prompt Sentence Examples

[1330] "The system of this invention recognizes the food in the refrigerator and suggests appropriate cooking recipes. The system has an emotion engine built in, which recognizes the user's emotions and reflects them in the recipe suggestions. If the user is smiling, it will tell us what kind of recipes should be suggested."

[1331] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1332] Step 1:

[1333] Food Recognition

[1334] Device behavior

[1335] The device activates the camera when the refrigerator door closes, scans the interior and takes pictures of food.

[1336] Input: Image data of the inside of the refrigerator acquired by a camera.

[1337] Data processing and calculation: Food information is extracted from image data using a food recognition library. This analysis generates a list of ingredients, such as "cabbage," "carrot," and "chicken."

[1338] Output: An ingredient list containing the generated food information.

[1339] Step 2:

[1340] User emotion recognition

[1341] Device behavior

[1342] The device takes a picture of the user's face with a camera and analyzes the user's emotions using an emotion recognition library.

[1343] Input: User's face image data acquired by camera.

[1344] Data processing and calculation: Using the emotion recognition library, the system analyzes facial expressions and voice tones to identify the user's emotions. Emotional information such as "excited," "neutral," and "sad" is generated.

[1345] Output: The generated emotion information.

[1346] Step 3:

[1347] Sending food and emotional information

[1348] Device behavior

[1349] The terminal transmits the generated food information and emotion information to the server.

[1350] Input: The food information and emotion information generated earlier.

[1351] Data processing and calculation: None in particular.

[1352] Output: Food information and emotion information sent to the server.

[1353] Step 4:

[1354] Recipe Search and Optimization

[1355] Server Operation

[1356] The server receives the food and emotion information and searches the recipe database based on it, optimizing the recipe taking into account past recipe choices, allergy information, and preferences.

[1357] Input: Food information, emotional information, past recipe selection data, allergy information, user preferences.

[1358] Data processing and data calculation: Execute database queries to filter the most suitable recipes from multiple options. Based on emotional information, narrow down the recipes that suit your mood.

[1359] Output: A list of suitable recipes.

[1360] Step 5:

[1361] Sending and displaying recipe lists

[1362] Server and terminal operation

[1363] The server sends the optimized recipe list to the terminal, which displays the recipe list on its display.

[1364] Input: Optimized recipe list.

[1365] Data processing and calculation: None in particular.

[1366] Output: Recipe list sent to the terminal, recipe list displayed on the display.

[1367] Step 6:

[1368] User recipe selection

[1369] User and device behavior

[1370] Users select the desired recipe using the touchscreen or voice recognition interface.

[1371] Input: User selection (touch or voice input).

[1372] Data processing / data calculation: Obtain data for the selected recipe.

[1373] Output: Selected recipe information.

[1374] Step 7:

[1375] Cooking procedure guide

[1376] Device behavior

[1377] Based on the selected recipe, the system provides audio and visual instructions for cooking, and adjusts the tone of the audio guide based on the user's emotional state.

[1378] Input: Selected recipe information, emotion information.

[1379] Data processing and calculation: Audio and visual guide generation. Optimizing the tone of audio guides based on emotional information.

[1380] Output: Audio and visual guide to cooking steps.

[1381] These steps allow users to efficiently utilize the ingredients in their refrigerator and enjoy a personalized cooking experience based on their emotions.

[1382] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1383] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1384] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1385] [Fourth embodiment]

[1386] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1387] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1388] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1389] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1390] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1391] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1392] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1393] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1394] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1395] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1396] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1397] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1398] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1399] The present invention is a system that uses a camera to recognize each food item in a refrigerator, analyzes image data based on the camera, proposes recipes to the user, and explains cooking procedures to the user. Below, the program processing of the system is explained in natural language.

[1400] System configuration

[1401] The system includes the following main components:

[1402] 1. Camera means: A camera for taking pictures of food in the refrigerator.

[1403] 2. Analysis means: A module that analyzes image data acquired by the camera means and generates food information.

[1404] 3. Search means: A server that searches for appropriate cooking recipes based on the generated food information.

[1405] 4. Display means: A refrigerator display that displays the recipes provided by the search means and allows the user to select one.

[1406] 5. Voice guidance means: A function that guides the user through cooking procedures by voice.

[1407] 6. Video guide means: A function that shows the cooking procedure to the user through video.

[1408] Processing flow

[1409] 1. Food Recognition in the Refrigerator

[1410] Terminal handling

[1411] A camera means within the refrigerator scans the interior periodically or each time the refrigerator door is closed, and acquires image data.

[1412] The acquired image data is sent to the analysis means.

[1413] The analysis means uses an object recognition algorithm to identify the photographed food items and generate an ingredient list.

[1414] Examples:

[1415] If there is cabbage, carrots, and chicken in the refrigerator, the camera will take a picture of them, and the analysis module will identify them and generate a list of ingredients such as "cabbage," "carrots," and "chicken." This list will be sent to the server.

[1416] 2. Recipe Suggestion Generation

[1417] Server Processing

[1418] The server receives the food information and queries a recipe database to retrieve recipes based on the food information.

[1419] Search results are optimized by taking into account profile data such as users' previous recipe choices, allergy information, and preferences.

[1420] The server generates a personalized recipe list and sends it to the refrigerator terminal.

[1421] Examples:

[1422] The server searches for "cabbage," "carrot," and "chicken," and based on the information that "the user likes stir-fried dishes," generates a list of recipes for "chicken and cabbage stir-fry" and "cabbage and carrot salad," and sends them to the refrigerator terminal.

[1423] 3. View and select recipes

[1424] Terminal and user handling

[1425] The terminal displays the recipe list on the display.

[1426] Users select recipes using a touchscreen or voice recognition interface.

[1427] Examples:

[1428] The refrigerator terminal's display shows "Stir-fried chicken and cabbage" and "Cabbage and carrot salad," and the user touches "Stir-fried chicken and cabbage" to select it.

[1429] 4. Cooking instructions

[1430] Terminal handling

[1431] The details of the selected recipe will be shown on the display.

[1432] The audio guide means guides the user through specific cooking procedures by voice.

[1433] A video guide means displays a video of each cooking step.

[1434] Examples:

[1435] After selecting "Stir-fried chicken and cabbage," the device will begin providing voice guidance, saying, "First, shred the cabbage," and play a video showing how to cut the cabbage accordingly. Next, it will provide voice and video guidance for each cooking step, such as, "Cut the chicken into bite-sized pieces and heat oil in a frying pan."

[1436] In this way, users can easily use the ingredients in their refrigerator and enjoy cooking effectively. This system plays a very beneficial role in reducing food waste and easing the burden of daily cooking.

[1437] The processing flow will be explained below.

[1438] Step 1:

[1439] Start scanning the inside of the refrigerator

[1440] Every time the refrigerator door is closed, the device activates the AI ​​camera and scans the inside.

[1441] Step 2:

[1442] Acquisition of image data

[1443] The device takes a photo of the inside of the refrigerator with its camera and acquires the image data.

[1444] Step 3:

[1445] Image data analysis

[1446] The device passes the acquired image data to an analysis means, which then runs a food recognition algorithm to generate food information, such as using an object detection algorithm to identify cabbage, carrots, and chicken.

[1447] Step 4:

[1448] Submitting food information

[1449] The device sends the analyzed food information to the server.

[1450] Step 5:

[1451] Recipe Search

[1452] The server searches a recipe database based on the received food information and extracts available recipes.

[1453] Step 6:

[1454] Recipe optimization

[1455] The server selects the best recipes based on the user's previous recipe choices, allergy information, and preferences. For example, if the user has a dairy allergy, dairy-free recipes will be prioritized.

[1456] Step 7:

[1457] Generate a recipe list

[1458] The server generates an optimized recipe list and transmits it to the refrigerator terminal.

[1459] Step 8:

[1460] Displaying the recipe list

[1461] The device displays the recipe list received from the server on the display. For example, "Stir-fried chicken and cabbage" or "Cabbage and carrot salad" are displayed.

[1462] Step 9:

[1463] User recipe selection

[1464] The user selects the desired recipe using the touchscreen or voice recognition interface.

[1465] Step 10:

[1466] View recipe details

[1467] The terminal displays detailed information about the selected recipe on the display.

[1468] Step 11:

[1469] Audio guide for cooking steps

[1470] The device will start a voice prompt and guide the user through each cooking step. For example, it might say, "First, shred the cabbage."

[1471] Step 12:

[1472] Video guide for cooking steps

[1473] The device will then display a visual guide for each cooking step in conjunction with the audio guide, for example showing how to cut cabbage.

[1474] Step 13:

[1475] User cooking execution

[1476] Users follow audio and video guides to cook and create delicious meals.

[1477] Through this processing step, users can efficiently utilize the ingredients in their refrigerator and enjoy cooking simply and effectively.

[1478] Example 1

[1479] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1480] In today's modern diet, efficiently managing the food in the refrigerator and providing recipes that utilize that food is important for reducing food waste and saving time and effort spent deciding what to cook. However, conventional refrigerators lacked these functions, forcing users to manually manage food and search for recipes. This led to problems such as food expiring and an increased burden on daily cooking.

[1481] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1482] In this invention, the server includes an imaging means for recognizing the food inside, an analysis means for analyzing the image information obtained by the imaging means to generate food data, and a database search means for transferring the food data generated by the analysis means and searching for optimal cooking recipes. This makes it possible to automatically and efficiently manage the food inside the refrigerator and suggest appropriate recipes to the user.

[1483] The "imaging means" is a device for photographing the inside of the refrigerator and acquiring image information.

[1484] The "analysis means" is a device or software module that analyzes the image information acquired by the imaging means, identifies the type and amount of food, and generates food data as a result.

[1485] The "database search means" is a device or software module for searching an existing recipe database for optimal cooking recipes based on the food data generated by the analysis means.

[1486] The "display means" is a device for visually displaying to the user the recipes retrieved by the database search means.

[1487] The "voice guidance means" is a device or software module that provides audible guidance to the user on the cooking steps based on the recipe selected by the user.

[1488] The "visual guidance means" is a device or software module that visually guides the user through the cooking steps based on the recipe selected by the user.

[1489] A "tactile interface" is an interface that allows users to use touch controls to input selections and instructions such as recipes.

[1490] A "voice recognition interface" is an interface that recognizes the user's voice input and performs operations according to the instructions.

[1491] This invention relates to a system for efficiently managing food in a refrigerator and suggesting suitable cooking recipes to users. Specifically, the system includes the following main components:

[1492] 1. Imaging means: A camera for taking pictures of food in the refrigerator.

[1493] 2. Analysis means: A software module that analyzes the image information acquired by the imaging means and generates food data.

[1494] 3. Database search means: A server for searching for optimal cooking recipes based on the generated food data.

[1495] 4. Display means: refrigerator display for displaying the searched recipes.

[1496] 5. Voice guidance: A function that provides voice guidance for cooking steps based on a recipe.

[1497] 6. Video guidance: A function that provides video guidance on cooking steps based on a recipe.

[1498] 7. Haptic interface: An interface that allows the user to interact with the device by touch.

[1499] 8. Voice recognition interface: An interface that recognizes the user's voice input and performs operations.

[1500] Hardware and Software Use

[1501] The imaging means is a commercially available camera installed inside the refrigerator. Specifically, existing network cameras or smart cameras can be used. The analysis means and database search means use software running on a server. The analysis means includes a software module that implements a food recognition algorithm (e.g., YOLO, Inception). The database search means can use SQL or NoSQL databases. The audio guidance means and video guidance means use speech synthesis software and video playback software. The haptic interface and speech recognition interface use a touchscreen device and a speech recognition engine (e.g., Google Speech-to-Text), respectively.

[1502] Specific examples

[1503] For example, if you have cabbage, carrots, and chicken in your refrigerator, the system will do the following:

[1504] 1. The device uses the camera inside the refrigerator to take a picture of the inside and sends the image data to the server.

[1505] 2. The server's analysis module analyzes the image data and generates food data such as "cabbage," "carrot," and "chicken."

[1506] 3. The server searches the recipe database based on the generated food data and lists the most suitable recipes, taking into account the user's profile information (past recipe selections, allergy information, preferences).

[1507] 4. The display means displays the recipes "Chicken and Cabbage Stir-fry" and "Cabbage and Carrot Salad." The user uses the touchscreen to select "Chicken and Cabbage Stir-fry."

[1508] 5. Audio and visual guidance means guide the user through the cooking process step by step, providing specific instructions such as "First, shred the cabbage."

[1509] Prompt Sentence Examples

[1510] Below are some example prompts that the system can input to the generative AI model:

[1511] Say you have cabbage, carrots, and chicken in your fridge and your user likes stir-frying. Suggest a suitable recipe and provide step-by-step audio and visual instructions detailing each cooking step.

[1512] This system allows users to efficiently manage the food in their refrigerator, lighten the burden of daily cooking, reduce food waste, and even beginners can enjoy cooking with peace of mind.

[1513] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1514] Step 1:

[1515] Terminal handling

[1516] Input: Image data captured by a camera inside the refrigerator.

[1517] How it works: The device activates the camera periodically or whenever the refrigerator door is closed and takes a picture of the inside of the refrigerator.

[1518] Data processing: The captured image data is sent directly to the analysis means.

[1519] Output: Image data sent to analysis means.

[1520] Step 2:

[1521] Server Processing

[1522] Input: Image data sent from the device.

[1523] Operation: The server's analysis means analyzes the received image data using an object recognition algorithm (e.g., YOLO, Inception).

[1524] Data calculation: The algorithm identifies the type and location of food in the image and lists it as food data.

[1525] Output: A list of food data (e.g. "cabbage", "carrot", "chicken").

[1526] Step 3:

[1527] Server Processing

[1528] Input: The generated food data list.

[1529] Operation: The server uses the database search means to query the recipe database based on the food data list.

[1530] Data Calculation: Recipes listed as query results are optimized taking into account the user's profile information (previous recipe choices, allergies, preferences).

[1531] Output: A personalized recipe list.

[1532] Step 4:

[1533] Terminal handling

[1534] Input: Recipe list sent from the server.

[1535] Operation: A recipe list is displayed by the display means of the terminal.

[1536] Output: A visually displayed list of recipes (e.g. "Chicken and Cabbage Stir-fry", "Cabbage and Carrot Salad").

[1537] Step 5:

[1538] User Action

[1539] Input: Recipe list shown on the display.

[1540] How it works: The user selects the desired recipe using the touchscreen or voice recognition interface.

[1541] Output: The selected recipe (e.g. "Stir-fried chicken and cabbage").

[1542] Step 6:

[1543] Terminal handling

[1544] Input: The recipe selected by the user.

[1545] Operation: The audio and visual guidance means of the terminal guides the user through the cooking procedure step by step using audio and visual means.

[1546] Specific steps: "First, shred the cabbage," the voice instructions begin, and a video showing the steps for shredding the cabbage plays on the screen. Next, the instructions continue: "Cut the chicken into bite-sized pieces and heat oil in a frying pan."

[1547] Output: Audio and visual instructions for cooking procedures.

[1548] Through the above processing steps, users can efficiently manage the food in their refrigerator, easily select appropriate recipes, and proceed with cooking, thereby reducing food waste and easing the burden of cooking.

[1549] (Application example 1)

[1550] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1551] In modern life, there is a demand for effective use of food in the refrigerator and reduction of food waste. Furthermore, when shopping in a physical store, users can plan their shopping and cooking efficiently if they are provided with cooking method and recipe suggestions based on the food they are purchasing. However, previous systems have only been able to suggest recipes and provide cooking instructions based on the ingredients in the refrigerator, and have not been able to provide support for shopping in a physical store. Therefore, there is a need for a system that can seamlessly support the entire process from food selection to cooking in a physical store.

[1552] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1553] In this invention, the server includes image acquisition means for recognizing foods in the refrigerator, data analysis means for analyzing image data acquired by the image acquisition means to generate food information, information retrieval means for collecting the food information generated by the data analysis means and searching for appropriate recipes, display means for displaying the recipes retrieved by the information retrieval means, audio guidance means and video guidance means for supporting the user in cooking based on the recipe display by the display means, a visual device installed in the store for recognizing product selections by customers, product recognition means for collecting the product information recognized by the visual device, and control means for transmitting the product information generated by the product recognition means to the information retrieval means and displaying the recipes recommended by the information retrieval means on the display means. This makes it possible not only to recognize ingredients in the refrigerator but also to provide seamless support from food selection to cooking in a physical store.

[1554] An "image capture means" is a device or system for capturing an image of an object.

[1555] "Data analysis means" refers to a system or algorithm that processes and analyzes acquired image data to generate information about the target object.

[1556] The "information retrieval means" is a system for retrieving related information and content from an appropriate database based on the generated information.

[1557] "Display means" refers to a device or system for providing the searched information to the user in a visually recognizable form.

[1558] "Voice guidance means" refers to a system that provides guidance and instructions to users using voice.

[1559] A "visual guide" is a system that uses video to provide guides and instructions to users.

[1560] A "visual device" is a device or system used to recognize items selected by a customer in a physical store.

[1561] The "product recognition means" is a system for analyzing information acquired through a visual device and generating information about the recognized product.

[1562] A "control means" is a system or component that has the role of managing and coordinating the operations of various means and modules in an integrated manner.

[1563] The system according to the present invention uses a wide range of hardware and software to provide a consistent shopping and cooking assistance. The system includes the following main components:

[1564] 1. Image acquisition method

[1565] This includes cameras installed in the device's refrigerator or visual devices installed in physical stores, which take photos of food or products periodically or in response to user actions.

[1566] 2. Data analysis methods

[1567] Using a generative AI model such as TensorFlow, the captured image data is analyzed to identify the target food or product. This analysis process is performed on the server and generates the identified food information.

[1568] 3. Information search methods

[1569] The server uses the collected food information to search a recipe database for suitable recipes, optimizing the search results based on the user's previous choices, allergy information, and preferences.

[1570] 4. Display means

[1571] The recognized food information and searched recipes are displayed on visual devices such as refrigerator displays or smart glasses, which are equipped with tactile and voice input interfaces for users to select cooking methods.

[1572] 5. Audio and visual guidance means

[1573] Based on the selected recipe, the audio guide means provides audio instructions on specific cooking steps, and the visual guide means displays images of each cooking step to provide visual support to the user.

[1574] Examples:

[1575] When a user selects cabbage, carrots, and chicken at the supermarket, the visual device recognizes this and sends the information to the server. Based on this, the server searches for recipes such as "stir-fried chicken and cabbage" or "cabbage and carrot salad" and displays them on the smart glasses. If the user selects "stir-fried chicken and cabbage," the smart glasses will start a voice prompt saying, "First, shred the cabbage," and play a video showing how to cut the cabbage accordingly.

[1576] Example prompt sentence:

[1577] "Can you recommend a recipe for cabbage, carrots, and chicken?"

[1578] This system allows users to receive assistance with every step of the process, from selecting food in a physical store to cooking, not just with the ingredients in their refrigerator, reducing food waste and making cooking more efficient.

[1579] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1580] Step 1: Image acquisition

[1581] When a user opens or closes the refrigerator door, the image capture means of the device captures a photo of the food items inside the refrigerator. Also, when a user scans an item using smart glasses in a physical store, the visual device recognizes the item. Product image data from inside the refrigerator or store is acquired as input and sent to the analysis means. The output is the acquired raw image data.

[1582] Step 2: Image analysis

[1583] The server receives the image data sent from the image acquisition means and analyzes the data using a generative AI model such as TensorFlow. As input, it receives the acquired image data and identifies food or product information (e.g., "cabbage," "carrot," "chicken," etc.). The output is a list of recognized foods or products.

[1584] Step 3: Information search

[1585] The server collects food information generated by image analysis and searches a recipe database for suitable recipes based on that information. As input, it receives a list of recognized foods and identifies the best recipes based on the user's past choices, allergy information, and preferences. The output is a list of optimized recipes.

[1586] Step 4: View the recipe

[1587] The user receives the recipe information generated by the server in the refrigerator or in the brick-and-mortar store. Through the display means of the terminal, these recipes are visualized and displayed on the refrigerator display or smart glasses. As input, the searched recipe data is received and sent to the visual device. The output is a recipe list that the user can visually check.

[1588] Step 5: Recipe Selection

[1589] The user selects a preferred recipe from a displayed list of recipes. The user selects a recipe using a tactile or voice input interface. The system receives as input a visualized list of recipes and the user's selection action. The output is the specific recipe selected by the user.

[1590] Step 6: Cooking instructions

[1591] The terminal displays the details of the selected recipe and provides the user with cooking instructions using audio and video guidance means. As input, it receives detailed data of the selected recipe and plays audio and video guidance based on the instructions. As output, it provides audio and video guidance according to the cooking instructions.

[1592] In this way, by utilizing the system, users can reduce food waste and shop and cook more efficiently.

[1593] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1594] This invention combines a system that recognizes food in the refrigerator and suggests appropriate recipes to the user with an emotion engine that recognizes the user's emotions. Below, the system's program processing is explained in natural language with concrete examples.

[1595] System configuration

[1596] The system includes the following main components:

[1597] 1. Camera means: A camera for taking pictures of food in the refrigerator.

[1598] 2. Analysis means: A module that analyzes image data acquired by the camera means and generates food information.

[1599] 3. Search means: A server that searches for appropriate cooking recipes based on the generated food information.

[1600] 4. Display means: A refrigerator display that displays the recipes provided by the search means and allows the user to select one.

[1601] 5. Voice guidance means: A function that guides the user through cooking procedures by voice.

[1602] 6. Video guide means: A function that shows the cooking procedure to the user through video.

[1603] 7. Emotion engine: An engine that recognizes the user's emotions and reflects them in the system's operations.

[1604] Processing flow

[1605] 1. Food Recognition in the Refrigerator

[1606] Terminal handling

[1607] Every time the refrigerator door is closed, the device activates the AI ​​camera to scan the inside.

[1608] The image data acquired by the camera means is passed to the analysis means.

[1609] The analysis means generates food information using an object recognition algorithm, for example, identifying cabbage, carrots, and chicken.

[1610] Examples:

[1611] If there is cabbage, carrots, and chicken in the refrigerator, the camera will take a picture of them, and the analysis module will generate a list of ingredients such as "cabbage," "carrots," and "chicken." This list will be sent to the server.

[1612] 2. Recipe Suggestion Generation

[1613] Server Processing

[1614] The server queries a recipe database based on the received food information to search for available recipes.

[1615] The server selects the best recipes for you, taking into account your previous recipe choices, allergy information, and preferences. For example, if you have a dairy allergy, it will suggest dairy-free recipes.

[1616] Examples:

[1617] The server searches for recipes using "cabbage," "carrots," and "chicken," and based on the information that "the user likes stir-fried dishes," generates a list of recipes for "chicken and cabbage stir-fry" and "cabbage and carrot salad," and sends these to the refrigerator terminal.

[1618] 3. View and select recipes

[1619] Terminal and user handling

[1620] The device displays the recipe list on the display.

[1621] The user selects the desired recipe using the touchscreen or voice recognition interface.

[1622] Examples:

[1623] The refrigerator terminal's display shows "Stir-fried chicken and cabbage" and "Cabbage and carrot salad," and the user touches "Stir-fried chicken and cabbage" to select it.

[1624] 4. User Emotion Recognition

[1625] Emotion engine processing

[1626] The emotion engine analyzes the user's facial expressions, vocal tone, and physical movements to identify emotions.

[1627] The emotion engine transmits the identified emotion information to a server.

[1628] Examples:

[1629] When a user selects a recipe, the emotion engine analyzes the user's facial expressions and vocal tone to identify whether they are in a good mood, and this information is sent to the server.

[1630] 5. Cooking instructions

[1631] Terminal handling

[1632] The server optimizes recipe suggestions based on emotional information and sends them back to the device.

[1633] The terminal displays detailed information about the selected recipe on the display, and the audio guide means provides audio guidance of the cooking procedure.

[1634] A video guide means displays a video of each cooking step.

[1635] Examples:

[1636] After "Stir-fried chicken and cabbage" is selected, the device starts a voice prompt saying, "First, shred the cabbage," and plays a video showing how to cut the cabbage. Based on the results of the emotion engine, the voice guide provides instructions in a bright tone, lifting the user's spirits.

[1637] This system allows users to efficiently utilize the ingredients in their refrigerator while enjoying a personalized cooking experience that adapts to their emotions.

[1638] The processing flow will be explained below.

[1639] Step 1:

[1640] Start scanning the inside of the refrigerator

[1641] Every time the refrigerator door is closed, the device activates the AI ​​camera and scans the inside.

[1642] Step 2:

[1643] Acquisition of image data

[1644] The device takes a photo of the inside of the refrigerator with its camera and acquires the image data.

[1645] Step 3:

[1646] Image data analysis

[1647] The device passes the acquired image data to an analysis means, which then runs a food recognition algorithm to generate food information, such as using an object detection algorithm to identify cabbage, carrots, and chicken.

[1648] Step 4:

[1649] Submitting food information

[1650] The device sends the analyzed food information to the server.

[1651] Step 5:

[1652] Recipe Search

[1653] The server searches a recipe database based on the received food information and extracts available recipes.

[1654] Step 6:

[1655] Recipe optimization

[1656] The server selects the best recipes based on the user's previous recipe choices, allergy information, and preferences. For example, if the user has a dairy allergy, dairy-free recipes will be prioritized.

[1657] Step 7:

[1658] Generate a recipe list

[1659] The server generates an optimized recipe list and transmits it to the refrigerator terminal.

[1660] Step 8:

[1661] Displaying the recipe list

[1662] The device displays the recipe list received from the server on the display. For example, "Stir-fried chicken and cabbage" or "Cabbage and carrot salad" are displayed.

[1663] Step 9:

[1664] User recipe selection

[1665] The user selects the desired recipe using the touchscreen or voice recognition interface.

[1666] Step 10:

[1667] User emotion recognition

[1668] The device's emotion engine analyzes the user's facial expressions, voice tone, and physical movements to identify emotions. It also analyzes the user's facial expressions and voice tone when selecting recipes to identify emotions.

[1669] Step 11:

[1670] Sending emotional information

[1671] The terminal transmits the identified emotion information to the server.

[1672] Step 12:

[1673] Emotion-based recipe re-suggestion

[1674] The server takes emotional information into account and optimizes recipe suggestions to match the user's emotions. For example, if the user is feeling depressed, it will suggest easy and hassle-free recipes.

[1675] Step 13:

[1676] View recipe details

[1677] The terminal displays detailed information about the selected recipe on the display.

[1678] Step 14:

[1679] Audio guide for cooking steps

[1680] The device will start a voice prompt and guide the user through each cooking step. For example, it might say, "First, shred the cabbage."

[1681] Step 15:

[1682] Video guide for cooking steps

[1683] The device will then display a visual guide for each cooking step in conjunction with the audio guide, for example showing how to cut cabbage.

[1684] Step 16:

[1685] User cooking execution

[1686] Users follow audio and video guides to cook and create delicious meals.

[1687] Through this process, users can efficiently utilize the ingredients in their refrigerators, enjoy cooking simply and effectively, and optimize their cooking experience based on their emotions, providing a more personalized service.

[1688] Example 2

[1689] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1690] Conventional refrigerator systems have limited functionality for efficiently recognizing ingredients in the refrigerator and providing users with recipe suggestions based on that information. It is also difficult to provide personalized suggestions that take into account the user's emotions and individual preferences. Therefore, there is a demand for a system that allows users to easily find the perfect dish and enjoy cooking. In particular, there is a lack of functionality that recognizes the user's emotions in real time during the cooking process and provides guidance accordingly.

[1691] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1692] In this invention, the server includes an imaging means for recognizing food in the refrigerator, an analysis means for analyzing the image data obtained by the imaging means to generate ingredient information, a search means for collecting the ingredient information generated by the analysis means and searching for appropriate cooking procedures, a display means for displaying the cooking procedures searched by the search means, an audio guide means and a visual guide means for assisting the user in cooking based on the cooking procedure display by the display means, and an emotion recognition means for recognizing the user's emotions and reflecting them in the operation content. This makes it possible to effectively use ingredients in the refrigerator and provide personalized recipe suggestions and cooking support according to the user's emotions and preferences.

[1693] The "imaging means" is a device that acquires image data to identify ingredients in the refrigerator.

[1694] The "analysis means" is a device or module that analyzes the image data acquired by the imaging means and identifies the type and amount of food.

[1695] The "search means" is a device or software that searches a database for appropriate cooking procedures and recipes using the ingredient information generated by the analysis means.

[1696] The "display means" is a device for visually showing the cooking steps and recipes found by the search means to the user, and an example of this is a display.

[1697] "Audio guidance means" refers to a device or software that provides audio guidance to the user on cooking methods and procedures.

[1698] A "visual guide means" is a device or software that visually explains cooking methods and procedures to the user.

[1699] "Emotion recognition means" refers to a device or software that analyzes a user's facial expressions, voice, and physical movements to recognize their emotions.

[1700] The present invention combines a system that recognizes food in a refrigerator and suggests suitable cooking recipes to the user with technology that recognizes user emotions. The system includes the following main components: an imaging means, an analysis means, a search means, a display means, an audio guide means, a visual guide means, and an emotion recognition means.

[1701] Imaging means

[1702] The device is equipped with a camera that recognizes the food inside the refrigerator. The AI ​​camera activates every time the refrigerator door is closed and scans the ingredients inside.

[1703] Analysis means

[1704] The acquired image data is analyzed using an object recognition algorithm (e.g., YOLOv4) to generate food information. The analysis means identifies ingredients such as cabbage, carrots, and chicken, and sends the information to a server.

[1705] Search methods

[1706] The server queries the recipe database based on the received food information to find available recipes, and then selects the most suitable recipe taking into account the user's past recipe choices, allergy information, and preferences (e.g., dairy allergy).

[1707] Display means

[1708] The terminal displays the recipe list sent from the server on a display, which includes a touchscreen interface and a voice recognition interface (e.g., Google Assistant) that the user uses to select a desired recipe.

[1709] Acoustic and visual guide means

[1710] Based on the selected recipe, the audio guide module provides audio instructions for the cooking steps, and the visual guide module displays videos of each cooking step on the display. The audio guide module uses Amazon Polly, and the visual guide module plays videos using the YouTube API.

[1711] emotion recognition means

[1712] The emotion engine captures the user's facial expressions with a camera and analyzes them using the Facial Action Coding System. It also uses voice analysis technology (e.g., DeepSpeech) to analyze the tone of the voice and identify the user's emotions. The identified emotion information is sent to the server and used to optimize recipe suggestions.

[1713] Specific examples

[1714] For example, if there is cabbage, carrots, and chicken in the refrigerator, the imaging means takes a picture of them, and the analysis means generates an ingredient list of "cabbage," "carrots," and "chicken." This list is sent to the server, which searches for available recipes. The server then sends recipes such as "stir-fried chicken and cabbage" and "cabbage and carrot salad" to the terminal, which displays them on the display. When the user selects "stir-fried chicken and cabbage," the audio guide means begins guiding the user by saying, "First, shred the cabbage," and the visual guide means plays a video on how to cut the cabbage. At the same time, the emotion recognition means analyzes the user's facial expressions and voice tone to provide personalized guidance tailored to their emotions.

[1715] Prompt Sentence Examples

[1716] An example prompt for a generative AI model is:

[1717] "Please explain the process by which an AI system recognizes that there is cabbage, carrots, and chicken in the refrigerator and suggests appropriate recipes to the user. Also, please include specific actions and techniques to identify the user's emotions and provide guidance tailored to those emotions."

[1718] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1719] Step 1:

[1720] Scanning for ingredient recognition (processing on device)

[1721] Input: Status inside the refrigerator, refrigerator door opening / closing information.

[1722] How it works: Every time the refrigerator door is closed, the device detects this with a sensor and activates the AI ​​camera, which then captures images of the inside of the refrigerator and generates image data.

[1723] Output: Image data of the inside of the refrigerator.

[1724] Step 2:

[1725] Image data analysis (terminal processing)

[1726] Input: Image data of the inside of the refrigerator.

[1727] How it works: The device passes the image data to the analysis means, which uses an image analysis algorithm (e.g., YOLOv4) to identify the type and location of the food.

[1728] Output: A list of identified ingredients (e.g., cabbage, carrot, chicken).

[1729] Step 3:

[1730] Sending ingredient information and searching for recipes (server processing)

[1731] Input: Ingredient list sent from the device.

[1732] How it works: The server receives the ingredient list, queries the recipe database, and selects the best recipe, taking into account the user's previous recipe choices, allergy information, and preferences.

[1733] Output: A list of available recipes (e.g., Chicken and Cabbage Stir-fry, Cabbage and Carrot Salad).

[1734] Step 4:

[1735] Displaying recipes (terminal processing and user processing)

[1736] Input: Recipe list sent from the server.

[1737] How it works: The device displays a list of recipes on the display. The user selects the desired recipe using the touchscreen or voice recognition interface.

[1738] Output: The recipe selected by the user.

[1739] Step 5:

[1740] Emotion recognition (emotion engine processing)

[1741] Input: User facial expressions, voice tone, and physical movements.

[1742] How it works: The emotion recognition unit captures the user's facial expressions with a camera and analyzes them using the Facial Action Coding System. It also uses voice analysis technology to analyze the tone of the voice and identify the user's emotions.

[1743] Output: Identified user's sentiment information.

[1744] Step 6:

[1745] Optimization and presentation of cooking procedures (terminal processing)

[1746] Input: Selected recipe and emotion information sent from the server.

[1747] Operation: The server optimizes recipe suggestions based on emotion information and resends them to the device. The device then displays detailed cooking instructions on the screen, uses audio guidance (e.g., Amazon Polly) to provide audible instructions, and plays videos using visual guidance (e.g., YouTube API).

[1748] Output: Personalized cooking instructions and guide.

[1749] Examples:

[1750] Step 1 through Step 6:

[1751] If there is cabbage, carrots, or chicken in the refrigerator, the AI ​​camera will take a photo of it, and the device will pass the image data to the analysis means. The analysis means will identify the cabbage, carrots, and chicken, and send the ingredient list to the server. The server will query the recipe database based on the ingredient list and user information, generate a list of recipes for "chicken and cabbage stir-fry" and "cabbage and carrot salad," and send this to the device. When the user selects "chicken and cabbage stir-fry" on the touchscreen, the emotion engine will analyze the user's facial expressions and voice tone, guide them through the cooking steps in a bright tone, and play a video on how to cut cabbage on the display.

[1752] (Application example 2)

[1753] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1754] Conventional recipe suggestion systems have the ability to suggest recipes using ingredients in the refrigerator, but they are unable to flexibly suggest recipes based on the user's emotions. For example, it is difficult to suggest dishes that suit the user's mood when they are feeling depressed or excited. For this reason, there is a need for personalized recipe suggestions that take the user's psychological state into account.

[1755] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes a camera means for recognizing food in the refrigerator, an analysis means for analyzing image data obtained by the camera means to generate food information, a search means for collecting the food information generated by the analysis means and searching for appropriate cooking recipes, a display means for displaying the recipes searched by the search means, and an emotion engine means for recognizing the user's emotions and reflecting them in recipe suggestions. This makes it possible to suggest recipes that take the user's emotional state into consideration.

[1756] "Camera means" refers to a photographic device used to recognize food items in the refrigerator.

[1757] The "analysis means" is a module that analyzes image data obtained by the camera means and generates food information.

[1758] The "search means" is a device or program that searches for appropriate cooking recipes based on the food information generated by the analysis means.

[1759] The "display means" is a device or interface for visually presenting the recipes retrieved by the search means to the user.

[1760] The "voice guide means" is a voice guide function that assists the user in cooking based on the recipe displayed by the display means.

[1761] The "visual guide means" is a function that presents the user with a visual guide to cooking procedures based on the recipe displayed by the display means.

[1762] The "emotion engine means" is an engine that recognizes the user's emotions and reflects them in the cooking recipe suggestions.

[1763] A "touch screen interface" is a tactile input device that allows a user to select recipes.

[1764] A "voice recognition interface" is a device or program that recognizes and inputs user voice instructions.

[1765] This invention combines a system that recognizes food in a refrigerator and suggests suitable cooking recipes to the user with an emotion engine that recognizes the user's emotions. Specific embodiments of the system will be described below.

[1766] System configuration

[1767] The system includes the following main components:

[1768] 1. Camera means: A camera for taking pictures of food in the refrigerator. This can be a camera built into a smartphone.

[1769] 2. Analysis means: A module that analyzes image data acquired by the camera means and generates food information. It uses a food recognition library.

[1770] 3. Search method: A server that searches for appropriate cooking recipes based on the generated food information. It uses the recipe search service API.

[1771] 4. Display means: An interface that displays recipes provided by the search means and allows the user to select one. An example is using the display of a smartphone.

[1772] 5. Voice guidance means: A function that guides the user through cooking procedures by voice. Use the voice guidance service.

[1773] 6. Video guide: A function that shows the cooking procedure to the user through video. It also has a video playback function.

[1774] 7. Emotion engine means: An engine that recognizes the user's emotions and reflects them in the system's operations. It uses an emotion recognition library.

[1775] System Operation

[1776] Food Recognition and Emotion Recognition

[1777] The device activates the camera when the refrigerator door is closed or based on a specific trigger, scans the interior, and photographs the food. The acquired image data is passed to an analysis means, which generates food information. A food recognition library is used to identify, for example, cabbage, carrots, chicken, etc. The device then analyzes the user's face to identify their emotions. The emotion recognition library analyzes the user's facial expressions and voice tone to generate emotional information.

[1778] Generating recipe suggestions

[1779] The server searches a recipe database based on the food information generated by the analysis means to query available recipes. It also optimizes recipes by taking into account the user's past recipe choices, allergy information, and preferences. It also takes into account emotional information to suggest recipes that suit the user's mood. For example, if the user is in high spirits, it will suggest a simple and fun recipe such as "stir-fried chicken and cabbage."

[1780] Viewing and Selecting Recipes

[1781] The provided recipes are displayed on the device's display, and the user can select them via the touchscreen or voice recognition interface. Based on the selected recipe, cooking instructions are provided through audio and visual guidance. The audio guidance is provided in a tone that corresponds to the user's emotions; for example, if the user is in a happy mood, a cheerful tone will be used to guide the user.

[1782] Specific examples

[1783] If there is cabbage, carrots, and chicken in the refrigerator, the camera takes a picture of them, and the analysis means generates a list of ingredients such as "cabbage," "carrots," and "chicken." This list is sent to the server. The server searches for recipes using "cabbage," "carrots," and "chicken," and based on the information that "the user likes stir-fries," generates a list of recipes such as "chicken and cabbage stir-fry" and "cabbage and carrot salad," and sends this to the refrigerator terminal. When the user selects a recipe, the emotion engine analyzes the user's facial expressions and voice tone to identify that they are in an excited mood. This information is sent to the server.

[1784] Prompt Sentence Examples

[1785] "The system of this invention recognizes the food in the refrigerator and suggests appropriate cooking recipes. The system has an emotion engine built in, which recognizes the user's emotions and reflects them in the recipe suggestions. If the user is smiling, it will tell us what kind of recipes should be suggested."

[1786] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1787] Step 1:

[1788] Food Recognition

[1789] Device behavior

[1790] The device activates the camera when the refrigerator door closes, scans the interior and takes pictures of food.

[1791] Input: Image data of the inside of the refrigerator acquired by a camera.

[1792] Data processing and calculation: Food information is extracted from image data using a food recognition library. This analysis generates a list of ingredients, such as "cabbage," "carrot," and "chicken."

[1793] Output: An ingredient list containing the generated food information.

[1794] Step 2:

[1795] User emotion recognition

[1796] Device behavior

[1797] The device takes a picture of the user's face with a camera and analyzes the user's emotions using an emotion recognition library.

[1798] Input: User's face image data acquired by camera.

[1799] Data processing and calculation: Using the emotion recognition library, the system analyzes facial expressions and voice tones to identify the user's emotions. Emotional information such as "excited," "neutral," and "sad" is generated.

[1800] Output: The generated emotion information.

[1801] Step 3:

[1802] Sending food and emotional information

[1803] Device behavior

[1804] The terminal transmits the generated food information and emotion information to the server.

[1805] Input: The food information and emotion information generated earlier.

[1806] Data processing and calculation: None in particular.

[1807] Output: Food information and emotion information sent to the server.

[1808] Step 4:

[1809] Recipe Search and Optimization

[1810] Server Operation

[1811] The server receives the food and emotion information and searches the recipe database based on it, optimizing the recipe taking into account past recipe choices, allergy information, and preferences.

[1812] Input: Food information, emotional information, past recipe selection data, allergy information, user preferences.

[1813] Data processing and data calculation: Execute database queries to filter the most suitable recipes from multiple options. Based on emotional information, narrow down the recipes that suit your mood.

[1814] Output: A list of suitable recipes.

[1815] Step 5:

[1816] Sending and displaying recipe lists

[1817] Server and terminal operation

[1818] The server sends the optimized recipe list to the terminal, which displays the recipe list on its display.

[1819] Input: Optimized recipe list.

[1820] Data processing and calculation: None in particular.

[1821] Output: Recipe list sent to the terminal, recipe list displayed on the display.

[1822] Step 6:

[1823] User recipe selection

[1824] User and device behavior

[1825] Users select the desired recipe using the touchscreen or voice recognition interface.

[1826] Input: User selection (touch or voice input).

[1827] Data processing / data calculation: Obtain data for the selected recipe.

[1828] Output: Selected recipe information.

[1829] Step 7:

[1830] Cooking procedure guide

[1831] Device behavior

[1832] Based on the selected recipe, the system provides audio and visual instructions for cooking, and adjusts the tone of the audio guide based on the user's emotional state.

[1833] Input: Selected recipe information, emotion information.

[1834] Data processing and calculation: Audio and visual guide generation. Optimizing the tone of audio guides based on emotional information.

[1835] Output: Audio and visual guide to cooking steps.

[1836] These steps allow users to efficiently utilize the ingredients in their refrigerator and enjoy a personalized cooking experience based on their emotions.

[1837] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1838] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1839] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1840] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1841] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1842] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1843] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1844] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1845] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1846] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1847] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1848] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1849] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1850] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1851] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1852] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1853] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1854] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1855] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1856] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1857] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1858] The following is further disclosed regarding the above embodiment.

[1859] (Claim 1)

[1860] camera means for recognizing food items in the refrigerator;

[1861] an analysis means for analyzing the image data obtained by the camera means and generating food information;

[1862] a search means for collecting the food information generated by the analysis means and searching for appropriate cooking recipes;

[1863] a display means for displaying the recipes searched by the search means;

[1864] an audio guide means and a video guide means for assisting the user in cooking based on the recipe displayed by the display means;

[1865] A system including:

[1866] (Claim 2)

[1867] 10. The system of claim 1, wherein the analyzing means includes means for optimizing the recipe based on the user's past recipe selections, allergy information, and preferences.

[1868] (Claim 3)

[1869] 10. The system of claim 1, wherein the display means has one of a touch screen interface and a voice recognition interface, providing a means for a user to select a recipe.

[1870] "Example 1"

[1871] (Claim 1)

[1872] imaging means for recognizing the food inside;

[1873] an analysis means for analyzing the image information obtained by the imaging means and generating food data;

[1874] a database search means for transferring the food data generated by the analysis means and searching for optimal cooking recipes;

[1875] a display means for displaying the recipes searched by the database search means;

[1876] a voice guidance means and a video guidance means for assisting a user in cooking based on a recipe displayed on a display means;

[1877] A system including:

[1878] (Claim 2)

[1879] 10. The system of claim 1, wherein the analysis means includes a method for optimizing recipes based on a user's past recipe selections, allergy information, and preferences.

[1880] (Claim 3)

[1881] 10. The system of claim 1, wherein the display means has one of a tactile interface and a voice recognition interface to provide a method for a user to select a recipe.

[1882] "Application Example 1"

[1883] (Claim 1)

[1884] image acquisition means for recognizing food items in the refrigerator;

[1885] a data analysis means for analyzing the image data obtained by the image acquisition means and generating food information;

[1886] an information retrieval means for collecting the food information generated by the data analysis means and retrieving an appropriate cooking method;

[1887] a display means for displaying the recipes searched by the information search means;

[1888] a voice guidance means and a video guidance means for assisting a user in cooking based on the recipe displayed by the display means;

[1889] a visual device installed in the store for recognizing a customer's product selection;

[1890] a product recognition means for collecting product information recognized by the visual device;

[1891] a control means for transmitting the product information generated by the product recognition means to an information retrieval means and displaying the recipe recommended by the information retrieval means on a display means;

[1892] A system including:

[1893] (Claim 2)

[1894] 10. The system of claim 1, wherein the data analysis means includes means for optimizing recipes based on the user's past recipe selections, allergy information, and preferences.

[1895] (Claim 3)

[1896] 10. The system of claim 1, wherein the display means has one of a tactile input interface and a voice input interface to provide a means for a user to select a recipe.

[1897] "Example 2: Combining Emotion Engines"

[1898] (Claim 1)

[1899] an imaging means for recognizing food in the refrigerator;

[1900] an analysis means for analyzing the image data obtained by the imaging means and generating food ingredient information;

[1901] a search means for collecting the ingredient information generated by the analysis means and searching for an appropriate cooking procedure;

[1902] a display means for displaying the cooking recipes searched by the search means;

[1903] an audio guide means and a visual guide means for assisting a user in cooking based on the cooking procedure displayed by the display means;

[1904] An emotion recognition means for recognizing the emotion of a user and reflecting it in the operation content;

[1905] A system including:

[1906] (Claim 2)

[1907] 10. The system of claim 1, wherein the analyzing means includes means for optimizing the cooking recipe based on the user's past cooking recipe choices, allergy information, and preferences.

[1908] (Claim 3)

[1909] 10. The system of claim 1, wherein the display means has one of a touch screen interface and a voice recognition interface, providing a means for a user to select a cooking recipe.

[1910] "Application example 2 when combining emotion engines"

[1911] (Claim 1)

[1912] camera means for recognizing food items in the refrigerator;

[1913] an analysis means for analyzing the image data obtained by the camera means and generating food information;

[1914] a search means for collecting the food information generated by the analysis means and searching for appropriate cooking recipes;

[1915] a display means for displaying the recipes searched by the search means;

[1916] an audio guide means and a video guide means for assisting the user in cooking based on the recipe displayed by the display means;

[1917] An emotion engine means for recognizing user emotions and reflecting them in recipe suggestions;

[1918] A system including:

[1919] (Claim 2)

[1920] 10. The system of claim 1, wherein the analyzing means includes means for optimizing the recipe based on the user's past recipe selections, allergy information, and preferences.

[1921] (Claim 3)

[1922] 10. The system of claim 1, wherein the display means has one of a touch screen interface and a voice recognition interface, providing a means for a user to select a recipe. [Explanation of symbols]

[1923] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. camera means for recognizing food items in the refrigerator; an analysis means for analyzing the image data obtained by the camera means and generating food information; a search means for collecting the food information generated by the analysis means and searching for appropriate cooking recipes; a display means for displaying the recipes searched by the search means; an audio guide means and a video guide means for assisting the user in cooking based on the recipe displayed by the display means; A system including:

2. 10. The system of claim 1, wherein the analyzing means includes means for optimizing the recipe based on the user's past recipe selections, allergy information, and preferences.

3. 10. The system of claim 1, wherein the display means comprises one of a touch screen interface and a voice recognition interface, providing a means for a user to select a recipe.

Images