System

Abstract

A system is provided.SOLUTION: A system comprising: an acquisition unit disposed inside a refrigeration facility and configured to acquire an image; a recognition unit configured to process the acquired image and automatically recognize an ingredient; a generation unit configured to generate a recipe based on ingredient information recognized by the recognition unit; and a transmission unit configured to transmit the recipe generated by the generation unit to a user terminal.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, many people struggle to make the most of the food left in their refrigerators and are left struggling to plan their daily meals. This waste of food also contributes to food waste and impacts the environment. Furthermore, in many cases, the system fails to suggest meals appropriate for the user's health condition or preferences. There is a need to solve these problems. [Means for solving the problem]

[0005] The present invention provides a system that is arranged in a refrigeration unit and includes an acquisition means for acquiring images, a recognition means for processing the acquired images and automatically recognizing ingredients, a generation means for generating a recipe based on ingredient information recognized by the recognition means, and a transmission means for transmitting the recipe generated by the generation means to a user terminal. Furthermore, the generation means includes a means for adjusting the recipe based on health information or preference information about the user, and a means for tracking the expiration dates of ingredients in the refrigeration unit and generating a recipe that prioritizes ingredients with upcoming expiration dates, thereby realizing efficient, healthy, and waste-free meal suggestions.

[0006] "Refrigeration equipment" is a mechanical device used to store ingredients and food at low temperatures.

[0007] "Acquisition means" refers to cameras or imaging devices that capture images of the inside of the refrigeration equipment, as well as related hardware and software.

[0008] The "recognition means" refers to an algorithm or system that analyzes the image captured by the acquisition means and identifies the ingredients contained therein.

[0009] "Generation means" refers to software or algorithms for creating appropriate recipes based on the ingredient information identified by the recognition means.

[0010] The "transmission means" refers to a communication module or a network interface for transferring the recipe generated by the generation means to the user terminal.

[0011] "User terminal" refers to an information terminal device used by a user, such as a smartphone or tablet.

[0012] "Health information" refers to health-related data such as a user's weight, number of steps, and age.

[0013] "Preference information" refers to information about a user's personal preferences, such as the type of food or seasoning they like.

[0014] "Expiration date" refers to the date by which food ingredients are used to maintain their quality.

[0015] "Tracking" refers to the act of continuously monitoring and recording the condition and location of food items within refrigeration equipment. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram illustrating a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention is a system that manages ingredients and suggests recipes for users using acquisition means, recognition means, generation means, and transmission means installed in refrigeration equipment. The system aims to suggest efficient and healthy meals by taking into account the user's health information and preference information.

[0038] Overall system configuration

[0039] The system mainly consists of the following components:

[0040] 1. Camera inside the refrigeration equipment (acquisition method)

[0041] 2. Server that performs AI processing (recognition, generation, and transmission means)

[0042] 3. User's smartphone app (user device)

[0043] System Operation Overview

[0044] 1. Camera inside the refrigeration equipment (acquisition method)

[0045] Terminal (refrigeration equipment): A camera inside the refrigeration equipment captures images of the inside at regular intervals or according to user instructions.

[0046] Terminal (refrigeration equipment): Uploads the acquired image data to the server.

[0047] 2. Image analysis and food ingredient recognition (recognition method)

[0048] Server: Receives the acquired image data and automatically recognizes the ingredients inside using an AI image analysis algorithm.

[0049] Server: Stores the recognized ingredient information in a database and updates the internal status of the refrigeration equipment.

[0050] 3. Recipe generation (generation method)

[0051] Server: Generates optimal recipes based on the user's health information (number of steps, weight, age, etc.) and preference information (favorite dishes and seasonings, etc.).

[0052] Server: Using AI technology, it suggests cooking recipes using recognized ingredients.

[0053] 4. Notification of information and recipe suggestions (transmission method)

[0054] Server: Sends the generated recipe to the user's smartphone app.

[0055] Users can view suggested recipes through a smartphone app.

[0056] Specific examples

[0057] For example, consider a case where cabbage, carrots, and chicken are stored in a refrigeration facility.

[0058] 1. Terminal (refrigeration equipment): The camera takes pictures of the inside of the refrigerator and uploads the image data to the server.

[0059] 2. Server: Analyzes the uploaded image and automatically recognizes "cabbage," "carrot," and "chicken."

[0060] 3. User: Enters the following information into the smartphone app: "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food." This information is sent to the server.

[0061] 4. Server: Integrates the ingredient list with user information and generates recipe suggestions such as "Teriyaki Chicken" or "Japanese-style Cabbage and Carrot Salad."

[0062] 5. User: Selects "Teriyaki Chicken" from the recipes suggested by the smartphone app and makes adjustments such as "making it stronger in seasoning."

[0063] 6. Server: Generates the fine-tuned recipe and sends the final cooking instructions and required seasoning list to the user's smartphone.

[0064] 7. User: Can cook food according to the recipe displayed on the smartphone app.

[0065] In this way, the present invention is a system that efficiently utilizes ingredients stored in refrigeration equipment and suggests optimal recipes based on the user's health and preference information, helping to reduce food waste and solving daily meal problems.

[0066] The processing flow will be explained below.

[0067] Step 1:

[0068] Terminal (refrigerator): The camera inside the refrigerator takes pictures of the inside at a set time. For example, if it is set to take pictures every day at 10:00, the camera will automatically operate at that time.

[0069] Step 2:

[0070] Terminal (refrigerator): The acquired image data is sent to the server via the internal storage device of the refrigeration equipment or the network.

[0071] Step 3:

[0072] Server: The server takes the received image data and begins processing it using AI image analysis algorithms, specifically detecting objects in the image and identifying which ingredients they correspond to.

[0073] Step 4:

[0074] Server: Stores the recognized ingredient information in a database. Newly recognized ingredients and updated information on existing ingredients are reflected in the database.

[0075] Step 5:

[0076] User: Opens the smartphone app and enters health and preference information. For example, enter "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food."

[0077] Step 6:

[0078] Terminal (smartphone): The entered health information and preference information is sent to the server.

[0079] Step 7:

[0080] Server: Integrates the acquired ingredient information with the user's health and preference information to generate optimal recipes. For example, it selects a Japanese recipe using cabbage, carrots, and chicken.

[0081] Step 8:

[0082] Server: Sends the generated recipe to the user's smartphone via the network.

[0083] Step 9:

[0084] User: Check the recipes displayed on the smartphone app and select the desired recipe. For example, select "Teriyaki Chicken."

[0085] Step 10:

[0086] User: Requests seasoning or portion adjustments for a selected recipe. For example, they might type, "I'd like it a little stronger today."

[0087] Step 11:

[0088] Server: Fine-tunes the recipe data based on the user's request and generates the final cooking steps and list of required seasonings.

[0089] Step 12:

[0090] Server: Sends the fine-tuned recipe and cooking instructions to the user's smartphone.

[0091] Step 13:

[0092] User: Cooks a dish by following the cooking instructions displayed on the smartphone app.

[0093] Step 14:

[0094] Server: Regularly checks the food database in the refrigeration facility to identify food items that are close to their expiration date.

[0095] Step 15:

[0096] Server: Generates recipes that prioritize ingredients with an approaching expiration date and sends notification information to the user's smartphone.

[0097] Step 16:

[0098] Terminal (refrigerator): Notifications regarding expiration dates are displayed on the refrigerator equipment display and on the user's smartphone app.

[0099] Step 17:

[0100] User: Check the notification, use the suggested recipes as a guide, and prioritize ingredients that are close to their expiration date.

[0101] Through these steps, users can efficiently utilize the food stored in their refrigerators, helping them manage their health and reduce food waste.

[0102] Example 1

[0103] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0104] In modern life, efficiently managing food stored in refrigerators and preparing healthy meals that suit individual tastes is a time-consuming and labor-intensive task. Conventional refrigerators lack the functionality to suggest optimal dishes based on the expiration date and type of ingredients, and it requires a great deal of effort for users to understand the condition of ingredients and select recipes based on health information and preferences. This has led to increased food waste and unhealthy food choices.

[0105] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0106] In this invention, the server includes an acquisition means, a recognition means, a generation means, a transmission means, and a user terminal. This allows for efficient management of ingredients in the refrigeration unit and suggests optimal recipes based on the user's health and preference information. Specifically, a camera installed inside the refrigeration unit captures images of ingredients and uploads them to the server. The server automatically recognizes ingredients from the uploaded images using an image analysis algorithm based on deep learning and stores the information in a database. The server then receives the user's health and preference information and uses a generative AI model to generate an optimal recipe based on this information. The generated recipe is then sent to the user's smartphone app, where the user can review the suggested recipe and make adjustments as needed. This makes it easier for users to prepare healthy meals that suit their preferences, contributing to reducing food waste.

[0107] The "acquisition means" is a device for acquiring image data using a camera or the like inside the refrigeration equipment and transmitting the image data to the server.

[0108] The "recognition means" refers to an AI image analysis algorithm and its processing device that analyzes image data uploaded to the server and automatically identifies the ingredients inside.

[0109] The "generation means" refers to an algorithm and a processing device for generating an optimal recipe based on the ingredient information identified by the recognition means and the user's health information and preference information.

[0110] The "transmission means" refers to a communication device and its protocol for transmitting the recipe information generated by the generation means to a user terminal such as a smartphone app of the user.

[0111] A "user terminal" is an electronic device such as a smartphone or tablet used by a user, and is used to receive, display, and adjust suggested recipe information.

[0112] "Health Information" refers collectively to a User's weight, number of steps, age, and other health-related data.

[0113] "Preference information" is a general term for information about the types of food and seasonings that a user likes.

[0114] A "generative AI model" is an advanced artificial intelligence algorithm, such as GPT-4, that generates new information, especially recipe information, based on input data.

[0115] A "prompt sentence" is an input sentence used to cause an AI model to generate a particular output.

[0116] This invention is a system that manages ingredients and suggests recipes to users using acquisition means, recognition means, generation means, and transmission means installed in refrigeration equipment. This system efficiently manages ingredients in the refrigerator and provides optimal recipes based on the user's health information and preference information, thereby helping to reduce food waste and promote healthy eating habits.

[0117] Hardware and Software Configuration

[0118] 1. Camera inside the refrigeration equipment (acquisition method)

[0119] Terminal (refrigeration equipment): A camera is installed inside the refrigerator and takes images of the inside of the refrigerator at regular intervals or according to the user's instructions. This camera uploads the image data to a server via an internet connection.

[0120] 2. Image analysis and food ingredient recognition (recognition method)

[0121] Server: The server receives the uploaded image data and automatically recognizes the ingredients in the refrigerator using an image analysis algorithm that uses deep learning. This algorithm uses a deep learning framework such as TensorFlow. The recognized ingredient information is stored in a database and the internal status of the refrigerator is updated in real time.

[0122] 3. Recipe generation (generation method)

[0123] Server: The server receives health information (number of steps, weight, age, etc.) and preference information (preferred cuisine, seasonings, etc.) from the user's smartphone app. Based on this, it uses a generative AI model (e.g., GPT-4) to generate the optimal recipe for the user. In this process, recipes are created taking into account the priority of ingredient use and nutritional balance.

[0124] 4. Notification of information and recipe suggestions (transmission method)

[0125] Server: Sends the generated recipe information to the user's smartphone app, where the user can review the suggested recipe and make adjustments as needed.

[0126] Specific examples

[0127] For example, if cabbage, carrots, and chicken are stored in the refrigerator, the following processing will occur:

[0128] 1. Terminal (refrigeration equipment): The camera takes pictures of the inside of the refrigerator and uploads the image data to the server.

[0129] 2. Server: Analyzes the uploaded image and automatically recognizes "cabbage," "carrot," and "chicken."

[0130] 3. User: Enters "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food" into the smartphone app and sends this information to the server.

[0131] 4. Server: Integrates the recognized ingredient information with the user's health information and generates recipe suggestions such as "Teriyaki Chicken" or "Japanese-style Cabbage and Carrot Salad."

[0132] 5. User: Check the suggested recipes on the smartphone app, select "Teriyaki Chicken," and make adjustments such as "Make it stronger in seasoning."

[0133] 6. Server: Regenerates the adjusted recipe and sends the final cooking instructions and required seasoning list to your smartphone.

[0134] 7. User: Can cook food according to the recipe displayed on the smartphone app.

[0135] Prompt Sentence Examples

[0136] For example, you can generate a recipe by inputting the following prompt into a generative AI model:

[0137] Prompt statement:

[0138] There are "cabbage," "carrots," and "chicken" in the refrigerator. The user's health information is as follows: "Today's steps: 5,000," "Weight: 70 kg," and "Preferences: Japanese food." Based on this, please suggest a healthy and delicious recipe.

[0139] output:

[0140] 1. Teriyaki chicken

[0141] 2. Japanese-style cabbage and carrot salad

[0142] In this way, the present invention realizes a system that can support a healthy diet by effectively utilizing ingredients stored in refrigeration equipment and providing optimal recipes based on the user's health and preference information.

[0143] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0144] Step 1:

[0145] Terminal (refrigeration equipment): A camera installed inside the refrigerator takes pictures of the inside of the refrigerator at regular intervals or according to the user's instructions.

[0146] Input: User instruction or scheduled trigger

[0147] Output: Image data of the inside of the refrigerator

[0148] Specifically, the camera captures high-resolution images from different angles inside the refrigerator, making all the ingredients inside visible.

[0149] Step 2:

[0150] Terminal (refrigeration equipment): Uploads the acquired image data to a server via an internet connection.

[0151] Input: Image data of the inside of the refrigerator

[0152] Output: Image data sent to the server

[0153] Specifically, the image data is compressed and transferred to a server using a secure protocol.

[0154] Step 3:

[0155] Server: Receives uploaded image data and performs image analysis using a deep learning framework (e.g., TensorFlow).

[0156] Input: Uploaded image data of the inside of the refrigerator

[0157] Output: Recognized ingredients

[0158] Specifically, the server runs an object detection algorithm to identify the type and location of ingredients in the image and stores this information in a database.

[0159] Step 4:

[0160] User: Enters health information (e.g., number of steps, weight, age) and preference information (e.g., favorite food, seasoning) into the smartphone app.

[0161] Input: User health and preference information

[0162] Output: User information sent to the server

[0163] Specifically, the user uses a smartphone app to input information using an intuitive interface, and then presses the send button to transfer the information to the server.

[0164] Step 5:

[0165] Server: Based on the collected ingredient information and user information, a generative AI model (e.g., GPT-4) is used to generate the optimal recipe.

[0166] Input: Recognized food ingredients and user health and preference information

[0167] Output: The generated recipe

[0168] Specifically, the generative AI model uses prompt text to create multiple recipes that meet the user's requirements and selects the most suitable one.

[0169] Step 6:

[0170] Server: Sends the generated recipe information to the user's smartphone app.

[0171] Input: Generated recipe

[0172] Output: Recipe information sent to the smartphone app

[0173] Specifically, recipe information is sent from the server using a secure protocol such as SSL, and a notification is displayed on the smartphone app.

[0174] Step 7:

[0175] User: Checks the suggested recipe on the smartphone app and makes adjustments as needed.

[0176] Input: Submitted recipe information

[0177] Output: Adjusted recipe request

[0178] Specifically, the user selects a recipe within the app, sets adjustments such as "stronger seasoning" or "add ingredients," and then sends a request to the server again.

[0179] Step 8:

[0180] Server: Based on the adjusted request, the server regenerates the final recipe and the list of required condiments and sends them to the smartphone.

[0181] Input: Adjustment request from user

[0182] Output: Final recipe and seasoning list

[0183] Specifically, the AI ​​model regenerates the recipe, determines the final recipe information reflecting the adjustments, and sends it to the smartphone app.

[0184] Step 9:

[0185] User: Cooks a dish according to the final recipe displayed on the smartphone app.

[0186] Input: Final recipe and seasoning list

[0187] Output: Finished dish

[0188] Specifically, the user follows the steps displayed on the smartphone app and completes the dish using the suggested seasonings.

[0189] (Application example 1)

[0190] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0191] In conventional refrigeration equipment, food ingredients are managed and expiration dates are checked manually, which leads to food waste and makes it difficult for users to receive appropriate recipe suggestions that take into account their health and preferences. Additionally, ordering ingredients that are in short supply is a time-consuming process. This creates problems that lower the quality of life for users.

[0192] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0193] In this invention, the server is located inside the refrigeration equipment and includes an acquisition means for acquiring images, a recognition means for processing the acquired images and automatically recognizing ingredients, a generation means for generating a recipe based on the recognized ingredient information, a transmission means for transmitting the generated recipe to a user terminal, and an ordering means for automatically ordering missing ingredients via the transmission means. This enables more efficient ingredient management, reduced food waste, recipe suggestions that take into account the user's health information and preference information, and automatic ordering of missing ingredients.

[0194] "Capture means" refers to the devices and techniques placed inside the refrigeration unit for capturing images.

[0195] The "recognition means" refers to the technology and algorithms for processing the acquired images and automatically recognizing ingredients.

[0196] The "generation means" refers to software and calculation techniques for generating recipes based on the ingredient information acquired by the recognition means.

[0197] The "transmission means" refers to a communication technique and device for transmitting the recipe generated by the generation means to the user terminal.

[0198] "Ordering means" refers to the system and protocol for automatically ordering missing ingredients via the transmission means.

[0199] "Health information" refers to health data such as the user's number of steps, weight, body fat percentage, and blood pressure.

[0200] "Preference information" refers to data regarding a user's favorite dishes and seasonings.

[0201] A "refrigeration equipment" is a home appliance used to store food at the appropriate temperature.

[0202] A "user terminal" is a mobile information terminal such as a smartphone or tablet operated by a user.

[0203] MODE FOR CARRYING OUT THE INVENTION

[0204] The present invention provides a system for managing ingredients for a user, proposing recipes, and automatically ordering ingredients that are in short supply. The system includes an acquisition unit, a recognition unit, a generation unit, a transmission unit, and an ordering unit that are arranged inside a refrigeration facility.

[0205] System Configuration

[0206] 1. Acquisition method:

[0207] A camera installed inside the refrigeration equipment functions as the acquisition means. This camera takes images of the inside at regular intervals or according to user instructions and uploads them to a server.

[0208] 2. Recognition means:

[0209] The AI ​​image analysis algorithm installed on the server processes the captured image data and automatically recognizes ingredients in the refrigerator, using software libraries such as TensorFlow and OpenCV.

[0210] 3. Generation means:

[0211] Based on the ingredient information stored on the server, a generative AI model is used to generate recipes based on the user's health and preference information.

[0212] 4. Means of transmission:

[0213] The generated recipe is sent to the user's device, which has a smartphone application installed, allowing the user to check the recipe suggestions.

[0214] 5. Ordering Method:

[0215] Based on the ingredient information recognized by the recognition means and the recipe generated by the generation means, the missing ingredients are automatically ordered from a food delivery service. This process uses a RESTful API.

[0216] System Operation Overview

[0217] The system of the present invention operates as follows:

[0218] 1. Images of the inside of the refrigeration equipment are periodically acquired by the acquisition means and uploaded to the server.

[0219] 2. The image data acquired by the recognition means is processed using an AI image analysis algorithm to recognize the ingredients in the refrigeration equipment.

[0220] 3. The recognized food ingredient information is stored on the server and integrated with the user's health and preference information.

[0221] 4. The generation means generates a recipe that takes into account the user's health information (number of steps, weight, etc.) and preference information (favorite dishes, seasonings, etc.).

[0222] 5. The suggested recipes are sent to the user's smartphone via the transmission means, where the user can check them.

[0223] 6. If necessary, any missing ingredients will be automatically ordered from the food delivery service via the ordering method.

[0224] Specific examples

[0225] For example, imagine a refrigerator contains cabbage, carrots, and chicken, but is short on pork and soy sauce. A camera takes a picture of the inside of the refrigerator and uploads the image data to a server. As a result of image analysis, "cabbage," "carrots," and "chicken" are recognized and stored on the server. The user's health information (for example, "Today's steps: 5,000 steps," "Weight: 70 kg," "Preferences: Japanese food") is also sent to the server. Based on this information, a generative AI model is used to suggest recipes such as "Teriyaki Chicken." The missing ingredients (pork and soy sauce) are automatically ordered and delivered to the user's home.

[0226] Prompt Sentence Examples

[0227] "Please suggest a Japanese recipe using cabbage and carrots. The user's health information is: 5,000 steps, weight: 70 kg."

[0228] As described above, by using this system, users can reduce the effort required for managing ingredients, while receiving recipe suggestions that match their health information and preferences, and automatically ordering any ingredients they are missing.

[0229] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0230] Step 1:

[0231] The terminal (a camera inside the refrigeration equipment) captures the image.

[0232] Input: Visual data of the inside of the refrigerator

[0233] Output: Acquired image data

[0234] Specific operation: A camera inside the refrigerator captures images of the inside of the refrigerator at regular intervals or according to the user's instructions. These images are temporarily stored on the device and then uploaded to the server.

[0235] Step 2:

[0236] The image data acquired by the terminal is uploaded to the server.

[0237] Input: Acquired image data

[0238] Output: Uploaded image data

[0239] Specific operation: The device sends image data to the server via an Internet connection, and the server stores the received data in temporary storage.

[0240] Step 3:

[0241] The server processes the image data using an AI image analysis algorithm and automatically recognizes the ingredients inside.

[0242] Input: Uploaded image data

[0243] Output: List of recognized ingredients

[0244] How it works: Using AI image analysis algorithms (e.g., TensorFlow and OpenCV) running on the server, each ingredient in the image is identified. The recognized ingredient information is then stored in a database.

[0245] Step 4:

[0246] The server retrieves the user's health and preference information from the database and integrates it with food ingredient information.

[0247] Input: Recognized ingredient list, user health information, and preference information

[0248] Output: Integrated user and ingredient information

[0249] How it works: The server retrieves the user's health and preference information from the database and combines it with the recognized ingredient list. This combined data is used to generate recipes.

[0250] Step 5:

[0251] The server uses a generative AI model to generate recipes based on the integrated data.

[0252] Input: Integrated user and ingredient information

[0253] Output: The generated recipe

[0254] How it works: The server uses a generative AI model to generate an optimal recipe from the integrated data. This process may involve prompts.

[0255] Step 6:

[0256] The server transmits the generated recipe to the user terminal.

[0257] Input: Generated recipe

[0258] Output: Recipe information sent to the user's device

[0259] Specific operation: The server sends the generated recipe to the user device (smartphone app), which receives it and displays it to the user.

[0260] Step 7:

[0261] The user terminal checks the recipe and places an order for any missing ingredients.

[0262] Input: Recipe information sent to the user's device

[0263] Output: List of missing ingredients and order request

[0264] How it works: The user checks the recipe and checks for missing ingredients through the app. The missing ingredients are automatically recognized and an order request is sent to the food delivery service through the ordering method.

[0265] Through these processing steps, users can efficiently manage ingredients and receive appropriate recipe suggestions. In addition, ingredients that are in short supply are automatically ordered, improving the quality of life.

[0266] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0267] The present invention is a system that equips a refrigeration unit with acquisition means, recognition means, generation means, transmission means, and an emotion engine, and not only manages ingredients and suggests recipes for the user, but also suggests individual recipes that take the user's emotions into consideration.The system aims to suggest more appropriate recipes based on the user's health information, preference information, and current emotional state.

[0268] Overall system configuration

[0269] The system mainly consists of the following components:

[0270] 1. Camera inside the refrigeration equipment (acquisition method)

[0271] 2. Server that performs AI processing (recognition means, generation means, transmission means, emotion engine)

[0272] 3. User's smartphone app (user device)

[0273] System Operation Overview

[0274] 1. Camera inside the refrigeration equipment (acquisition method)

[0275] Terminal (refrigerator): A camera inside the refrigeration equipment captures images of the interior at regular intervals or according to user instructions.

[0276] Terminal (refrigerator): Uploads the acquired image data to the server.

[0277] 2. Image analysis and food ingredient recognition (recognition method)

[0278] Server: The server acquires the received image data and automatically recognizes the ingredients using an AI image analysis algorithm.

[0279] Server: Stores the recognized ingredient information in a database and updates the status inside the refrigeration equipment.

[0280] 3. Recipe generation (generation method)

[0281] Server: Generates optimal recipes based on the user's health information (number of steps, weight, age, etc.), preference information (favorite dishes and seasonings, etc.), and emotional information.

[0282] Server: For example, when a user is feeling stressed, the server suggests recipes that will help them relax.

[0283] 4. Emotion Recognition (Emotion Engine)

[0284] User: Emotions are captured from the user's voice, facial expressions, and text input via a smartphone app or a camera or microphone built into the refrigeration equipment display.

[0285] Server: Analyzes the acquired emotion data and identifies the user's current emotional state.

[0286] 5. Notification of information and recipe suggestions (transmission method)

[0287] Server: Sends the generated recipe to the user's smartphone app.

[0288] Users can view suggested recipes through a smartphone app.

[0289] Specific examples

[0290] For example, consider a case where cabbage, carrots, and chicken are stored in a refrigeration facility.

[0291] 1. Terminal (refrigerator): The camera takes pictures of the inside of the refrigerator and uploads the image data to the server.

[0292] 2. Server: Analyzes the uploaded image and automatically recognizes "cabbage," "carrot," and "chicken."

[0293] 3. User: Enters the following information into the smartphone app: "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food." At this time, the built-in camera and microphone determine that the user is feeling a little stressed.

[0294] 4. Server: Integrates the ingredient list, health information, preference information, and emotional information to suggest dishes like "teriyaki chicken" and "Japanese-style cabbage and carrot salad," as well as "herbal tea" for relaxation.

[0295] 5. User: Selects "Teriyaki Chicken" from the recipes suggested by the smartphone app and makes adjustments such as "making it stronger in seasoning."

[0296] 6. Server: Generates the fine-tuned recipe and sends the final cooking instructions and required seasoning list to the user's smartphone.

[0297] 7. User: Cooks food according to the recipe displayed on the smartphone app.

[0298] In this way, the present invention is a system that efficiently utilizes ingredients stored in refrigeration equipment and suggests optimal recipes based on the user's health information, preference information, and emotional state, helping to reduce food waste and solving daily meal problems.

[0299] The processing flow will be explained below.

[0300] Step 1:

[0301] Terminal (refrigerator): The camera inside the refrigerator takes pictures of the inside at a set time. For example, if it is set to take pictures every day at 10:00, the camera will automatically operate at that time.

[0302] Step 2:

[0303] Terminal (refrigerator): The acquired image data is sent to the server via the internal storage device of the refrigeration equipment or the network.

[0304] Step 3:

[0305] Server: The server takes the received image data and begins processing it using AI image analysis algorithms, specifically detecting objects in the image and identifying which ingredients they correspond to.

[0306] Step 4:

[0307] Server: Stores the recognized ingredient information in a database. Newly recognized ingredients and updated information on existing ingredients are reflected in the database.

[0308] Step 5:

[0309] User: Opens the smartphone app and enters health and preference information. For example, enter "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food."

[0310] Step 6:

[0311] Terminal (smartphone): The entered health information and preference information is sent to the server.

[0312] Step 7:

[0313] User: Using a smartphone app and the built-in camera and microphone in the refrigerator display, emotional information is acquired from voice and facial expressions. For example, emotions such as "I'm tired" or "I want to relax" are analyzed.

[0314] Step 8:

[0315] Server: Using the emotion engine, analyze the acquired emotion data and identify the user's current emotional state.

[0316] Step 9:

[0317] Server: Integrates the acquired information on ingredients, health, preferences, and emotions to generate optimal recipes. For example, a user feeling stressed might be suggested a recipe such as "herbal tea" that has a relaxing effect.

[0318] Step 10:

[0319] Server: Sends the generated recipe to the user's smartphone.

[0320] Step 11:

[0321] User: Check the recipes displayed on the smartphone app and select the desired recipe. For example, select "Teriyaki Chicken."

[0322] Step 12:

[0323] User: Requests seasoning or portion adjustments for a selected recipe. For example, they might type, "I'd like it a little stronger today."

[0324] Step 13:

[0325] Server: Fine-tunes the recipe data based on the user's request and generates the final cooking steps and list of required seasonings.

[0326] Step 14:

[0327] Server: Sends the fine-tuned recipe and cooking instructions to the user's smartphone.

[0328] Step 15:

[0329] User: Cooks a dish by following the cooking instructions displayed on the smartphone app.

[0330] Step 16:

[0331] Server: Regularly checks the food database in the refrigeration facility to identify food items that are close to their expiration date.

[0332] Step 17:

[0333] Server: Generates recipes that prioritize ingredients with an approaching expiration date and sends notification information to the user's smartphone.

[0334] Step 18:

[0335] Terminal (refrigerator): Notifications regarding expiration dates are displayed on the refrigerator equipment display and on the user's smartphone app.

[0336] Step 19:

[0337] User: Check the notification, use the suggested recipes as a guide, and prioritize ingredients that are close to their expiration date.

[0338] Through these steps, users can efficiently utilize the food stored in their refrigerators, helping them manage their health and reduce food waste.

[0339] Example 2

[0340] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0341] Today, there is a demand for efficient use of ingredients in refrigeration equipment, reducing food waste, and individually optimized recipe suggestions based on the user's health information, preferences, and even emotional state. However, conventional systems do not adequately consider the user's emotional state when managing ingredients or suggesting recipes, resulting in insufficient improvement in user satisfaction. The present invention aims to solve these problems by suggesting optimal recipes based on the user's health, preferences, and emotional state.

[0342] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server is disposed inside the refrigeration equipment and includes an acquisition means for acquiring images, a recognition means for processing the acquired images and automatically recognizing ingredients, a generation means for generating recipes based on ingredient information recognized by the recognition means, an emotion recognition means for analyzing user emotion information and generating recipes based on the emotional state, and a transmission means for transmitting the recipes generated by the generation means to a user terminal. This makes it possible to efficiently utilize ingredients in the refrigeration equipment and propose individually optimized recipes based on the user's health, preferences, and emotional state.

[0343] "Acquisition means" is a general term for hardware such as cameras and sensors that are placed inside the refrigeration equipment to acquire image data, and the software that controls them.

[0344] "Recognition means" is a general term for AI algorithms and software processing that processes image data acquired by the acquisition means and automatically identifies ingredients.

[0345] The "generation means" is a general term for programs and algorithms for generating recipes based on the ingredient information recognized by the recognition means and other related data.

[0346] "Emotion recognition means" is a general term for hardware and software that acquires emotional data from a user's voice, facial expression, text input, etc., and analyzes it to identify the user's emotional state.

[0347] The "transmission means" is a general term for the communication function for transmitting the recipe and notification information generated by the generation means to the user's terminal and the program that controls it.

[0348] A "user terminal" is a device that a user operates and views, and includes a smartphone, tablet, PC, etc.

[0349] MODE FOR CARRYING OUT THE INVENTION

[0350] The present invention is a system for managing ingredients and suggesting recipes to users by equipping a refrigeration unit with an acquisition means, a recognition means, a generation means, a transmission means, and an emotion recognition means. This system uses hardware and software such as a camera inside the refrigeration unit as an acquisition means, an AI image analysis algorithm as a recognition means, a machine learning model and a rule-based algorithm as a generation means, a communication function as a transmission means, and a voice recognition and facial expression analysis algorithm as an emotion recognition means.

[0351] Hardware and Software Use

[0352] 1. Camera inside the refrigeration equipment (acquisition method)

[0353] Terminal (refrigerator): A camera installed inside the refrigeration equipment captures images of the inside periodically or at the user's command. For example, the camera saves the image data in JPEG format and uploads it to a server via Wi-Fi.

[0354] 2. Image analysis and food ingredient recognition (recognition method)

[0355] Server: The server processes the received image data using an AI image analysis algorithm using TensorFlow and OpenCV to automatically recognize ingredients. The recognized ingredient information is stored in a database in JSON format.

[0356] 3. Emotion recognition (emotion recognition means)

[0357] User: Emotional data is collected from facial expressions, voice, and text input using a smartphone app or a camera or microphone built into the refrigeration equipment display. For example, emotional data is input by answering questions within the app.

[0358] Server: The emotion recognition engine analyzes the acquired data using natural language processing algorithms to identify the user's current emotional state, and records the results in a database.

[0359] 4. Recipe generation (generation method)

[0360] Server: The server integrates the user's health information (e.g., number of steps and weight), preference information (e.g., favorite dishes and seasonings), and emotional information to generate optimal recipes using machine learning models and rule-based algorithms. For example, if the user is feeling stressed, it will suggest recipes with a relaxing effect.

[0361] 5. Notification of information and recipe suggestions (transmission method)

[0362] Server: The generated recipe is sent to the user's device in JSON or XML format.

[0363] User: Using the smartphone app, view the suggested recipes and make adjustments to finalize the recipe. Recipe details and cooking instructions are displayed in the app.

[0364] Examples of specific examples and prompts

[0365] For example, imagine a refrigerator containing cabbage, carrots, and chicken. The refrigerator's camera takes a picture of the interior at 8:00 AM and uploads it to the server as "fridge_image_20230315_0800.jpg." The server uses TensorFlow to recognize "cabbage," "carrot," and "chicken" and stores them in a database. The user enters "Today's steps: 5,000," "Weight: 70 kg," and "Preferences: Japanese food" into a smartphone app, and the built-in camera and microphone determine that the user is "feeling a little stressed." Based on this information, the server suggests "teriyaki chicken," "Japanese-style cabbage and carrot salad," and "herbal tea" for relaxation. The user then adjusts and confirms the recipe.

[0366] Prompt Sentence Examples

[0367] "The ingredients stored in the refrigerator are cabbage, carrots, and chicken. The user likes Japanese food, has walked 5,000 steps today, and weighs 70 kg. The user is also feeling a little stressed. Based on this information, please suggest a Japanese recipe that will have a relaxing effect."

[0368] As described above, the present invention is a system that efficiently manages ingredients in a refrigeration facility and provides individually optimized recipes that take into account the user's health information, preference information, and even emotional state.

[0369] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0370] Step 1: Acquire and upload images

[0371] Terminal (refrigerator):

[0372] A camera inside the refrigeration equipment periodically captures images of the interior, for example, at 8:00 a.m. every day. Users can also manually instruct the camera to take pictures.

[0373] Input: User instructions or set scheduled time

[0374] Output: JPEG format image file

[0375] The captured image is saved as "fridge_image_yyyyMMdd_HHmm.jpg" and uploaded to the server via Wi-Fi.

[0376] Step 2: Image analysis and ingredient recognition

[0377] server:

[0378] The image data received by the server is stored in " / images / received / ".

[0379] Input: Uploaded JPEG image file

[0380] Processing: Recognize ingredients in the image using image analysis algorithms based on TensorFlow and OpenCV

[0381] Output: Ingredient information in JSON format

[0382] The recognized ingredient information is stored in the "food_items" table. For example, "cabbage," "carrot," and "chicken" are identified.

[0383] Step 3: Emotion Recognition

[0384] User:

[0385] Facial expressions, voice, and text input are provided via a smartphone app or a camera and microphone built into the refrigeration equipment display.

[0386] Input: Voice data, facial expression images, text data

[0387] Output: Emotion data

[0388] server:

[0389] An emotion recognition engine analyzes the input data and uses natural language processing algorithms to identify the user's emotional state.

[0390] Input: Voice data, facial expression images, text data

[0391] Processing: Emotional data analysis

[0392] Output: Emotional state (e.g., "I'm feeling a little stressed")

[0393] The analysis results are recorded in the "user_emotions" table.

[0394] Step 4: Recipe Generation

[0395] server:

[0396] The system integrates the user's health information, preference information, emotional information, and information on ingredients in the refrigerator to generate optimal recipes.

[0397] Input: Health information (number of steps, weight), preference information, emotional information, information on ingredients in the refrigerator

[0398] Processing: Generate recipes using machine learning models and rule-based algorithms

[0399] Output: Suggested recipe

[0400] For example, if a user is feeling stressed, the app will suggest relaxing dishes such as "teriyaki chicken," "Japanese-style cabbage and carrot salad," and even "herbal tea."

[0401] Step 5: Information and recipe suggestions

[0402] server:

[0403] The generated recipe is sent to the user's device in JSON or XML format.

[0404] Input: The generated recipe and other relevant information

[0405] Output: Data sent to the user terminal

[0406] User:

[0407] The suggested recipe is viewed on a smartphone app, and adjustments are made as needed. Finally, the final recipe is confirmed.

[0408] Input: Recipe information sent from the server

[0409] Output: Finalized recipe and cooking instructions

[0410] For example, if a refrigerator contains cabbage, carrots, and chicken, the refrigerator's camera periodically captures images and uploads them to a server. The server then analyzes the images and recognizes the ingredients. The user provides health and emotional information through a smartphone app, and the server then suggests recipes with a relaxing effect. Finally, the user can view and adjust the recipe and cook the food.

[0411] Example prompt sentence:

[0412] "The ingredients stored in the refrigerator are cabbage, carrots, and chicken. The user likes Japanese food, has walked 5,000 steps today, and weighs 70 kg. The user is also feeling a little stressed. Based on this information, please suggest a Japanese recipe that will have a relaxing effect."

[0413] (Application example 2)

[0414] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0415] For modern consumers, managing ingredients in refrigerated facilities and proposing optimal recipes based on that information is a very time-consuming task. Managing expiration dates for ingredients, creating recipes based on that information, and quickly sourcing missing ingredients are also challenges. Furthermore, most existing systems do not offer recipe suggestions that take into account the user's health information or emotional state. This situation could exacerbate food waste and health management issues. The present invention aims to solve these challenges by providing a system that manages ingredients, proposes recipes, automatically orders ingredients, and provides virtual experiences.

[0416] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0417] In this invention, the server includes an acquisition means for acquiring images, a recognition means for processing the acquired images and automatically recognizing ingredients, a generation means for generating a recipe based on the ingredient information recognized by the recognition means, a transmission means for transmitting the recipe generated by the generation means to a user terminal, an ordering means for automatically ordering missing ingredients, and a virtual reality means for providing a virtual experience, thereby enabling the user to efficiently manage ingredients in the refrigeration equipment, receive optimal recipe suggestions based on health information and emotional state, quickly order missing ingredients, and enjoy an intuitive virtual shopping experience.

[0418] Definition of Terms

[0419] "Means for acquiring images" refers to cameras or sensors that take images of the inside of the refrigeration equipment and acquire them as digital data.

[0420] "Means for processing acquired images and automatically recognizing ingredients" refers to a device or software that analyzes acquired image data and identifies the ingredients contained therein using AI or machine learning algorithms.

[0421] "Generation means for generating a recipe based on the ingredient information recognized by the recognition means" refers to a device or software that creates an optimal recipe based on the ingredient information obtained by the recognition means, taking into consideration the user's health information, preference information, emotional state, etc.

[0422] "Transmission means for transmitting the recipe generated by the generation means to a user terminal" refers to a device or software that transmits the generated recipe to a terminal such as a user's smartphone or tablet via a network.

[0423] "Ordering means for automatically ordering missing ingredients" means a device or software that automatically orders missing ingredients online based on the generated recipe.

[0424] "Virtual reality means for providing a virtual experience" means a device or software that provides a virtual reality (VR) or augmented reality (AR) environment that can be intuitively operated by a user.

[0425] "Health information" refers to physical data such as the user's number of steps, weight, age, etc., and particularly includes data related to dietary habits.

[0426] "Preference information" refers to information about a user's favorite dishes and seasonings, and particularly includes data that reflects an individual's food preferences.

[0427] "Emotional state" refers to the mental state of the user and includes data for inferring emotions from voice, facial expressions, text input, etc.

[0428] MODE FOR CARRYING OUT THE INVENTION

[0429] This invention realizes a system that uses an AI model to recognize ingredients based on images captured by a camera inside the refrigeration equipment, and then generates and suggests recipes to the user. It also has the ability to suggest more personalized recipes and automatically order missing ingredients by taking into account the user's health information and emotional state. Below, we will explain each element of this system in detail.

[0430] Hardware and software used

[0431] Hardware: Cameras in refrigeration equipment, smartphones, server PCs, smart glasses

[0432] Software: Python, Flask, TensorFlow, OpenAI API

[0433] System configuration and operation

[0434] 1. Camera inside the refrigeration equipment (acquisition method)

[0435] Cameras placed inside the refrigeration equipment periodically or at the user's command capture images of the interior, which are then uploaded as digital data to a server.

[0436] 2. Image analysis and food ingredient recognition (recognition method)

[0437] The server analyzes the image data received from the acquisition means and automatically recognizes ingredients using an AI model trained using TensorFlow.

[0438] 3. Recipe generation (generation method)

[0439] The server generates the most suitable recipe based on the ingredient information identified by the recognition means, taking into consideration the user's health information (e.g., number of steps, weight) and preference information (e.g., favorite dishes, seasonings). Furthermore, to take the user's emotional state into account, it uses emotional information obtained by analyzing their voice and facial expressions. At this point, the generated recipe is sent to the smartphone app.

[0440] 4. Automatic ordering of missing ingredients (ordering method)

[0441] Based on the generated recipe, any missing ingredients are automatically ordered online, allowing users to quickly get the ingredients they need.

[0442] 5. Virtual Experience (Virtual Reality Methods)

[0443] Users can enjoy an intuitive virtual experience using smart glasses or a smartphone, making refrigerator food management and online shopping even more convenient.

[0444] Specific examples

[0445] For example, imagine a refrigerator containing cabbage, carrots, and chicken. A camera takes images of these ingredients and uploads them to a server. The server recognizes these ingredients and compares them with the user's health information (e.g., 5,000 steps today, weight 70 kg) and preference information (e.g., likes Japanese food) against the input data. It also takes into account the user's emotional state, as determined from their voice and facial expressions (e.g., feeling a little stressed), and generates the most suitable recipe (e.g., teriyaki chicken, Japanese-style cabbage and carrot salad, and relaxing herbal tea).

[0446] The generated recipe is sent to a smartphone app, where users can use it to create the dish, and if any ingredients are missing, the system will automatically order them online and have them available immediately.

[0447] Prompt Sentence Examples

[0448] "Ingredients in the refrigerator: [cabbage, carrots, chicken]. Health information: {"steps": 5000, "weight": 70kg}. Preference information: "I like Japanese food, and I like it strong." Emotional state: "I'm feeling a little stressed." Please suggest recipes based on this."

[0449] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0450] Processing steps of the system that realizes the application example

[0451] Step 1:

[0452] The terminal (a camera inside the refrigerator) periodically captures images of the inside of the refrigerator or when instructed by the user. This image data is captured and saved in digital format. The input is the current state of the inside of the refrigerator, and the output is the digital image data.

[0453] Step 2:

[0454] The image data acquired by the device is uploaded to the server. The input here is the image data acquired in step 1, and the output is the image data transferred to the server.

[0455] Step 3:

[0456] The server processes the uploaded image data and automatically recognizes ingredients using an AI image recognition algorithm. A model using TensorFlow analyzes the image. The image data is given as input, and the output is the recognized ingredient information.

[0457] Step 4:

[0458] The recognized ingredient information is stored in a database on the server, and the status inside the refrigeration equipment is updated. The input is the recognized ingredient information, and the output is the updated database information.

[0459] Step 5:

[0460] Users use a smartphone app to input health information (e.g., number of steps, weight) and preference information (e.g., favorite food). In addition, emotional information is acquired through the built-in camera and microphone. The user's health information, preference information, and emotional information are given as input, and the output is the integrated data.

[0461] Step 6:

[0462] The server generates the optimal recipe based on the integrated information (ingredient information, health information, preference information, and emotional information). In this process, recipe generation is performed based on prompts using OpenAI's API. The input is the integrated data, and the output is the generated recipe.

[0463] Step 7:

[0464] The generated recipe is sent to the user device (smartphone app). The input is the generated recipe, and the output is the recipe information displayed on the user device.

[0465] Step 8:

[0466] If necessary, the user can automatically order missing ingredients online based on the generated recipe. The input is the missing ingredient information, and the output is the ingredients obtained by online ordering.

[0467] Step 9:

[0468] Users use smart glasses or smartphones to control refrigeration equipment and enjoy virtual experiences, including ingredient management, recipe suggestions, and even virtual shopping. The input is the user's operation, and the output is the user experience.

[0469] Through these steps, the system will be able to provide comprehensive services, from ingredient management to recipe suggestions, online ordering of missing ingredients, and virtual experiences.

[0470] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0471] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0472] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0473] [Second embodiment]

[0474] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0475] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0476] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0477] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0478] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0479] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0480] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0481] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0482] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0483] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0484] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0485] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0486] The present invention is a system that manages ingredients and suggests recipes for users using acquisition means, recognition means, generation means, and transmission means installed in refrigeration equipment. The system aims to suggest efficient and healthy meals by taking into account the user's health information and preference information.

[0487] Overall system configuration

[0488] The system mainly consists of the following components:

[0489] 1. Camera inside the refrigeration equipment (acquisition method)

[0490] 2. Server that performs AI processing (recognition, generation, and transmission means)

[0491] 3. User's smartphone app (user device)

[0492] System Operation Overview

[0493] 1. Camera inside the refrigeration equipment (acquisition method)

[0494] Terminal (refrigeration equipment): A camera inside the refrigeration equipment captures images of the inside at regular intervals or according to user instructions.

[0495] Terminal (refrigeration equipment): Uploads the acquired image data to the server.

[0496] 2. Image analysis and food ingredient recognition (recognition method)

[0497] Server: Receives the acquired image data and automatically recognizes the ingredients inside using an AI image analysis algorithm.

[0498] Server: Stores the recognized ingredient information in a database and updates the internal status of the refrigeration equipment.

[0499] 3. Recipe generation (generation method)

[0500] Server: Generates optimal recipes based on the user's health information (number of steps, weight, age, etc.) and preference information (favorite dishes and seasonings, etc.).

[0501] Server: Using AI technology, it suggests cooking recipes using recognized ingredients.

[0502] 4. Notification of information and recipe suggestions (transmission method)

[0503] Server: Sends the generated recipe to the user's smartphone app.

[0504] Users can view suggested recipes through a smartphone app.

[0505] Specific examples

[0506] For example, consider a case where cabbage, carrots, and chicken are stored in a refrigeration facility.

[0507] 1. Terminal (refrigeration equipment): The camera takes pictures of the inside of the refrigerator and uploads the image data to the server.

[0508] 2. Server: Analyzes the uploaded image and automatically recognizes "cabbage," "carrot," and "chicken."

[0509] 3. User: Enters the following information into the smartphone app: "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food." This information is sent to the server.

[0510] 4. Server: Integrates the ingredient list with user information and generates recipe suggestions such as "Teriyaki Chicken" or "Japanese-style Cabbage and Carrot Salad."

[0511] 5. User: Selects "Teriyaki Chicken" from the recipes suggested by the smartphone app and makes adjustments such as "making it stronger in seasoning."

[0512] 6. Server: Generates the fine-tuned recipe and sends the final cooking instructions and required seasoning list to the user's smartphone.

[0513] 7. User: Can cook food according to the recipe displayed on the smartphone app.

[0514] In this way, the present invention is a system that efficiently utilizes ingredients stored in refrigeration equipment and suggests optimal recipes based on the user's health and preference information, helping to reduce food waste and solving daily meal problems.

[0515] The processing flow will be explained below.

[0516] Step 1:

[0517] Terminal (refrigerator): The camera inside the refrigerator takes pictures of the inside at a set time. For example, if it is set to take pictures every day at 10:00, the camera will automatically operate at that time.

[0518] Step 2:

[0519] Terminal (refrigerator): The acquired image data is sent to the server via the internal storage device of the refrigeration equipment or the network.

[0520] Step 3:

[0521] Server: The server takes the received image data and begins processing it using AI image analysis algorithms, specifically detecting objects in the image and identifying which ingredients they correspond to.

[0522] Step 4:

[0523] Server: Stores the recognized ingredient information in a database. Newly recognized ingredients and updated information on existing ingredients are reflected in the database.

[0524] Step 5:

[0525] User: Opens the smartphone app and enters health and preference information. For example, enter "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food."

[0526] Step 6:

[0527] Terminal (smartphone): The entered health information and preference information is sent to the server.

[0528] Step 7:

[0529] Server: Integrates the acquired ingredient information with the user's health and preference information to generate optimal recipes. For example, it selects a Japanese recipe using cabbage, carrots, and chicken.

[0530] Step 8:

[0531] Server: Sends the generated recipe to the user's smartphone via the network.

[0532] Step 9:

[0533] User: Check the recipes displayed on the smartphone app and select the desired recipe. For example, select "Teriyaki Chicken."

[0534] Step 10:

[0535] User: Requests seasoning or portion adjustments for a selected recipe. For example, they might type, "I'd like it a little stronger today."

[0536] Step 11:

[0537] Server: Fine-tunes the recipe data based on the user's request and generates the final cooking steps and list of required seasonings.

[0538] Step 12:

[0539] Server: Sends the fine-tuned recipe and cooking instructions to the user's smartphone.

[0540] Step 13:

[0541] User: Cooks a dish by following the cooking instructions displayed on the smartphone app.

[0542] Step 14:

[0543] Server: Regularly checks the food database in the refrigeration facility to identify food items that are close to their expiration date.

[0544] Step 15:

[0545] Server: Generates recipes that prioritize ingredients with an approaching expiration date and sends notification information to the user's smartphone.

[0546] Step 16:

[0547] Terminal (refrigerator): Notifications regarding expiration dates are displayed on the refrigerator equipment display and on the user's smartphone app.

[0548] Step 17:

[0549] User: Check the notification, use the suggested recipes as a guide, and prioritize ingredients that are close to their expiration date.

[0550] Through these steps, users can efficiently utilize the food stored in their refrigerators, helping them manage their health and reduce food waste.

[0551] Example 1

[0552] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0553] In modern life, efficiently managing food stored in refrigerators and preparing healthy meals that suit individual tastes is a time-consuming and labor-intensive task. Conventional refrigerators lack the functionality to suggest optimal dishes based on the expiration date and type of ingredients, and it requires a great deal of effort for users to understand the condition of ingredients and select recipes based on health information and preferences. This has led to increased food waste and unhealthy food choices.

[0554] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0555] In this invention, the server includes an acquisition means, a recognition means, a generation means, a transmission means, and a user terminal. This allows for efficient management of ingredients in the refrigeration unit and suggests optimal recipes based on the user's health and preference information. Specifically, a camera installed inside the refrigeration unit captures images of ingredients and uploads them to the server. The server automatically recognizes ingredients from the uploaded images using an image analysis algorithm based on deep learning and stores the information in a database. The server then receives the user's health and preference information and uses a generative AI model to generate an optimal recipe based on this information. The generated recipe is then sent to the user's smartphone app, where the user can review the suggested recipe and make adjustments as needed. This makes it easier for users to prepare healthy meals that suit their preferences, contributing to reducing food waste.

[0556] The "acquisition means" is a device for acquiring image data using a camera or the like inside the refrigeration equipment and transmitting the image data to the server.

[0557] The "recognition means" refers to an AI image analysis algorithm and its processing device that analyzes image data uploaded to the server and automatically identifies the ingredients inside.

[0558] The "generation means" refers to an algorithm and a processing device for generating an optimal recipe based on the ingredient information identified by the recognition means and the user's health information and preference information.

[0559] The "transmission means" refers to a communication device and its protocol for transmitting the recipe information generated by the generation means to a user terminal such as a smartphone app of the user.

[0560] A "user terminal" is an electronic device such as a smartphone or tablet used by a user, and is used to receive, display, and adjust suggested recipe information.

[0561] "Health Information" refers collectively to a User's weight, number of steps, age, and other health-related data.

[0562] "Preference information" is a general term for information about the types of food and seasonings that a user likes.

[0563] A "generative AI model" is an advanced artificial intelligence algorithm, such as GPT-4, that generates new information, especially recipe information, based on input data.

[0564] A "prompt sentence" is an input sentence used to cause an AI model to generate a particular output.

[0565] This invention is a system that manages ingredients and suggests recipes to users using acquisition means, recognition means, generation means, and transmission means installed in refrigeration equipment. This system efficiently manages ingredients in the refrigerator and provides optimal recipes based on the user's health information and preference information, thereby helping to reduce food waste and promote healthy eating habits.

[0566] Hardware and Software Configuration

[0567] 1. Camera inside the refrigeration equipment (acquisition method)

[0568] Terminal (refrigeration equipment): A camera is installed inside the refrigerator and takes images of the inside of the refrigerator at regular intervals or according to the user's instructions. This camera uploads the image data to a server via an internet connection.

[0569] 2. Image analysis and food ingredient recognition (recognition method)

[0570] Server: The server receives the uploaded image data and automatically recognizes the ingredients in the refrigerator using an image analysis algorithm that uses deep learning. This algorithm uses a deep learning framework such as TensorFlow. The recognized ingredient information is stored in a database and the internal status of the refrigerator is updated in real time.

[0571] 3. Recipe generation (generation method)

[0572] Server: The server receives health information (number of steps, weight, age, etc.) and preference information (preferred cuisine, seasonings, etc.) from the user's smartphone app. Based on this, it uses a generative AI model (e.g., GPT-4) to generate the optimal recipe for the user. In this process, recipes are created taking into account the priority of ingredient use and nutritional balance.

[0573] 4. Notification of information and recipe suggestions (transmission method)

[0574] Server: Sends the generated recipe information to the user's smartphone app, where the user can review the suggested recipe and make adjustments as needed.

[0575] Specific examples

[0576] For example, if cabbage, carrots, and chicken are stored in the refrigerator, the following processing will occur:

[0577] 1. Terminal (refrigeration equipment): The camera takes pictures of the inside of the refrigerator and uploads the image data to the server.

[0578] 2. Server: Analyzes the uploaded image and automatically recognizes "cabbage," "carrot," and "chicken."

[0579] 3. User: Enters "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food" into the smartphone app and sends this information to the server.

[0580] 4. Server: Integrates the recognized ingredient information with the user's health information and generates recipe suggestions such as "Teriyaki Chicken" or "Japanese-style Cabbage and Carrot Salad."

[0581] 5. User: Check the suggested recipes on the smartphone app, select "Teriyaki Chicken," and make adjustments such as "Make it stronger in seasoning."

[0582] 6. Server: Regenerates the adjusted recipe and sends the final cooking instructions and required seasoning list to your smartphone.

[0583] 7. User: Can cook food according to the recipe displayed on the smartphone app.

[0584] Prompt Sentence Examples

[0585] For example, you can generate a recipe by inputting the following prompt into a generative AI model:

[0586] Prompt statement:

[0587] There are "cabbage," "carrots," and "chicken" in the refrigerator. The user's health information is as follows: "Today's steps: 5,000," "Weight: 70 kg," and "Preferences: Japanese food." Based on this, please suggest a healthy and delicious recipe.

[0588] output:

[0589] 1. Teriyaki chicken

[0590] 2. Japanese-style cabbage and carrot salad

[0591] In this way, the present invention realizes a system that can support a healthy diet by effectively utilizing ingredients stored in refrigeration equipment and providing optimal recipes based on the user's health and preference information.

[0592] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0593] Step 1:

[0594] Terminal (refrigeration equipment): A camera installed inside the refrigerator takes pictures of the inside of the refrigerator at regular intervals or according to the user's instructions.

[0595] Input: User instruction or scheduled trigger

[0596] Output: Image data of the inside of the refrigerator

[0597] Specifically, the camera captures high-resolution images from different angles inside the refrigerator, making all the ingredients inside visible.

[0598] Step 2:

[0599] Terminal (refrigeration equipment): Uploads the acquired image data to a server via an internet connection.

[0600] Input: Image data of the inside of the refrigerator

[0601] Output: Image data sent to the server

[0602] Specifically, the image data is compressed and transferred to a server using a secure protocol.

[0603] Step 3:

[0604] Server: Receives uploaded image data and performs image analysis using a deep learning framework (e.g., TensorFlow).

[0605] Input: Uploaded image data of the inside of the refrigerator

[0606] Output: Recognized ingredients

[0607] Specifically, the server runs an object detection algorithm to identify the type and location of ingredients in the image and stores this information in a database.

[0608] Step 4:

[0609] User: Enters health information (e.g., number of steps, weight, age) and preference information (e.g., favorite food, seasoning) into the smartphone app.

[0610] Input: User health and preference information

[0611] Output: User information sent to the server

[0612] Specifically, the user uses a smartphone app to input information using an intuitive interface, and then presses the send button to transfer the information to the server.

[0613] Step 5:

[0614] Server: Based on the collected ingredient information and user information, a generative AI model (e.g., GPT-4) is used to generate the optimal recipe.

[0615] Input: Recognized food ingredients and user health and preference information

[0616] Output: The generated recipe

[0617] Specifically, the generative AI model uses prompt text to create multiple recipes that meet the user's requirements and selects the most suitable one.

[0618] Step 6:

[0619] Server: Sends the generated recipe information to the user's smartphone app.

[0620] Input: Generated recipe

[0621] Output: Recipe information sent to the smartphone app

[0622] Specifically, recipe information is sent from the server using a secure protocol such as SSL, and a notification is displayed on the smartphone app.

[0623] Step 7:

[0624] User: Checks the suggested recipe on the smartphone app and makes adjustments as needed.

[0625] Input: Submitted recipe information

[0626] Output: Adjusted recipe request

[0627] Specifically, the user selects a recipe within the app, sets adjustments such as "stronger seasoning" or "add ingredients," and then sends a request to the server again.

[0628] Step 8:

[0629] Server: Based on the adjusted request, the server regenerates the final recipe and the list of required condiments and sends them to the smartphone.

[0630] Input: Adjustment request from user

[0631] Output: Final recipe and seasoning list

[0632] Specifically, the AI ​​model regenerates the recipe, determines the final recipe information reflecting the adjustments, and sends it to the smartphone app.

[0633] Step 9:

[0634] User: Cooks a dish according to the final recipe displayed on the smartphone app.

[0635] Input: Final recipe and seasoning list

[0636] Output: Finished dish

[0637] Specifically, the user follows the steps displayed on the smartphone app and completes the dish using the suggested seasonings.

[0638] (Application example 1)

[0639] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0640] In conventional refrigeration equipment, food ingredients are managed and expiration dates are checked manually, which leads to food waste and makes it difficult for users to receive appropriate recipe suggestions that take into account their health and preferences. Additionally, ordering ingredients that are in short supply is a time-consuming process. This creates problems that lower the quality of life for users.

[0641] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0642] In this invention, the server is located inside the refrigeration equipment and includes an acquisition means for acquiring images, a recognition means for processing the acquired images and automatically recognizing ingredients, a generation means for generating a recipe based on the recognized ingredient information, a transmission means for transmitting the generated recipe to a user terminal, and an ordering means for automatically ordering missing ingredients via the transmission means. This enables more efficient ingredient management, reduced food waste, recipe suggestions that take into account the user's health information and preference information, and automatic ordering of missing ingredients.

[0643] "Capture means" refers to the devices and techniques placed inside the refrigeration unit for capturing images.

[0644] The "recognition means" refers to the technology and algorithms for processing the acquired images and automatically recognizing ingredients.

[0645] The "generation means" refers to software and calculation techniques for generating recipes based on the ingredient information acquired by the recognition means.

[0646] The "transmission means" refers to a communication technique and device for transmitting the recipe generated by the generation means to the user terminal.

[0647] "Ordering means" refers to the system and protocol for automatically ordering missing ingredients via the transmission means.

[0648] "Health information" refers to health data such as the user's number of steps, weight, body fat percentage, and blood pressure.

[0649] "Preference information" refers to data regarding a user's favorite dishes and seasonings.

[0650] A "refrigeration equipment" is a home appliance used to store food at the appropriate temperature.

[0651] A "user terminal" is a mobile information terminal such as a smartphone or tablet operated by a user.

[0652] MODE FOR CARRYING OUT THE INVENTION

[0653] The present invention provides a system for managing ingredients for a user, proposing recipes, and automatically ordering ingredients that are in short supply. The system includes an acquisition unit, a recognition unit, a generation unit, a transmission unit, and an ordering unit that are arranged inside a refrigeration facility.

[0654] System Configuration

[0655] 1. Acquisition method:

[0656] A camera installed inside the refrigeration equipment functions as the acquisition means. This camera takes images of the inside at regular intervals or according to user instructions and uploads them to a server.

[0657] 2. Recognition means:

[0658] The AI ​​image analysis algorithm installed on the server processes the captured image data and automatically recognizes ingredients in the refrigerator, using software libraries such as TensorFlow and OpenCV.

[0659] 3. Generation means:

[0660] Based on the ingredient information stored on the server, a generative AI model is used to generate recipes based on the user's health and preference information.

[0661] 4. Means of transmission:

[0662] The generated recipe is sent to the user's device, which has a smartphone application installed, allowing the user to check the recipe suggestions.

[0663] 5. Ordering Method:

[0664] Based on the ingredient information recognized by the recognition means and the recipe generated by the generation means, the missing ingredients are automatically ordered from a food delivery service. This process uses a RESTful API.

[0665] System Operation Overview

[0666] The system of the present invention operates as follows:

[0667] 1. Images of the inside of the refrigeration equipment are periodically acquired by the acquisition means and uploaded to the server.

[0668] 2. The image data acquired by the recognition means is processed using an AI image analysis algorithm to recognize the ingredients in the refrigeration equipment.

[0669] 3. The recognized food ingredient information is stored on the server and integrated with the user's health and preference information.

[0670] 4. The generation means generates a recipe that takes into account the user's health information (number of steps, weight, etc.) and preference information (favorite dishes, seasonings, etc.).

[0671] 5. The suggested recipes are sent to the user's smartphone via the transmission means, where the user can check them.

[0672] 6. If necessary, any missing ingredients will be automatically ordered from the food delivery service via the ordering method.

[0673] Specific examples

[0674] For example, imagine a refrigerator contains cabbage, carrots, and chicken, but is short on pork and soy sauce. A camera takes a picture of the inside of the refrigerator and uploads the image data to a server. As a result of image analysis, "cabbage," "carrots," and "chicken" are recognized and stored on the server. The user's health information (for example, "Today's steps: 5,000 steps," "Weight: 70 kg," "Preferences: Japanese food") is also sent to the server. Based on this information, a generative AI model is used to suggest recipes such as "Teriyaki Chicken." The missing ingredients (pork and soy sauce) are automatically ordered and delivered to the user's home.

[0675] Prompt Sentence Examples

[0676] "Please suggest a Japanese recipe using cabbage and carrots. The user's health information is: 5,000 steps, weight: 70 kg."

[0677] As described above, by using this system, users can reduce the effort required for managing ingredients, while receiving recipe suggestions that match their health information and preferences, and automatically ordering any ingredients they are missing.

[0678] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0679] Step 1:

[0680] The terminal (a camera inside the refrigeration equipment) captures the image.

[0681] Input: Visual data of the inside of the refrigerator

[0682] Output: Acquired image data

[0683] Specific operation: A camera inside the refrigerator captures images of the inside of the refrigerator at regular intervals or according to the user's instructions. These images are temporarily stored on the device and then uploaded to the server.

[0684] Step 2:

[0685] The image data acquired by the terminal is uploaded to the server.

[0686] Input: Acquired image data

[0687] Output: Uploaded image data

[0688] Specific operation: The device sends image data to the server via an Internet connection, and the server stores the received data in temporary storage.

[0689] Step 3:

[0690] The server processes the image data using an AI image analysis algorithm and automatically recognizes the ingredients inside.

[0691] Input: Uploaded image data

[0692] Output: List of recognized ingredients

[0693] How it works: Using AI image analysis algorithms (e.g., TensorFlow and OpenCV) running on the server, each ingredient in the image is identified. The recognized ingredient information is then stored in a database.

[0694] Step 4:

[0695] The server retrieves the user's health and preference information from the database and integrates it with food ingredient information.

[0696] Input: Recognized ingredient list, user health information, and preference information

[0697] Output: Integrated user and ingredient information

[0698] How it works: The server retrieves the user's health and preference information from the database and combines it with the recognized ingredient list. This combined data is used to generate recipes.

[0699] Step 5:

[0700] The server uses a generative AI model to generate recipes based on the integrated data.

[0701] Input: Integrated user and ingredient information

[0702] Output: The generated recipe

[0703] How it works: The server uses a generative AI model to generate an optimal recipe from the integrated data. This process may involve prompts.

[0704] Step 6:

[0705] The server transmits the generated recipe to the user terminal.

[0706] Input: Generated recipe

[0707] Output: Recipe information sent to the user's device

[0708] Specific operation: The server sends the generated recipe to the user device (smartphone app), which receives it and displays it to the user.

[0709] Step 7:

[0710] The user terminal checks the recipe and places an order for any missing ingredients.

[0711] Input: Recipe information sent to the user's device

[0712] Output: List of missing ingredients and order request

[0713] How it works: The user checks the recipe and checks for missing ingredients through the app. The missing ingredients are automatically recognized and an order request is sent to the food delivery service through the ordering method.

[0714] Through these processing steps, users can efficiently manage ingredients and receive appropriate recipe suggestions. In addition, ingredients that are in short supply are automatically ordered, improving the quality of life.

[0715] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0716] The present invention is a system that equips a refrigeration unit with acquisition means, recognition means, generation means, transmission means, and an emotion engine, and not only manages ingredients and suggests recipes for the user, but also suggests individual recipes that take the user's emotions into consideration.The system aims to suggest more appropriate recipes based on the user's health information, preference information, and current emotional state.

[0717] Overall system configuration

[0718] The system mainly consists of the following components:

[0719] 1. Camera inside the refrigeration equipment (acquisition method)

[0720] 2. Server that performs AI processing (recognition means, generation means, transmission means, emotion engine)

[0721] 3. User's smartphone app (user device)

[0722] System Operation Overview

[0723] 1. Camera inside the refrigeration equipment (acquisition method)

[0724] Terminal (refrigerator): A camera inside the refrigeration equipment captures images of the interior at regular intervals or according to user instructions.

[0725] Terminal (refrigerator): Uploads the acquired image data to the server.

[0726] 2. Image analysis and food ingredient recognition (recognition method)

[0727] Server: The server acquires the received image data and automatically recognizes the ingredients using an AI image analysis algorithm.

[0728] Server: Stores the recognized ingredient information in a database and updates the status inside the refrigeration equipment.

[0729] 3. Recipe generation (generation method)

[0730] Server: Generates optimal recipes based on the user's health information (number of steps, weight, age, etc.), preference information (favorite dishes and seasonings, etc.), and emotional information.

[0731] Server: For example, when a user is feeling stressed, the server suggests recipes that will help them relax.

[0732] 4. Emotion Recognition (Emotion Engine)

[0733] User: Emotions are captured from the user's voice, facial expressions, and text input via a smartphone app or a camera or microphone built into the refrigeration equipment display.

[0734] Server: Analyzes the acquired emotion data and identifies the user's current emotional state.

[0735] 5. Notification of information and recipe suggestions (transmission method)

[0736] Server: Sends the generated recipe to the user's smartphone app.

[0737] Users can view suggested recipes through a smartphone app.

[0738] Specific examples

[0739] For example, consider a case where cabbage, carrots, and chicken are stored in a refrigeration facility.

[0740] 1. Terminal (refrigerator): The camera takes pictures of the inside of the refrigerator and uploads the image data to the server.

[0741] 2. Server: Analyzes the uploaded image and automatically recognizes "cabbage," "carrot," and "chicken."

[0742] 3. User: Enters the following information into the smartphone app: "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food." At this time, the built-in camera and microphone determine that the user is feeling a little stressed.

[0743] 4. Server: Integrates the ingredient list, health information, preference information, and emotional information to suggest dishes like "teriyaki chicken" and "Japanese-style cabbage and carrot salad," as well as "herbal tea" for relaxation.

[0744] 5. User: Selects "Teriyaki Chicken" from the recipes suggested by the smartphone app and makes adjustments such as "making it stronger in seasoning."

[0745] 6. Server: Generates the fine-tuned recipe and sends the final cooking instructions and required seasoning list to the user's smartphone.

[0746] 7. User: Cooks food according to the recipe displayed on the smartphone app.

[0747] In this way, the present invention is a system that efficiently utilizes ingredients stored in refrigeration equipment and suggests optimal recipes based on the user's health information, preference information, and emotional state, helping to reduce food waste and solving daily meal problems.

[0748] The processing flow will be explained below.

[0749] Step 1:

[0750] Terminal (refrigerator): The camera inside the refrigerator takes pictures of the inside at a set time. For example, if it is set to take pictures every day at 10:00, the camera will automatically operate at that time.

[0751] Step 2:

[0752] Terminal (refrigerator): The acquired image data is sent to the server via the internal storage device of the refrigeration equipment or the network.

[0753] Step 3:

[0754] Server: The server takes the received image data and begins processing it using AI image analysis algorithms, specifically detecting objects in the image and identifying which ingredients they correspond to.

[0755] Step 4:

[0756] Server: Stores the recognized ingredient information in a database. Newly recognized ingredients and updated information on existing ingredients are reflected in the database.

[0757] Step 5:

[0758] User: Opens the smartphone app and enters health and preference information. For example, enter "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food."

[0759] Step 6:

[0760] Terminal (smartphone): The entered health information and preference information is sent to the server.

[0761] Step 7:

[0762] User: Using a smartphone app and the built-in camera and microphone in the refrigerator display, emotional information is acquired from voice and facial expressions. For example, emotions such as "I'm tired" or "I want to relax" are analyzed.

[0763] Step 8:

[0764] Server: Using the emotion engine, analyze the acquired emotion data and identify the user's current emotional state.

[0765] Step 9:

[0766] Server: Integrates the acquired information on ingredients, health, preferences, and emotions to generate optimal recipes. For example, a user feeling stressed might be suggested a recipe such as "herbal tea" that has a relaxing effect.

[0767] Step 10:

[0768] Server: Sends the generated recipe to the user's smartphone.

[0769] Step 11:

[0770] User: Check the recipes displayed on the smartphone app and select the desired recipe. For example, select "Teriyaki Chicken."

[0771] Step 12:

[0772] User: Requests seasoning or portion adjustments for a selected recipe. For example, they might type, "I'd like it a little stronger today."

[0773] Step 13:

[0774] Server: Fine-tunes the recipe data based on the user's request and generates the final cooking steps and list of required seasonings.

[0775] Step 14:

[0776] Server: Sends the fine-tuned recipe and cooking instructions to the user's smartphone.

[0777] Step 15:

[0778] User: Cooks a dish by following the cooking instructions displayed on the smartphone app.

[0779] Step 16:

[0780] Server: Regularly checks the food database in the refrigeration facility to identify food items that are close to their expiration date.

[0781] Step 17:

[0782] Server: Generates recipes that prioritize ingredients with an approaching expiration date and sends notification information to the user's smartphone.

[0783] Step 18:

[0784] Terminal (refrigerator): Notifications regarding expiration dates are displayed on the refrigerator equipment display and on the user's smartphone app.

[0785] Step 19:

[0786] User: Check the notification, use the suggested recipes as a guide, and prioritize ingredients that are close to their expiration date.

[0787] Through these steps, users can efficiently utilize the food stored in their refrigerators, helping them manage their health and reduce food waste.

[0788] Example 2

[0789] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0790] Today, there is a demand for efficient use of ingredients in refrigeration equipment, reducing food waste, and individually optimized recipe suggestions based on the user's health information, preferences, and even emotional state. However, conventional systems do not adequately consider the user's emotional state when managing ingredients or suggesting recipes, resulting in insufficient improvement in user satisfaction. The present invention aims to solve these problems by suggesting optimal recipes based on the user's health, preferences, and emotional state.

[0791] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server is disposed inside the refrigeration equipment and includes an acquisition means for acquiring images, a recognition means for processing the acquired images and automatically recognizing ingredients, a generation means for generating recipes based on ingredient information recognized by the recognition means, an emotion recognition means for analyzing user emotion information and generating recipes based on the emotional state, and a transmission means for transmitting the recipes generated by the generation means to a user terminal. This makes it possible to efficiently utilize ingredients in the refrigeration equipment and propose individually optimized recipes based on the user's health, preferences, and emotional state.

[0792] "Acquisition means" is a general term for hardware such as cameras and sensors that are placed inside the refrigeration equipment to acquire image data, and the software that controls them.

[0793] "Recognition means" is a general term for AI algorithms and software processing that processes image data acquired by the acquisition means and automatically identifies ingredients.

[0794] The "generation means" is a general term for programs and algorithms for generating recipes based on the ingredient information recognized by the recognition means and other related data.

[0795] "Emotion recognition means" is a general term for hardware and software that acquires emotional data from a user's voice, facial expression, text input, etc., and analyzes it to identify the user's emotional state.

[0796] The "transmission means" is a general term for the communication function for transmitting the recipe and notification information generated by the generation means to the user's terminal and the program that controls it.

[0797] A "user terminal" is a device that a user operates and views, and includes a smartphone, tablet, PC, etc.

[0798] MODE FOR CARRYING OUT THE INVENTION

[0799] The present invention is a system for managing ingredients and suggesting recipes to users by equipping a refrigeration unit with an acquisition means, a recognition means, a generation means, a transmission means, and an emotion recognition means. This system uses hardware and software such as a camera inside the refrigeration unit as an acquisition means, an AI image analysis algorithm as a recognition means, a machine learning model and a rule-based algorithm as a generation means, a communication function as a transmission means, and a voice recognition and facial expression analysis algorithm as an emotion recognition means.

[0800] Hardware and Software Use

[0801] 1. Camera inside the refrigeration equipment (acquisition method)

[0802] Terminal (refrigerator): A camera installed inside the refrigeration equipment captures images of the inside periodically or at the user's command. For example, the camera saves the image data in JPEG format and uploads it to a server via Wi-Fi.

[0803] 2. Image analysis and food ingredient recognition (recognition method)

[0804] Server: The server processes the received image data using an AI image analysis algorithm using TensorFlow and OpenCV to automatically recognize ingredients. The recognized ingredient information is stored in a database in JSON format.

[0805] 3. Emotion recognition (emotion recognition means)

[0806] User: Emotional data is collected from facial expressions, voice, and text input using a smartphone app or a camera or microphone built into the refrigeration equipment display. For example, emotional data is input by answering questions within the app.

[0807] Server: The emotion recognition engine analyzes the acquired data using natural language processing algorithms to identify the user's current emotional state, and records the results in a database.

[0808] 4. Recipe generation (generation method)

[0809] Server: The server integrates the user's health information (e.g., number of steps and weight), preference information (e.g., favorite dishes and seasonings), and emotional information to generate optimal recipes using machine learning models and rule-based algorithms. For example, if the user is feeling stressed, it will suggest recipes with a relaxing effect.

[0810] 5. Notification of information and recipe suggestions (transmission method)

[0811] Server: The generated recipe is sent to the user's device in JSON or XML format.

[0812] User: Using the smartphone app, view the suggested recipes and make adjustments to finalize the recipe. Recipe details and cooking instructions are displayed in the app.

[0813] Examples of specific examples and prompts

[0814] For example, imagine a refrigerator containing cabbage, carrots, and chicken. The refrigerator's camera takes a picture of the interior at 8:00 AM and uploads it to the server as "fridge_image_20230315_0800.jpg." The server uses TensorFlow to recognize "cabbage," "carrot," and "chicken" and stores them in a database. The user enters "Today's steps: 5,000," "Weight: 70 kg," and "Preferences: Japanese food" into a smartphone app, and the built-in camera and microphone determine that the user is "feeling a little stressed." Based on this information, the server suggests "teriyaki chicken," "Japanese-style cabbage and carrot salad," and "herbal tea" for relaxation. The user then adjusts and confirms the recipe.

[0815] Prompt Sentence Examples

[0816] "The ingredients stored in the refrigerator are cabbage, carrots, and chicken. The user likes Japanese food, has walked 5,000 steps today, and weighs 70 kg. The user is also feeling a little stressed. Based on this information, please suggest a Japanese recipe that will have a relaxing effect."

[0817] As described above, the present invention is a system that efficiently manages ingredients in a refrigeration facility and provides individually optimized recipes that take into account the user's health information, preference information, and even emotional state.

[0818] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0819] Step 1: Acquire and upload images

[0820] Terminal (refrigerator):

[0821] A camera inside the refrigeration equipment periodically captures images of the interior, for example, at 8:00 a.m. every day. Users can also manually instruct the camera to take pictures.

[0822] Input: User instructions or set scheduled time

[0823] Output: JPEG format image file

[0824] The captured image is saved as "fridge_image_yyyyMMdd_HHmm.jpg" and uploaded to the server via Wi-Fi.

[0825] Step 2: Image analysis and ingredient recognition

[0826] server:

[0827] The image data received by the server is stored in " / images / received / ".

[0828] Input: Uploaded JPEG image file

[0829] Processing: Recognize ingredients in the image using image analysis algorithms based on TensorFlow and OpenCV

[0830] Output: Ingredient information in JSON format

[0831] The recognized ingredient information is stored in the "food_items" table. For example, "cabbage," "carrot," and "chicken" are identified.

[0832] Step 3: Emotion Recognition

[0833] User:

[0834] Facial expressions, voice, and text input are provided via a smartphone app or a camera and microphone built into the refrigeration equipment display.

[0835] Input: Voice data, facial expression images, text data

[0836] Output: Emotion data

[0837] server:

[0838] An emotion recognition engine analyzes the input data and uses natural language processing algorithms to identify the user's emotional state.

[0839] Input: Voice data, facial expression images, text data

[0840] Processing: Emotional data analysis

[0841] Output: Emotional state (e.g., "I'm feeling a little stressed")

[0842] The analysis results are recorded in the "user_emotions" table.

[0843] Step 4: Recipe Generation

[0844] server:

[0845] The system integrates the user's health information, preference information, emotional information, and information on ingredients in the refrigerator to generate optimal recipes.

[0846] Input: Health information (number of steps, weight), preference information, emotional information, information on ingredients in the refrigerator

[0847] Processing: Generate recipes using machine learning models and rule-based algorithms

[0848] Output: Suggested recipe

[0849] For example, if a user is feeling stressed, the app will suggest relaxing dishes such as "teriyaki chicken," "Japanese-style cabbage and carrot salad," and even "herbal tea."

[0850] Step 5: Information and recipe suggestions

[0851] server:

[0852] The generated recipe is sent to the user's device in JSON or XML format.

[0853] Input: The generated recipe and other relevant information

[0854] Output: Data sent to the user terminal

[0855] User:

[0856] The suggested recipe is viewed on a smartphone app, and adjustments are made as needed. Finally, the final recipe is confirmed.

[0857] Input: Recipe information sent from the server

[0858] Output: Finalized recipe and cooking instructions

[0859] For example, if a refrigerator contains cabbage, carrots, and chicken, the refrigerator's camera periodically captures images and uploads them to a server. The server then analyzes the images and recognizes the ingredients. The user provides health and emotional information through a smartphone app, and the server then suggests recipes with a relaxing effect. Finally, the user can view and adjust the recipe and cook the food.

[0860] Example prompt sentence:

[0861] "The ingredients stored in the refrigerator are cabbage, carrots, and chicken. The user likes Japanese food, has walked 5,000 steps today, and weighs 70 kg. The user is also feeling a little stressed. Based on this information, please suggest a Japanese recipe that will have a relaxing effect."

[0862] (Application example 2)

[0863] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0864] For modern consumers, managing ingredients in refrigerated facilities and proposing optimal recipes based on that information is a very time-consuming task. Managing expiration dates for ingredients, creating recipes based on that information, and quickly sourcing missing ingredients are also challenges. Furthermore, most existing systems do not offer recipe suggestions that take into account the user's health information or emotional state. This situation could exacerbate food waste and health management issues. The present invention aims to solve these challenges by providing a system that manages ingredients, proposes recipes, automatically orders ingredients, and provides virtual experiences.

[0865] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0866] In this invention, the server includes an acquisition means for acquiring images, a recognition means for processing the acquired images and automatically recognizing ingredients, a generation means for generating a recipe based on the ingredient information recognized by the recognition means, a transmission means for transmitting the recipe generated by the generation means to a user terminal, an ordering means for automatically ordering missing ingredients, and a virtual reality means for providing a virtual experience, thereby enabling the user to efficiently manage ingredients in the refrigeration equipment, receive optimal recipe suggestions based on health information and emotional state, quickly order missing ingredients, and enjoy an intuitive virtual shopping experience.

[0867] Definition of Terms

[0868] "Means for acquiring images" refers to cameras or sensors that take images of the inside of the refrigeration equipment and acquire them as digital data.

[0869] "Means for processing acquired images and automatically recognizing ingredients" refers to a device or software that analyzes acquired image data and identifies the ingredients contained therein using AI or machine learning algorithms.

[0870] "Generation means for generating a recipe based on the ingredient information recognized by the recognition means" refers to a device or software that creates an optimal recipe based on the ingredient information obtained by the recognition means, taking into consideration the user's health information, preference information, emotional state, etc.

[0871] "Transmission means for transmitting the recipe generated by the generation means to a user terminal" refers to a device or software that transmits the generated recipe to a terminal such as a user's smartphone or tablet via a network.

[0872] "Ordering means for automatically ordering missing ingredients" means a device or software that automatically orders missing ingredients online based on the generated recipe.

[0873] "Virtual reality means for providing a virtual experience" means a device or software that provides a virtual reality (VR) or augmented reality (AR) environment that can be intuitively operated by a user.

[0874] "Health information" refers to physical data such as the user's number of steps, weight, age, etc., and particularly includes data related to dietary habits.

[0875] "Preference information" refers to information about a user's favorite dishes and seasonings, and particularly includes data that reflects an individual's food preferences.

[0876] "Emotional state" refers to the mental state of the user and includes data for inferring emotions from voice, facial expressions, text input, etc.

[0877] MODE FOR CARRYING OUT THE INVENTION

[0878] This invention realizes a system that uses an AI model to recognize ingredients based on images captured by a camera inside the refrigeration equipment, and then generates and suggests recipes to the user. It also has the ability to suggest more personalized recipes and automatically order missing ingredients by taking into account the user's health information and emotional state. Below, we will explain each element of this system in detail.

[0879] Hardware and software used

[0880] Hardware: Cameras in refrigeration equipment, smartphones, server PCs, smart glasses

[0881] Software: Python, Flask, TensorFlow, OpenAI API

[0882] System configuration and operation

[0883] 1. Camera inside the refrigeration equipment (acquisition method)

[0884] Cameras placed inside the refrigeration equipment periodically or at the user's command capture images of the interior, which are then uploaded as digital data to a server.

[0885] 2. Image analysis and food ingredient recognition (recognition method)

[0886] The server analyzes the image data received from the acquisition means and automatically recognizes ingredients using an AI model trained using TensorFlow.

[0887] 3. Recipe generation (generation method)

[0888] The server generates the most suitable recipe based on the ingredient information identified by the recognition means, taking into consideration the user's health information (e.g., number of steps, weight) and preference information (e.g., favorite dishes, seasonings). Furthermore, to take the user's emotional state into account, it uses emotional information obtained by analyzing their voice and facial expressions. At this point, the generated recipe is sent to the smartphone app.

[0889] 4. Automatic ordering of missing ingredients (ordering method)

[0890] Based on the generated recipe, any missing ingredients are automatically ordered online, allowing users to quickly get the ingredients they need.

[0891] 5. Virtual Experience (Virtual Reality Methods)

[0892] Users can enjoy an intuitive virtual experience using smart glasses or a smartphone, making refrigerator food management and online shopping even more convenient.

[0893] Specific examples

[0894] For example, imagine a refrigerator containing cabbage, carrots, and chicken. A camera takes images of these ingredients and uploads them to a server. The server recognizes these ingredients and compares them with the user's health information (e.g., 5,000 steps today, weight 70 kg) and preference information (e.g., likes Japanese food) against the input data. It also takes into account the user's emotional state, as determined from their voice and facial expressions (e.g., feeling a little stressed), and generates the most suitable recipe (e.g., teriyaki chicken, Japanese-style cabbage and carrot salad, and relaxing herbal tea).

[0895] The generated recipe is sent to a smartphone app, where users can use it to create the dish, and if any ingredients are missing, the system will automatically order them online and have them available immediately.

[0896] Prompt Sentence Examples

[0897] "Ingredients in the refrigerator: [cabbage, carrots, chicken]. Health information: {"steps": 5000, "weight": 70kg}. Preference information: "I like Japanese food, and I like it strong." Emotional state: "I'm feeling a little stressed." Please suggest recipes based on this."

[0898] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0899] Processing steps of the system that realizes the application example

[0900] Step 1:

[0901] The terminal (a camera inside the refrigerator) periodically captures images of the inside of the refrigerator or when instructed by the user. This image data is captured and saved in digital format. The input is the current state of the inside of the refrigerator, and the output is the digital image data.

[0902] Step 2:

[0903] The image data acquired by the device is uploaded to the server. The input here is the image data acquired in step 1, and the output is the image data transferred to the server.

[0904] Step 3:

[0905] The server processes the uploaded image data and automatically recognizes ingredients using an AI image recognition algorithm. A model using TensorFlow analyzes the image. The image data is given as input, and the output is the recognized ingredient information.

[0906] Step 4:

[0907] The recognized ingredient information is stored in a database on the server, and the status inside the refrigeration equipment is updated. The input is the recognized ingredient information, and the output is the updated database information.

[0908] Step 5:

[0909] Users use a smartphone app to input health information (e.g., number of steps, weight) and preference information (e.g., favorite food). In addition, emotional information is acquired through the built-in camera and microphone. The user's health information, preference information, and emotional information are given as input, and the output is the integrated data.

[0910] Step 6:

[0911] The server generates the optimal recipe based on the integrated information (ingredient information, health information, preference information, and emotional information). In this process, recipe generation is performed based on prompts using OpenAI's API. The input is the integrated data, and the output is the generated recipe.

[0912] Step 7:

[0913] The generated recipe is sent to the user device (smartphone app). The input is the generated recipe, and the output is the recipe information displayed on the user device.

[0914] Step 8:

[0915] If necessary, the user can automatically order missing ingredients online based on the generated recipe. The input is the missing ingredient information, and the output is the ingredients obtained by online ordering.

[0916] Step 9:

[0917] Users use smart glasses or smartphones to control refrigeration equipment and enjoy virtual experiences, including ingredient management, recipe suggestions, and even virtual shopping. The input is the user's operation, and the output is the user experience.

[0918] Through these steps, the system will be able to provide comprehensive services, from ingredient management to recipe suggestions, online ordering of missing ingredients, and virtual experiences.

[0919] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0920] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0921] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0922] [Third embodiment]

[0923] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0924] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0925] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0926] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0927] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0928] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0929] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0930] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0931] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0932] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0933] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0934] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0935] The present invention is a system that manages ingredients and suggests recipes for users using acquisition means, recognition means, generation means, and transmission means installed in refrigeration equipment. The system aims to suggest efficient and healthy meals by taking into account the user's health information and preference information.

[0936] Overall system configuration

[0937] The system mainly consists of the following components:

[0938] 1. Camera inside the refrigeration equipment (acquisition method)

[0939] 2. Server that performs AI processing (recognition, generation, and transmission means)

[0940] 3. User's smartphone app (user device)

[0941] System Operation Overview

[0942] 1. Camera inside the refrigeration equipment (acquisition method)

[0943] Terminal (refrigeration equipment): A camera inside the refrigeration equipment captures images of the inside at regular intervals or according to user instructions.

[0944] Terminal (refrigeration equipment): Uploads the acquired image data to the server.

[0945] 2. Image analysis and food ingredient recognition (recognition method)

[0946] Server: Receives the acquired image data and automatically recognizes the ingredients inside using an AI image analysis algorithm.

[0947] Server: Stores the recognized ingredient information in a database and updates the internal status of the refrigeration equipment.

[0948] 3. Recipe generation (generation method)

[0949] Server: Generates optimal recipes based on the user's health information (number of steps, weight, age, etc.) and preference information (favorite dishes and seasonings, etc.).

[0950] Server: Using AI technology, it suggests cooking recipes using recognized ingredients.

[0951] 4. Notification of information and recipe suggestions (transmission method)

[0952] Server: Sends the generated recipe to the user's smartphone app.

[0953] Users can view suggested recipes through a smartphone app.

[0954] Specific examples

[0955] For example, consider a case where cabbage, carrots, and chicken are stored in a refrigeration facility.

[0956] 1. Terminal (refrigeration equipment): The camera takes pictures of the inside of the refrigerator and uploads the image data to the server.

[0957] 2. Server: Analyzes the uploaded image and automatically recognizes "cabbage," "carrot," and "chicken."

[0958] 3. User: Enters the following information into the smartphone app: "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food." This information is sent to the server.

[0959] 4. Server: Integrates the ingredient list with user information and generates recipe suggestions such as "Teriyaki Chicken" or "Japanese-style Cabbage and Carrot Salad."

[0960] 5. User: Selects "Teriyaki Chicken" from the recipes suggested by the smartphone app and makes adjustments such as "making it stronger in seasoning."

[0961] 6. Server: Generates the fine-tuned recipe and sends the final cooking instructions and required seasoning list to the user's smartphone.

[0962] 7. User: Can cook food according to the recipe displayed on the smartphone app.

[0963] In this way, the present invention is a system that efficiently utilizes ingredients stored in refrigeration equipment and suggests optimal recipes based on the user's health and preference information, helping to reduce food waste and solving daily meal problems.

[0964] The processing flow will be explained below.

[0965] Step 1:

[0966] Terminal (refrigerator): The camera inside the refrigerator takes pictures of the inside at a set time. For example, if it is set to take pictures every day at 10:00, the camera will automatically operate at that time.

[0967] Step 2:

[0968] Terminal (refrigerator): The acquired image data is sent to the server via the internal storage device of the refrigeration equipment or the network.

[0969] Step 3:

[0970] Server: The server takes the received image data and begins processing it using AI image analysis algorithms, specifically detecting objects in the image and identifying which ingredients they correspond to.

[0971] Step 4:

[0972] Server: Stores the recognized ingredient information in a database. Newly recognized ingredients and updated information on existing ingredients are reflected in the database.

[0973] Step 5:

[0974] User: Opens the smartphone app and enters health and preference information. For example, enter "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food."

[0975] Step 6:

[0976] Terminal (smartphone): The entered health information and preference information is sent to the server.

[0977] Step 7:

[0978] Server: Integrates the acquired ingredient information with the user's health and preference information to generate optimal recipes. For example, it selects a Japanese recipe using cabbage, carrots, and chicken.

[0979] Step 8:

[0980] Server: Sends the generated recipe to the user's smartphone via the network.

[0981] Step 9:

[0982] User: Check the recipes displayed on the smartphone app and select the desired recipe. For example, select "Teriyaki Chicken."

[0983] Step 10:

[0984] User: Requests seasoning or portion adjustments for a selected recipe. For example, they might type, "I'd like it a little stronger today."

[0985] Step 11:

[0986] Server: Fine-tunes the recipe data based on the user's request and generates the final cooking steps and list of required seasonings.

[0987] Step 12:

[0988] Server: Sends the fine-tuned recipe and cooking instructions to the user's smartphone.

[0989] Step 13:

[0990] User: Cooks a dish by following the cooking instructions displayed on the smartphone app.

[0991] Step 14:

[0992] Server: Regularly checks the food database in the refrigeration facility to identify food items that are close to their expiration date.

[0993] Step 15:

[0994] Server: Generates recipes that prioritize ingredients with an approaching expiration date and sends notification information to the user's smartphone.

[0995] Step 16:

[0996] Terminal (refrigerator): Notifications regarding expiration dates are displayed on the refrigerator equipment display and on the user's smartphone app.

[0997] Step 17:

[0998] User: Check the notification, use the suggested recipes as a guide, and prioritize ingredients that are close to their expiration date.

[0999] Through these steps, users can efficiently utilize the food stored in their refrigerators, helping them manage their health and reduce food waste.

[1000] Example 1

[1001] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1002] In modern life, efficiently managing food stored in refrigerators and preparing healthy meals that suit individual tastes is a time-consuming and labor-intensive task. Conventional refrigerators lack the functionality to suggest optimal dishes based on the expiration date and type of ingredients, and it requires a great deal of effort for users to understand the condition of ingredients and select recipes based on health information and preferences. This has led to increased food waste and unhealthy food choices.

[1003] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1004] In this invention, the server includes an acquisition means, a recognition means, a generation means, a transmission means, and a user terminal. This allows for efficient management of ingredients in the refrigeration unit and suggests optimal recipes based on the user's health and preference information. Specifically, a camera installed inside the refrigeration unit captures images of ingredients and uploads them to the server. The server automatically recognizes ingredients from the uploaded images using an image analysis algorithm based on deep learning and stores the information in a database. The server then receives the user's health and preference information and uses a generative AI model to generate an optimal recipe based on this information. The generated recipe is then sent to the user's smartphone app, where the user can review the suggested recipe and make adjustments as needed. This makes it easier for users to prepare healthy meals that suit their preferences, contributing to reducing food waste.

[1005] The "acquisition means" is a device for acquiring image data using a camera or the like inside the refrigeration equipment and transmitting the image data to the server.

[1006] The "recognition means" refers to an AI image analysis algorithm and its processing device that analyzes image data uploaded to the server and automatically identifies the ingredients inside.

[1007] The "generation means" refers to an algorithm and a processing device for generating an optimal recipe based on the ingredient information identified by the recognition means and the user's health information and preference information.

[1008] The "transmission means" refers to a communication device and its protocol for transmitting the recipe information generated by the generation means to a user terminal such as a smartphone app of the user.

[1009] A "user terminal" is an electronic device such as a smartphone or tablet used by a user, and is used to receive, display, and adjust suggested recipe information.

[1010] "Health Information" refers collectively to a User's weight, number of steps, age, and other health-related data.

[1011] "Preference information" is a general term for information about the types of food and seasonings that a user likes.

[1012] A "generative AI model" is an advanced artificial intelligence algorithm, such as GPT-4, that generates new information, especially recipe information, based on input data.

[1013] A "prompt sentence" is an input sentence used to cause an AI model to generate a particular output.

[1014] This invention is a system that manages ingredients and suggests recipes to users using acquisition means, recognition means, generation means, and transmission means installed in refrigeration equipment. This system efficiently manages ingredients in the refrigerator and provides optimal recipes based on the user's health information and preference information, thereby helping to reduce food waste and promote healthy eating habits.

[1015] Hardware and Software Configuration

[1016] 1. Camera inside the refrigeration equipment (acquisition method)

[1017] Terminal (refrigeration equipment): A camera is installed inside the refrigerator and takes images of the inside of the refrigerator at regular intervals or according to the user's instructions. This camera uploads the image data to a server via an internet connection.

[1018] 2. Image analysis and food ingredient recognition (recognition method)

[1019] Server: The server receives the uploaded image data and automatically recognizes the ingredients in the refrigerator using an image analysis algorithm that uses deep learning. This algorithm uses a deep learning framework such as TensorFlow. The recognized ingredient information is stored in a database and the internal status of the refrigerator is updated in real time.

[1020] 3. Recipe generation (generation method)

[1021] Server: The server receives health information (number of steps, weight, age, etc.) and preference information (preferred cuisine, seasonings, etc.) from the user's smartphone app. Based on this, it uses a generative AI model (e.g., GPT-4) to generate the optimal recipe for the user. In this process, recipes are created taking into account the priority of ingredient use and nutritional balance.

[1022] 4. Notification of information and recipe suggestions (transmission method)

[1023] Server: Sends the generated recipe information to the user's smartphone app, where the user can review the suggested recipe and make adjustments as needed.

[1024] Specific examples

[1025] For example, if cabbage, carrots, and chicken are stored in the refrigerator, the following processing will occur:

[1026] 1. Terminal (refrigeration equipment): The camera takes pictures of the inside of the refrigerator and uploads the image data to the server.

[1027] 2. Server: Analyzes the uploaded image and automatically recognizes "cabbage," "carrot," and "chicken."

[1028] 3. User: Enters "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food" into the smartphone app and sends this information to the server.

[1029] 4. Server: Integrates the recognized ingredient information with the user's health information and generates recipe suggestions such as "Teriyaki Chicken" or "Japanese-style Cabbage and Carrot Salad."

[1030] 5. User: Check the suggested recipes on the smartphone app, select "Teriyaki Chicken," and make adjustments such as "Make it stronger in seasoning."

[1031] 6. Server: Regenerates the adjusted recipe and sends the final cooking instructions and required seasoning list to your smartphone.

[1032] 7. User: Can cook food according to the recipe displayed on the smartphone app.

[1033] Prompt Sentence Examples

[1034] For example, you can generate a recipe by inputting the following prompt into a generative AI model:

[1035] Prompt statement:

[1036] There are "cabbage," "carrots," and "chicken" in the refrigerator. The user's health information is as follows: "Today's steps: 5,000," "Weight: 70 kg," and "Preferences: Japanese food." Based on this, please suggest a healthy and delicious recipe.

[1037] output:

[1038] 1. Teriyaki chicken

[1039] 2. Japanese-style cabbage and carrot salad

[1040] In this way, the present invention realizes a system that can support a healthy diet by effectively utilizing ingredients stored in refrigeration equipment and providing optimal recipes based on the user's health and preference information.

[1041] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1042] Step 1:

[1043] Terminal (refrigeration equipment): A camera installed inside the refrigerator takes pictures of the inside of the refrigerator at regular intervals or according to the user's instructions.

[1044] Input: User instruction or scheduled trigger

[1045] Output: Image data of the inside of the refrigerator

[1046] Specifically, the camera captures high-resolution images from different angles inside the refrigerator, making all the ingredients inside visible.

[1047] Step 2:

[1048] Terminal (refrigeration equipment): Uploads the acquired image data to a server via an internet connection.

[1049] Input: Image data of the inside of the refrigerator

[1050] Output: Image data sent to the server

[1051] Specifically, the image data is compressed and transferred to a server using a secure protocol.

[1052] Step 3:

[1053] Server: Receives uploaded image data and performs image analysis using a deep learning framework (e.g., TensorFlow).

[1054] Input: Uploaded image data of the inside of the refrigerator

[1055] Output: Recognized ingredients

[1056] Specifically, the server runs an object detection algorithm to identify the type and location of ingredients in the image and stores this information in a database.

[1057] Step 4:

[1058] User: Enters health information (e.g., number of steps, weight, age) and preference information (e.g., favorite food, seasoning) into the smartphone app.

[1059] Input: User health and preference information

[1060] Output: User information sent to the server

[1061] Specifically, the user uses a smartphone app to input information using an intuitive interface, and then presses the send button to transfer the information to the server.

[1062] Step 5:

[1063] Server: Based on the collected ingredient information and user information, a generative AI model (e.g., GPT-4) is used to generate the optimal recipe.

[1064] Input: Recognized food ingredients and user health and preference information

[1065] Output: The generated recipe

[1066] Specifically, the generative AI model uses prompt text to create multiple recipes that meet the user's requirements and selects the most suitable one.

[1067] Step 6:

[1068] Server: Sends the generated recipe information to the user's smartphone app.

[1069] Input: Generated recipe

[1070] Output: Recipe information sent to the smartphone app

[1071] Specifically, recipe information is sent from the server using a secure protocol such as SSL, and a notification is displayed on the smartphone app.

[1072] Step 7:

[1073] User: Checks the suggested recipe on the smartphone app and makes adjustments as needed.

[1074] Input: Submitted recipe information

[1075] Output: Adjusted recipe request

[1076] Specifically, the user selects a recipe within the app, sets adjustments such as "stronger seasoning" or "add ingredients," and then sends a request to the server again.

[1077] Step 8:

[1078] Server: Based on the adjusted request, the server regenerates the final recipe and the list of required condiments and sends them to the smartphone.

[1079] Input: Adjustment request from user

[1080] Output: Final recipe and seasoning list

[1081] Specifically, the AI ​​model regenerates the recipe, determines the final recipe information reflecting the adjustments, and sends it to the smartphone app.

[1082] Step 9:

[1083] User: Cooks a dish according to the final recipe displayed on the smartphone app.

[1084] Input: Final recipe and seasoning list

[1085] Output: Finished dish

[1086] Specifically, the user follows the steps displayed on the smartphone app and completes the dish using the suggested seasonings.

[1087] (Application example 1)

[1088] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1089] In conventional refrigeration equipment, food ingredients are managed and expiration dates are checked manually, which leads to food waste and makes it difficult for users to receive appropriate recipe suggestions that take into account their health and preferences. Additionally, ordering ingredients that are in short supply is a time-consuming process. This creates problems that lower the quality of life for users.

[1090] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1091] In this invention, the server is located inside the refrigeration equipment and includes an acquisition means for acquiring images, a recognition means for processing the acquired images and automatically recognizing ingredients, a generation means for generating a recipe based on the recognized ingredient information, a transmission means for transmitting the generated recipe to a user terminal, and an ordering means for automatically ordering missing ingredients via the transmission means. This enables more efficient ingredient management, reduced food waste, recipe suggestions that take into account the user's health information and preference information, and automatic ordering of missing ingredients.

[1092] "Capture means" refers to the devices and techniques placed inside the refrigeration unit for capturing images.

[1093] The "recognition means" refers to the technology and algorithms for processing the acquired images and automatically recognizing ingredients.

[1094] The "generation means" refers to software and calculation techniques for generating recipes based on the ingredient information acquired by the recognition means.

[1095] The "transmission means" refers to a communication technique and device for transmitting the recipe generated by the generation means to the user terminal.

[1096] "Ordering means" refers to the system and protocol for automatically ordering missing ingredients via the transmission means.

[1097] "Health information" refers to health data such as the user's number of steps, weight, body fat percentage, and blood pressure.

[1098] "Preference information" refers to data regarding a user's favorite dishes and seasonings.

[1099] A "refrigeration equipment" is a home appliance used to store food at the appropriate temperature.

[1100] A "user terminal" is a mobile information terminal such as a smartphone or tablet operated by a user.

[1101] MODE FOR CARRYING OUT THE INVENTION

[1102] The present invention provides a system for managing ingredients for a user, proposing recipes, and automatically ordering ingredients that are in short supply. The system includes an acquisition unit, a recognition unit, a generation unit, a transmission unit, and an ordering unit that are arranged inside a refrigeration facility.

[1103] System Configuration

[1104] 1. Acquisition method:

[1105] A camera installed inside the refrigeration equipment functions as the acquisition means. This camera takes images of the inside at regular intervals or according to user instructions and uploads them to a server.

[1106] 2. Recognition means:

[1107] The AI ​​image analysis algorithm installed on the server processes the captured image data and automatically recognizes ingredients in the refrigerator, using software libraries such as TensorFlow and OpenCV.

[1108] 3. Generation means:

[1109] Based on the ingredient information stored on the server, a generative AI model is used to generate recipes based on the user's health and preference information.

[1110] 4. Means of transmission:

[1111] The generated recipe is sent to the user's device, which has a smartphone application installed, allowing the user to check the recipe suggestions.

[1112] 5. Ordering Method:

[1113] Based on the ingredient information recognized by the recognition means and the recipe generated by the generation means, the missing ingredients are automatically ordered from a food delivery service. This process uses a RESTful API.

[1114] System Operation Overview

[1115] The system of the present invention operates as follows:

[1116] 1. Images of the inside of the refrigeration equipment are periodically acquired by the acquisition means and uploaded to the server.

[1117] 2. The image data acquired by the recognition means is processed using an AI image analysis algorithm to recognize the ingredients in the refrigeration equipment.

[1118] 3. The recognized food ingredient information is stored on the server and integrated with the user's health and preference information.

[1119] 4. The generation means generates a recipe that takes into account the user's health information (number of steps, weight, etc.) and preference information (favorite dishes, seasonings, etc.).

[1120] 5. The suggested recipes are sent to the user's smartphone via the transmission means, where the user can check them.

[1121] 6. If necessary, any missing ingredients will be automatically ordered from the food delivery service via the ordering method.

[1122] Specific examples

[1123] For example, imagine a refrigerator contains cabbage, carrots, and chicken, but is short on pork and soy sauce. A camera takes a picture of the inside of the refrigerator and uploads the image data to a server. As a result of image analysis, "cabbage," "carrots," and "chicken" are recognized and stored on the server. The user's health information (for example, "Today's steps: 5,000 steps," "Weight: 70 kg," "Preferences: Japanese food") is also sent to the server. Based on this information, a generative AI model is used to suggest recipes such as "Teriyaki Chicken." The missing ingredients (pork and soy sauce) are automatically ordered and delivered to the user's home.

[1124] Prompt Sentence Examples

[1125] "Please suggest a Japanese recipe using cabbage and carrots. The user's health information is: 5,000 steps, weight: 70 kg."

[1126] As described above, by using this system, users can reduce the effort required for managing ingredients, while receiving recipe suggestions that match their health information and preferences, and automatically ordering any ingredients they are missing.

[1127] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1128] Step 1:

[1129] The terminal (a camera inside the refrigeration equipment) captures the image.

[1130] Input: Visual data of the inside of the refrigerator

[1131] Output: Acquired image data

[1132] Specific operation: A camera inside the refrigerator captures images of the inside of the refrigerator at regular intervals or according to the user's instructions. These images are temporarily stored on the device and then uploaded to the server.

[1133] Step 2:

[1134] The image data acquired by the terminal is uploaded to the server.

[1135] Input: Acquired image data

[1136] Output: Uploaded image data

[1137] Specific operation: The device sends image data to the server via an Internet connection, and the server stores the received data in temporary storage.

[1138] Step 3:

[1139] The server processes the image data using an AI image analysis algorithm and automatically recognizes the ingredients inside.

[1140] Input: Uploaded image data

[1141] Output: List of recognized ingredients

[1142] How it works: Using AI image analysis algorithms (e.g., TensorFlow and OpenCV) running on the server, each ingredient in the image is identified. The recognized ingredient information is then stored in a database.

[1143] Step 4:

[1144] The server retrieves the user's health and preference information from the database and integrates it with food ingredient information.

[1145] Input: Recognized ingredient list, user health information, and preference information

[1146] Output: Integrated user and ingredient information

[1147] How it works: The server retrieves the user's health and preference information from the database and combines it with the recognized ingredient list. This combined data is used to generate recipes.

[1148] Step 5:

[1149] The server uses a generative AI model to generate recipes based on the integrated data.

[1150] Input: Integrated user and ingredient information

[1151] Output: The generated recipe

[1152] How it works: The server uses a generative AI model to generate an optimal recipe from the integrated data. This process may involve prompts.

[1153] Step 6:

[1154] The server transmits the generated recipe to the user terminal.

[1155] Input: Generated recipe

[1156] Output: Recipe information sent to the user's device

[1157] Specific operation: The server sends the generated recipe to the user device (smartphone app), which receives it and displays it to the user.

[1158] Step 7:

[1159] The user terminal checks the recipe and places an order for any missing ingredients.

[1160] Input: Recipe information sent to the user's device

[1161] Output: List of missing ingredients and order request

[1162] How it works: The user checks the recipe and checks for missing ingredients through the app. The missing ingredients are automatically recognized and an order request is sent to the food delivery service through the ordering method.

[1163] Through these processing steps, users can efficiently manage ingredients and receive appropriate recipe suggestions. In addition, ingredients that are in short supply are automatically ordered, improving the quality of life.

[1164] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1165] The present invention is a system that equips a refrigeration unit with acquisition means, recognition means, generation means, transmission means, and an emotion engine, and not only manages ingredients and suggests recipes for the user, but also suggests individual recipes that take the user's emotions into consideration.The system aims to suggest more appropriate recipes based on the user's health information, preference information, and current emotional state.

[1166] Overall system configuration

[1167] The system mainly consists of the following components:

[1168] 1. Camera inside the refrigeration equipment (acquisition method)

[1169] 2. Server that performs AI processing (recognition means, generation means, transmission means, emotion engine)

[1170] 3. User's smartphone app (user device)

[1171] System Operation Overview

[1172] 1. Camera inside the refrigeration equipment (acquisition method)

[1173] Terminal (refrigerator): A camera inside the refrigeration equipment captures images of the interior at regular intervals or according to user instructions.

[1174] Terminal (refrigerator): Uploads the acquired image data to the server.

[1175] 2. Image analysis and food ingredient recognition (recognition method)

[1176] Server: The server acquires the received image data and automatically recognizes the ingredients using an AI image analysis algorithm.

[1177] Server: Stores the recognized ingredient information in a database and updates the status inside the refrigeration equipment.

[1178] 3. Recipe generation (generation method)

[1179] Server: Generates optimal recipes based on the user's health information (number of steps, weight, age, etc.), preference information (favorite dishes and seasonings, etc.), and emotional information.

[1180] Server: For example, when a user is feeling stressed, the server suggests recipes that will help them relax.

[1181] 4. Emotion Recognition (Emotion Engine)

[1182] User: Emotions are captured from the user's voice, facial expressions, and text input via a smartphone app or a camera or microphone built into the refrigeration equipment display.

[1183] Server: Analyzes the acquired emotion data and identifies the user's current emotional state.

[1184] 5. Notification of information and recipe suggestions (transmission method)

[1185] Server: Sends the generated recipe to the user's smartphone app.

[1186] Users can view suggested recipes through a smartphone app.

[1187] Specific examples

[1188] For example, consider a case where cabbage, carrots, and chicken are stored in a refrigeration facility.

[1189] 1. Terminal (refrigerator): The camera takes pictures of the inside of the refrigerator and uploads the image data to the server.

[1190] 2. Server: Analyzes the uploaded image and automatically recognizes "cabbage," "carrot," and "chicken."

[1191] 3. User: Enters the following information into the smartphone app: "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food." At this time, the built-in camera and microphone determine that the user is feeling a little stressed.

[1192] 4. Server: Integrates the ingredient list, health information, preference information, and emotional information to suggest dishes like "teriyaki chicken" and "Japanese-style cabbage and carrot salad," as well as "herbal tea" for relaxation.

[1193] 5. User: Selects "Teriyaki Chicken" from the recipes suggested by the smartphone app and makes adjustments such as "making it stronger in seasoning."

[1194] 6. Server: Generates the fine-tuned recipe and sends the final cooking instructions and required seasoning list to the user's smartphone.

[1195] 7. User: Cooks food according to the recipe displayed on the smartphone app.

[1196] In this way, the present invention is a system that efficiently utilizes ingredients stored in refrigeration equipment and suggests optimal recipes based on the user's health information, preference information, and emotional state, helping to reduce food waste and solving daily meal problems.

[1197] The processing flow will be explained below.

[1198] Step 1:

[1199] Terminal (refrigerator): The camera inside the refrigerator takes pictures of the inside at a set time. For example, if it is set to take pictures every day at 10:00, the camera will automatically operate at that time.

[1200] Step 2:

[1201] Terminal (refrigerator): The acquired image data is sent to the server via the internal storage device of the refrigeration equipment or the network.

[1202] Step 3:

[1203] Server: The server takes the received image data and begins processing it using AI image analysis algorithms, specifically detecting objects in the image and identifying which ingredients they correspond to.

[1204] Step 4:

[1205] Server: Stores the recognized ingredient information in a database. Newly recognized ingredients and updated information on existing ingredients are reflected in the database.

[1206] Step 5:

[1207] User: Opens the smartphone app and enters health and preference information. For example, enter "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food."

[1208] Step 6:

[1209] Terminal (smartphone): The entered health information and preference information is sent to the server.

[1210] Step 7:

[1211] User: Using a smartphone app and the built-in camera and microphone in the refrigerator display, emotional information is acquired from voice and facial expressions. For example, emotions such as "I'm tired" or "I want to relax" are analyzed.

[1212] Step 8:

[1213] Server: Using the emotion engine, analyze the acquired emotion data and identify the user's current emotional state.

[1214] Step 9:

[1215] Server: Integrates the acquired information on ingredients, health, preferences, and emotions to generate optimal recipes. For example, a user feeling stressed might be suggested a recipe such as "herbal tea" that has a relaxing effect.

[1216] Step 10:

[1217] Server: Sends the generated recipe to the user's smartphone.

[1218] Step 11:

[1219] User: Check the recipes displayed on the smartphone app and select the desired recipe. For example, select "Teriyaki Chicken."

[1220] Step 12:

[1221] User: Requests seasoning or portion adjustments for a selected recipe. For example, they might type, "I'd like it a little stronger today."

[1222] Step 13:

[1223] Server: Fine-tunes the recipe data based on the user's request and generates the final cooking steps and list of required seasonings.

[1224] Step 14:

[1225] Server: Sends the fine-tuned recipe and cooking instructions to the user's smartphone.

[1226] Step 15:

[1227] User: Cooks a dish by following the cooking instructions displayed on the smartphone app.

[1228] Step 16:

[1229] Server: Regularly checks the food database in the refrigeration facility to identify food items that are close to their expiration date.

[1230] Step 17:

[1231] Server: Generates recipes that prioritize ingredients with an approaching expiration date and sends notification information to the user's smartphone.

[1232] Step 18:

[1233] Terminal (refrigerator): Notifications regarding expiration dates are displayed on the refrigerator equipment display and on the user's smartphone app.

[1234] Step 19:

[1235] User: Check the notification, use the suggested recipes as a guide, and prioritize ingredients that are close to their expiration date.

[1236] Through these steps, users can efficiently utilize the food stored in their refrigerators, helping them manage their health and reduce food waste.

[1237] Example 2

[1238] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1239] Today, there is a demand for efficient use of ingredients in refrigeration equipment, reducing food waste, and individually optimized recipe suggestions based on the user's health information, preferences, and even emotional state. However, conventional systems do not adequately consider the user's emotional state when managing ingredients or suggesting recipes, resulting in insufficient improvement in user satisfaction. The present invention aims to solve these problems by suggesting optimal recipes based on the user's health, preferences, and emotional state.

[1240] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server is disposed inside the refrigeration equipment and includes an acquisition means for acquiring images, a recognition means for processing the acquired images and automatically recognizing ingredients, a generation means for generating recipes based on ingredient information recognized by the recognition means, an emotion recognition means for analyzing user emotion information and generating recipes based on the emotional state, and a transmission means for transmitting the recipes generated by the generation means to a user terminal. This makes it possible to efficiently utilize ingredients in the refrigeration equipment and propose individually optimized recipes based on the user's health, preferences, and emotional state.

[1241] "Acquisition means" is a general term for hardware such as cameras and sensors that are placed inside the refrigeration equipment to acquire image data, and the software that controls them.

[1242] "Recognition means" is a general term for AI algorithms and software processing that processes image data acquired by the acquisition means and automatically identifies ingredients.

[1243] The "generation means" is a general term for programs and algorithms for generating recipes based on the ingredient information recognized by the recognition means and other related data.

[1244] "Emotion recognition means" is a general term for hardware and software that acquires emotional data from a user's voice, facial expression, text input, etc., and analyzes it to identify the user's emotional state.

[1245] The "transmission means" is a general term for the communication function for transmitting the recipe and notification information generated by the generation means to the user's terminal and the program that controls it.

[1246] A "user terminal" is a device that a user operates and views, and includes a smartphone, tablet, PC, etc.

[1247] MODE FOR CARRYING OUT THE INVENTION

[1248] The present invention is a system for managing ingredients and suggesting recipes to users by equipping a refrigeration unit with an acquisition means, a recognition means, a generation means, a transmission means, and an emotion recognition means. This system uses hardware and software such as a camera inside the refrigeration unit as an acquisition means, an AI image analysis algorithm as a recognition means, a machine learning model and a rule-based algorithm as a generation means, a communication function as a transmission means, and a voice recognition and facial expression analysis algorithm as an emotion recognition means.

[1249] Hardware and Software Use

[1250] 1. Camera inside the refrigeration equipment (acquisition method)

[1251] Terminal (refrigerator): A camera installed inside the refrigeration equipment captures images of the inside periodically or at the user's command. For example, the camera saves the image data in JPEG format and uploads it to a server via Wi-Fi.

[1252] 2. Image analysis and food ingredient recognition (recognition method)

[1253] Server: The server processes the received image data using an AI image analysis algorithm using TensorFlow and OpenCV to automatically recognize ingredients. The recognized ingredient information is stored in a database in JSON format.

[1254] 3. Emotion recognition (emotion recognition means)

[1255] User: Emotional data is collected from facial expressions, voice, and text input using a smartphone app or a camera or microphone built into the refrigeration equipment display. For example, emotional data is input by answering questions within the app.

[1256] Server: The emotion recognition engine analyzes the acquired data using natural language processing algorithms to identify the user's current emotional state, and records the results in a database.

[1257] 4. Recipe generation (generation method)

[1258] Server: The server integrates the user's health information (e.g., number of steps and weight), preference information (e.g., favorite dishes and seasonings), and emotional information to generate optimal recipes using machine learning models and rule-based algorithms. For example, if the user is feeling stressed, it will suggest recipes with a relaxing effect.

[1259] 5. Notification of information and recipe suggestions (transmission method)

[1260] Server: The generated recipe is sent to the user's device in JSON or XML format.

[1261] User: Using the smartphone app, view the suggested recipes and make adjustments to finalize the recipe. Recipe details and cooking instructions are displayed in the app.

[1262] Examples of specific examples and prompts

[1263] For example, imagine a refrigerator containing cabbage, carrots, and chicken. The refrigerator's camera takes a picture of the interior at 8:00 AM and uploads it to the server as "fridge_image_20230315_0800.jpg." The server uses TensorFlow to recognize "cabbage," "carrot," and "chicken" and stores them in a database. The user enters "Today's steps: 5,000," "Weight: 70 kg," and "Preferences: Japanese food" into a smartphone app, and the built-in camera and microphone determine that the user is "feeling a little stressed." Based on this information, the server suggests "teriyaki chicken," "Japanese-style cabbage and carrot salad," and "herbal tea" for relaxation. The user then adjusts and confirms the recipe.

[1264] Prompt Sentence Examples

[1265] "The ingredients stored in the refrigerator are cabbage, carrots, and chicken. The user likes Japanese food, has walked 5,000 steps today, and weighs 70 kg. The user is also feeling a little stressed. Based on this information, please suggest a Japanese recipe that will have a relaxing effect."

[1266] As described above, the present invention is a system that efficiently manages ingredients in a refrigeration facility and provides individually optimized recipes that take into account the user's health information, preference information, and even emotional state.

[1267] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1268] Step 1: Acquire and upload images

[1269] Terminal (refrigerator):

[1270] A camera inside the refrigeration equipment periodically captures images of the interior, for example, at 8:00 a.m. every day. Users can also manually instruct the camera to take pictures.

[1271] Input: User instructions or set scheduled time

[1272] Output: JPEG format image file

[1273] The captured image is saved as "fridge_image_yyyyMMdd_HHmm.jpg" and uploaded to the server via Wi-Fi.

[1274] Step 2: Image analysis and ingredient recognition

[1275] server:

[1276] The image data received by the server is stored in " / images / received / ".

[1277] Input: Uploaded JPEG image file

[1278] Processing: Recognize ingredients in the image using image analysis algorithms based on TensorFlow and OpenCV

[1279] Output: Ingredient information in JSON format

[1280] The recognized ingredient information is stored in the "food_items" table. For example, "cabbage," "carrot," and "chicken" are identified.

[1281] Step 3: Emotion Recognition

[1282] User:

[1283] Facial expressions, voice, and text input are provided via a smartphone app or a camera and microphone built into the refrigeration equipment display.

[1284] Input: Voice data, facial expression images, text data

[1285] Output: Emotion data

[1286] server:

[1287] An emotion recognition engine analyzes the input data and uses natural language processing algorithms to identify the user's emotional state.

[1288] Input: Voice data, facial expression images, text data

[1289] Processing: Emotional data analysis

[1290] Output: Emotional state (e.g., "I'm feeling a little stressed")

[1291] The analysis results are recorded in the "user_emotions" table.

[1292] Step 4: Recipe Generation

[1293] server:

[1294] The system integrates the user's health information, preference information, emotional information, and information on ingredients in the refrigerator to generate optimal recipes.

[1295] Input: Health information (number of steps, weight), preference information, emotional information, information on ingredients in the refrigerator

[1296] Processing: Generate recipes using machine learning models and rule-based algorithms

[1297] Output: Suggested recipe

[1298] For example, if a user is feeling stressed, the app will suggest relaxing dishes such as "teriyaki chicken," "Japanese-style cabbage and carrot salad," and even "herbal tea."

[1299] Step 5: Information and recipe suggestions

[1300] server:

[1301] The generated recipe is sent to the user's device in JSON or XML format.

[1302] Input: The generated recipe and other relevant information

[1303] Output: Data sent to the user terminal

[1304] User:

[1305] The suggested recipe is viewed on a smartphone app, and adjustments are made as needed. Finally, the final recipe is confirmed.

[1306] Input: Recipe information sent from the server

[1307] Output: Finalized recipe and cooking instructions

[1308] For example, if a refrigerator contains cabbage, carrots, and chicken, the refrigerator's camera periodically captures images and uploads them to a server. The server then analyzes the images and recognizes the ingredients. The user provides health and emotional information through a smartphone app, and the server then suggests recipes with a relaxing effect. Finally, the user can view and adjust the recipe and cook the food.

[1309] Example prompt sentence:

[1310] "The ingredients stored in the refrigerator are cabbage, carrots, and chicken. The user likes Japanese food, has walked 5,000 steps today, and weighs 70 kg. The user is also feeling a little stressed. Based on this information, please suggest a Japanese recipe that will have a relaxing effect."

[1311] (Application example 2)

[1312] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1313] For modern consumers, managing ingredients in refrigerated facilities and proposing optimal recipes based on that information is a very time-consuming task. Managing expiration dates for ingredients, creating recipes based on that information, and quickly sourcing missing ingredients are also challenges. Furthermore, most existing systems do not offer recipe suggestions that take into account the user's health information or emotional state. This situation could exacerbate food waste and health management issues. The present invention aims to solve these challenges by providing a system that manages ingredients, proposes recipes, automatically orders ingredients, and provides virtual experiences.

[1314] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1315] In this invention, the server includes an acquisition means for acquiring images, a recognition means for processing the acquired images and automatically recognizing ingredients, a generation means for generating a recipe based on the ingredient information recognized by the recognition means, a transmission means for transmitting the recipe generated by the generation means to a user terminal, an ordering means for automatically ordering missing ingredients, and a virtual reality means for providing a virtual experience, thereby enabling the user to efficiently manage ingredients in the refrigeration equipment, receive optimal recipe suggestions based on health information and emotional state, quickly order missing ingredients, and enjoy an intuitive virtual shopping experience.

[1316] Definition of Terms

[1317] "Means for acquiring images" refers to cameras or sensors that take images of the inside of the refrigeration equipment and acquire them as digital data.

[1318] "Means for processing acquired images and automatically recognizing ingredients" refers to a device or software that analyzes acquired image data and identifies the ingredients contained therein using AI or machine learning algorithms.

[1319] "Generation means for generating a recipe based on the ingredient information recognized by the recognition means" refers to a device or software that creates an optimal recipe based on the ingredient information obtained by the recognition means, taking into consideration the user's health information, preference information, emotional state, etc.

[1320] "Transmission means for transmitting the recipe generated by the generation means to a user terminal" refers to a device or software that transmits the generated recipe to a terminal such as a user's smartphone or tablet via a network.

[1321] "Ordering means for automatically ordering missing ingredients" means a device or software that automatically orders missing ingredients online based on the generated recipe.

[1322] "Virtual reality means for providing a virtual experience" means a device or software that provides a virtual reality (VR) or augmented reality (AR) environment that can be intuitively operated by a user.

[1323] "Health information" refers to physical data such as the user's number of steps, weight, age, etc., and particularly includes data related to dietary habits.

[1324] "Preference information" refers to information about a user's favorite dishes and seasonings, and particularly includes data that reflects an individual's food preferences.

[1325] "Emotional state" refers to the mental state of the user and includes data for inferring emotions from voice, facial expressions, text input, etc.

[1326] MODE FOR CARRYING OUT THE INVENTION

[1327] This invention realizes a system that uses an AI model to recognize ingredients based on images captured by a camera inside the refrigeration equipment, and then generates and suggests recipes to the user. It also has the ability to suggest more personalized recipes and automatically order missing ingredients by taking into account the user's health information and emotional state. Below, we will explain each element of this system in detail.

[1328] Hardware and software used

[1329] Hardware: Cameras in refrigeration equipment, smartphones, server PCs, smart glasses

[1330] Software: Python, Flask, TensorFlow, OpenAI API

[1331] System configuration and operation

[1332] 1. Camera inside the refrigeration equipment (acquisition method)

[1333] Cameras placed inside the refrigeration equipment periodically or at the user's command capture images of the interior, which are then uploaded as digital data to a server.

[1334] 2. Image analysis and food ingredient recognition (recognition method)

[1335] The server analyzes the image data received from the acquisition means and automatically recognizes ingredients using an AI model trained using TensorFlow.

[1336] 3. Recipe generation (generation method)

[1337] The server generates the most suitable recipe based on the ingredient information identified by the recognition means, taking into consideration the user's health information (e.g., number of steps, weight) and preference information (e.g., favorite dishes, seasonings). Furthermore, to take the user's emotional state into account, it uses emotional information obtained by analyzing their voice and facial expressions. At this point, the generated recipe is sent to the smartphone app.

[1338] 4. Automatic ordering of missing ingredients (ordering method)

[1339] Based on the generated recipe, any missing ingredients are automatically ordered online, allowing users to quickly get the ingredients they need.

[1340] 5. Virtual Experience (Virtual Reality Methods)

[1341] Users can enjoy an intuitive virtual experience using smart glasses or a smartphone, making refrigerator food management and online shopping even more convenient.

[1342] Specific examples

[1343] For example, imagine a refrigerator containing cabbage, carrots, and chicken. A camera takes images of these ingredients and uploads them to a server. The server recognizes these ingredients and compares them with the user's health information (e.g., 5,000 steps today, weight 70 kg) and preference information (e.g., likes Japanese food) against the input data. It also takes into account the user's emotional state, as determined from their voice and facial expressions (e.g., feeling a little stressed), and generates the most suitable recipe (e.g., teriyaki chicken, Japanese-style cabbage and carrot salad, and relaxing herbal tea).

[1344] The generated recipe is sent to a smartphone app, where users can use it to create the dish, and if any ingredients are missing, the system will automatically order them online and have them available immediately.

[1345] Prompt Sentence Examples

[1346] "Ingredients in the refrigerator: [cabbage, carrots, chicken]. Health information: {"steps": 5000, "weight": 70kg}. Preference information: "I like Japanese food, and I like it strong." Emotional state: "I'm feeling a little stressed." Please suggest recipes based on this."

[1347] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1348] Processing steps of the system that realizes the application example

[1349] Step 1:

[1350] The terminal (a camera inside the refrigerator) periodically captures images of the inside of the refrigerator or when instructed by the user. This image data is captured and saved in digital format. The input is the current state of the inside of the refrigerator, and the output is the digital image data.

[1351] Step 2:

[1352] The image data acquired by the device is uploaded to the server. The input here is the image data acquired in step 1, and the output is the image data transferred to the server.

[1353] Step 3:

[1354] The server processes the uploaded image data and automatically recognizes ingredients using an AI image recognition algorithm. A model using TensorFlow analyzes the image. The image data is given as input, and the output is the recognized ingredient information.

[1355] Step 4:

[1356] The recognized ingredient information is stored in a database on the server, and the status inside the refrigeration equipment is updated. The input is the recognized ingredient information, and the output is the updated database information.

[1357] Step 5:

[1358] Users use a smartphone app to input health information (e.g., number of steps, weight) and preference information (e.g., favorite food). In addition, emotional information is acquired through the built-in camera and microphone. The user's health information, preference information, and emotional information are given as input, and the output is the integrated data.

[1359] Step 6:

[1360] The server generates the optimal recipe based on the integrated information (ingredient information, health information, preference information, and emotional information). In this process, recipe generation is performed based on prompts using OpenAI's API. The input is the integrated data, and the output is the generated recipe.

[1361] Step 7:

[1362] The generated recipe is sent to the user device (smartphone app). The input is the generated recipe, and the output is the recipe information displayed on the user device.

[1363] Step 8:

[1364] If necessary, the user can automatically order missing ingredients online based on the generated recipe. The input is the missing ingredient information, and the output is the ingredients obtained by online ordering.

[1365] Step 9:

[1366] Users use smart glasses or smartphones to control refrigeration equipment and enjoy virtual experiences, including ingredient management, recipe suggestions, and even virtual shopping. The input is the user's operation, and the output is the user experience.

[1367] Through these steps, the system will be able to provide comprehensive services, from ingredient management to recipe suggestions, online ordering of missing ingredients, and virtual experiences.

[1368] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1369] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1370] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1371] [Fourth embodiment]

[1372] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1373] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1374] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1375] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1376] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1377] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1378] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1379] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1380] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1381] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1382] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1383] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1384] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1385] The present invention is a system that manages ingredients and suggests recipes for users using acquisition means, recognition means, generation means, and transmission means installed in refrigeration equipment. The system aims to suggest efficient and healthy meals by taking into account the user's health information and preference information.

[1386] Overall system configuration

[1387] The system mainly consists of the following components:

[1388] 1. Camera inside the refrigeration equipment (acquisition method)

[1389] 2. Server that performs AI processing (recognition, generation, and transmission means)

[1390] 3. User's smartphone app (user device)

[1391] System Operation Overview

[1392] 1. Camera inside the refrigeration equipment (acquisition method)

[1393] Terminal (refrigeration equipment): A camera inside the refrigeration equipment captures images of the inside at regular intervals or according to user instructions.

[1394] Terminal (refrigeration equipment): Uploads the acquired image data to the server.

[1395] 2. Image analysis and food ingredient recognition (recognition method)

[1396] Server: Receives the acquired image data and automatically recognizes the ingredients inside using an AI image analysis algorithm.

[1397] Server: Stores the recognized ingredient information in a database and updates the internal status of the refrigeration equipment.

[1398] 3. Recipe generation (generation method)

[1399] Server: Generates optimal recipes based on the user's health information (number of steps, weight, age, etc.) and preference information (favorite dishes and seasonings, etc.).

[1400] Server: Using AI technology, it suggests cooking recipes using recognized ingredients.

[1401] 4. Notification of information and recipe suggestions (transmission method)

[1402] Server: Sends the generated recipe to the user's smartphone app.

[1403] Users can view suggested recipes through a smartphone app.

[1404] Specific examples

[1405] For example, consider a case where cabbage, carrots, and chicken are stored in a refrigeration facility.

[1406] 1. Terminal (refrigeration equipment): The camera takes pictures of the inside of the refrigerator and uploads the image data to the server.

[1407] 2. Server: Analyzes the uploaded image and automatically recognizes "cabbage," "carrot," and "chicken."

[1408] 3. User: Enters the following information into the smartphone app: "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food." This information is sent to the server.

[1409] 4. Server: Integrates the ingredient list with user information and generates recipe suggestions such as "Teriyaki Chicken" or "Japanese-style Cabbage and Carrot Salad."

[1410] 5. User: Selects "Teriyaki Chicken" from the recipes suggested by the smartphone app and makes adjustments such as "making it stronger in seasoning."

[1411] 6. Server: Generates the fine-tuned recipe and sends the final cooking instructions and required seasoning list to the user's smartphone.

[1412] 7. User: Can cook food according to the recipe displayed on the smartphone app.

[1413] In this way, the present invention is a system that efficiently utilizes ingredients stored in refrigeration equipment and suggests optimal recipes based on the user's health and preference information, helping to reduce food waste and solving daily meal problems.

[1414] The processing flow will be explained below.

[1415] Step 1:

[1416] Terminal (refrigerator): The camera inside the refrigerator takes pictures of the inside at a set time. For example, if it is set to take pictures every day at 10:00, the camera will automatically operate at that time.

[1417] Step 2:

[1418] Terminal (refrigerator): The acquired image data is sent to the server via the internal storage device of the refrigeration equipment or the network.

[1419] Step 3:

[1420] Server: The server takes the received image data and begins processing it using AI image analysis algorithms, specifically detecting objects in the image and identifying which ingredients they correspond to.

[1421] Step 4:

[1422] Server: Stores the recognized ingredient information in a database. Newly recognized ingredients and updated information on existing ingredients are reflected in the database.

[1423] Step 5:

[1424] User: Opens the smartphone app and enters health and preference information. For example, enter "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food."

[1425] Step 6:

[1426] Terminal (smartphone): The entered health information and preference information is sent to the server.

[1427] Step 7:

[1428] Server: Integrates the acquired ingredient information with the user's health and preference information to generate optimal recipes. For example, it selects a Japanese recipe using cabbage, carrots, and chicken.

[1429] Step 8:

[1430] Server: Sends the generated recipe to the user's smartphone via the network.

[1431] Step 9:

[1432] User: Check the recipes displayed on the smartphone app and select the desired recipe. For example, select "Teriyaki Chicken."

[1433] Step 10:

[1434] User: Requests seasoning or portion adjustments for a selected recipe. For example, they might type, "I'd like it a little stronger today."

[1435] Step 11:

[1436] Server: Fine-tunes the recipe data based on the user's request and generates the final cooking steps and list of required seasonings.

[1437] Step 12:

[1438] Server: Sends the fine-tuned recipe and cooking instructions to the user's smartphone.

[1439] Step 13:

[1440] User: Cooks a dish by following the cooking instructions displayed on the smartphone app.

[1441] Step 14:

[1442] Server: Regularly checks the food database in the refrigeration facility to identify food items that are close to their expiration date.

[1443] Step 15:

[1444] Server: Generates recipes that prioritize ingredients with an approaching expiration date and sends notification information to the user's smartphone.

[1445] Step 16:

[1446] Terminal (refrigerator): Notifications regarding expiration dates are displayed on the refrigerator equipment display and on the user's smartphone app.

[1447] Step 17:

[1448] User: Check the notification, use the suggested recipes as a guide, and prioritize ingredients that are close to their expiration date.

[1449] Through these steps, users can efficiently utilize the food stored in their refrigerators, helping them manage their health and reduce food waste.

[1450] Example 1

[1451] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1452] In modern life, efficiently managing food stored in refrigerators and preparing healthy meals that suit individual tastes is a time-consuming and labor-intensive task. Conventional refrigerators lack the functionality to suggest optimal dishes based on the expiration date and type of ingredients, and it requires a great deal of effort for users to understand the condition of ingredients and select recipes based on health information and preferences. This has led to increased food waste and unhealthy food choices.

[1453] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1454] In this invention, the server includes an acquisition means, a recognition means, a generation means, a transmission means, and a user terminal. This allows for efficient management of ingredients in the refrigeration unit and suggests optimal recipes based on the user's health and preference information. Specifically, a camera installed inside the refrigeration unit captures images of ingredients and uploads them to the server. The server automatically recognizes ingredients from the uploaded images using an image analysis algorithm based on deep learning and stores the information in a database. The server then receives the user's health and preference information and uses a generative AI model to generate an optimal recipe based on this information. The generated recipe is then sent to the user's smartphone app, where the user can review the suggested recipe and make adjustments as needed. This makes it easier for users to prepare healthy meals that suit their preferences, contributing to reducing food waste.

[1455] The "acquisition means" is a device for acquiring image data using a camera or the like inside the refrigeration equipment and transmitting the image data to the server.

[1456] The "recognition means" refers to an AI image analysis algorithm and its processing device that analyzes image data uploaded to the server and automatically identifies the ingredients inside.

[1457] The "generation means" refers to an algorithm and a processing device for generating an optimal recipe based on the ingredient information identified by the recognition means and the user's health information and preference information.

[1458] The "transmission means" refers to a communication device and its protocol for transmitting the recipe information generated by the generation means to a user terminal such as a smartphone app of the user.

[1459] A "user terminal" is an electronic device such as a smartphone or tablet used by a user, and is used to receive, display, and adjust suggested recipe information.

[1460] "Health Information" refers collectively to a User's weight, number of steps, age, and other health-related data.

[1461] "Preference information" is a general term for information about the types of food and seasonings that a user likes.

[1462] A "generative AI model" is an advanced artificial intelligence algorithm, such as GPT-4, that generates new information, especially recipe information, based on input data.

[1463] A "prompt sentence" is an input sentence used to cause an AI model to generate a particular output.

[1464] This invention is a system that manages ingredients and suggests recipes to users using acquisition means, recognition means, generation means, and transmission means installed in refrigeration equipment. This system efficiently manages ingredients in the refrigerator and provides optimal recipes based on the user's health information and preference information, thereby helping to reduce food waste and promote healthy eating habits.

[1465] Hardware and Software Configuration

[1466] 1. Camera inside the refrigeration equipment (acquisition method)

[1467] Terminal (refrigeration equipment): A camera is installed inside the refrigerator and takes images of the inside of the refrigerator at regular intervals or according to the user's instructions. This camera uploads the image data to a server via an internet connection.

[1468] 2. Image analysis and food ingredient recognition (recognition method)

[1469] Server: The server receives the uploaded image data and automatically recognizes the ingredients in the refrigerator using an image analysis algorithm that uses deep learning. This algorithm uses a deep learning framework such as TensorFlow. The recognized ingredient information is stored in a database and the internal status of the refrigerator is updated in real time.

[1470] 3. Recipe generation (generation method)

[1471] Server: The server receives health information (number of steps, weight, age, etc.) and preference information (preferred cuisine, seasonings, etc.) from the user's smartphone app. Based on this, it uses a generative AI model (e.g., GPT-4) to generate the optimal recipe for the user. In this process, recipes are created taking into account the priority of ingredient use and nutritional balance.

[1472] 4. Notification of information and recipe suggestions (transmission method)

[1473] Server: Sends the generated recipe information to the user's smartphone app, where the user can review the suggested recipe and make adjustments as needed.

[1474] Specific examples

[1475] For example, if cabbage, carrots, and chicken are stored in the refrigerator, the following processing will occur:

[1476] 1. Terminal (refrigeration equipment): The camera takes pictures of the inside of the refrigerator and uploads the image data to the server.

[1477] 2. Server: Analyzes the uploaded image and automatically recognizes "cabbage," "carrot," and "chicken."

[1478] 3. User: Enters "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food" into the smartphone app and sends this information to the server.

[1479] 4. Server: Integrates the recognized ingredient information with the user's health information and generates recipe suggestions such as "Teriyaki Chicken" or "Japanese-style Cabbage and Carrot Salad."

[1480] 5. User: Check the suggested recipes on the smartphone app, select "Teriyaki Chicken," and make adjustments such as "Make it stronger in seasoning."

[1481] 6. Server: Regenerates the adjusted recipe and sends the final cooking instructions and required seasoning list to your smartphone.

[1482] 7. User: Can cook food according to the recipe displayed on the smartphone app.

[1483] Prompt Sentence Examples

[1484] For example, you can generate a recipe by inputting the following prompt into a generative AI model:

[1485] Prompt statement:

[1486] There are "cabbage," "carrots," and "chicken" in the refrigerator. The user's health information is as follows: "Today's steps: 5,000," "Weight: 70 kg," and "Preferences: Japanese food." Based on this, please suggest a healthy and delicious recipe.

[1487] output:

[1488] 1. Teriyaki chicken

[1489] 2. Japanese-style cabbage and carrot salad

[1490] In this way, the present invention realizes a system that can support a healthy diet by effectively utilizing ingredients stored in refrigeration equipment and providing optimal recipes based on the user's health and preference information.

[1491] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1492] Step 1:

[1493] Terminal (refrigeration equipment): A camera installed inside the refrigerator takes pictures of the inside of the refrigerator at regular intervals or according to the user's instructions.

[1494] Input: User instruction or scheduled trigger

[1495] Output: Image data of the inside of the refrigerator

[1496] Specifically, the camera captures high-resolution images from different angles inside the refrigerator, making all the ingredients inside visible.

[1497] Step 2:

[1498] Terminal (refrigeration equipment): Uploads the acquired image data to a server via an internet connection.

[1499] Input: Image data of the inside of the refrigerator

[1500] Output: Image data sent to the server

[1501] Specifically, the image data is compressed and transferred to a server using a secure protocol.

[1502] Step 3:

[1503] Server: Receives uploaded image data and performs image analysis using a deep learning framework (e.g., TensorFlow).

[1504] Input: Uploaded image data of the inside of the refrigerator

[1505] Output: Recognized ingredients

[1506] Specifically, the server runs an object detection algorithm to identify the type and location of ingredients in the image and stores this information in a database.

[1507] Step 4:

[1508] User: Enters health information (e.g., number of steps, weight, age) and preference information (e.g., favorite food, seasoning) into the smartphone app.

[1509] Input: User health and preference information

[1510] Output: User information sent to the server

[1511] Specifically, the user uses a smartphone app to input information using an intuitive interface, and then presses the send button to transfer the information to the server.

[1512] Step 5:

[1513] Server: Based on the collected ingredient information and user information, a generative AI model (e.g., GPT-4) is used to generate the optimal recipe.

[1514] Input: Recognized food ingredients and user health and preference information

[1515] Output: The generated recipe

[1516] Specifically, the generative AI model uses prompt text to create multiple recipes that meet the user's requirements and selects the most suitable one.

[1517] Step 6:

[1518] Server: Sends the generated recipe information to the user's smartphone app.

[1519] Input: Generated recipe

[1520] Output: Recipe information sent to the smartphone app

[1521] Specifically, recipe information is sent from the server using a secure protocol such as SSL, and a notification is displayed on the smartphone app.

[1522] Step 7:

[1523] User: Checks the suggested recipe on the smartphone app and makes adjustments as needed.

[1524] Input: Submitted recipe information

[1525] Output: Adjusted recipe request

[1526] Specifically, the user selects a recipe within the app, sets adjustments such as "stronger seasoning" or "add ingredients," and then sends a request to the server again.

[1527] Step 8:

[1528] Server: Based on the adjusted request, the server regenerates the final recipe and the list of required condiments and sends them to the smartphone.

[1529] Input: Adjustment request from user

[1530] Output: Final recipe and seasoning list

[1531] Specifically, the AI ​​model regenerates the recipe, determines the final recipe information reflecting the adjustments, and sends it to the smartphone app.

[1532] Step 9:

[1533] User: Cooks a dish according to the final recipe displayed on the smartphone app.

[1534] Input: Final recipe and seasoning list

[1535] Output: Finished dish

[1536] Specifically, the user follows the steps displayed on the smartphone app and completes the dish using the suggested seasonings.

[1537] (Application example 1)

[1538] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1539] In conventional refrigeration equipment, food ingredients are managed and expiration dates are checked manually, which leads to food waste and makes it difficult for users to receive appropriate recipe suggestions that take into account their health and preferences. Additionally, ordering ingredients that are in short supply is a time-consuming process. This creates problems that lower the quality of life for users.

[1540] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1541] In this invention, the server is located inside the refrigeration equipment and includes an acquisition means for acquiring images, a recognition means for processing the acquired images and automatically recognizing ingredients, a generation means for generating a recipe based on the recognized ingredient information, a transmission means for transmitting the generated recipe to a user terminal, and an ordering means for automatically ordering missing ingredients via the transmission means. This enables more efficient ingredient management, reduced food waste, recipe suggestions that take into account the user's health information and preference information, and automatic ordering of missing ingredients.

[1542] "Capture means" refers to the devices and techniques placed inside the refrigeration unit for capturing images.

[1543] The "recognition means" refers to the technology and algorithms for processing the acquired images and automatically recognizing ingredients.

[1544] The "generation means" refers to software and calculation techniques for generating recipes based on the ingredient information acquired by the recognition means.

[1545] The "transmission means" refers to a communication technique and device for transmitting the recipe generated by the generation means to the user terminal.

[1546] "Ordering means" refers to the system and protocol for automatically ordering missing ingredients via the transmission means.

[1547] "Health information" refers to health data such as the user's number of steps, weight, body fat percentage, and blood pressure.

[1548] "Preference information" refers to data regarding a user's favorite dishes and seasonings.

[1549] A "refrigeration equipment" is a home appliance used to store food at the appropriate temperature.

[1550] A "user terminal" is a mobile information terminal such as a smartphone or tablet operated by a user.

[1551] MODE FOR CARRYING OUT THE INVENTION

[1552] The present invention provides a system for managing ingredients for a user, proposing recipes, and automatically ordering ingredients that are in short supply. The system includes an acquisition unit, a recognition unit, a generation unit, a transmission unit, and an ordering unit that are arranged inside a refrigeration facility.

[1553] System Configuration

[1554] 1. Acquisition method:

[1555] A camera installed inside the refrigeration equipment functions as the acquisition means. This camera takes images of the inside at regular intervals or according to user instructions and uploads them to a server.

[1556] 2. Recognition means:

[1557] The AI ​​image analysis algorithm installed on the server processes the captured image data and automatically recognizes ingredients in the refrigerator, using software libraries such as TensorFlow and OpenCV.

[1558] 3. Generation means:

[1559] Based on the ingredient information stored on the server, a generative AI model is used to generate recipes based on the user's health and preference information.

[1560] 4. Means of transmission:

[1561] The generated recipe is sent to the user's device, which has a smartphone application installed, allowing the user to check the recipe suggestions.

[1562] 5. Ordering Method:

[1563] Based on the ingredient information recognized by the recognition means and the recipe generated by the generation means, the missing ingredients are automatically ordered from a food delivery service. This process uses a RESTful API.

[1564] System Operation Overview

[1565] The system of the present invention operates as follows:

[1566] 1. Images of the inside of the refrigeration equipment are periodically acquired by the acquisition means and uploaded to the server.

[1567] 2. The image data acquired by the recognition means is processed using an AI image analysis algorithm to recognize the ingredients in the refrigeration equipment.

[1568] 3. The recognized food ingredient information is stored on the server and integrated with the user's health and preference information.

[1569] 4. The generation means generates a recipe that takes into account the user's health information (number of steps, weight, etc.) and preference information (favorite dishes, seasonings, etc.).

[1570] 5. The suggested recipes are sent to the user's smartphone via the transmission means, where the user can check them.

[1571] 6. If necessary, any missing ingredients will be automatically ordered from the food delivery service via the ordering method.

[1572] Specific examples

[1573] For example, imagine a refrigerator contains cabbage, carrots, and chicken, but is short on pork and soy sauce. A camera takes a picture of the inside of the refrigerator and uploads the image data to a server. As a result of image analysis, "cabbage," "carrots," and "chicken" are recognized and stored on the server. The user's health information (for example, "Today's steps: 5,000 steps," "Weight: 70 kg," "Preferences: Japanese food") is also sent to the server. Based on this information, a generative AI model is used to suggest recipes such as "Teriyaki Chicken." The missing ingredients (pork and soy sauce) are automatically ordered and delivered to the user's home.

[1574] Prompt Sentence Examples

[1575] "Please suggest a Japanese recipe using cabbage and carrots. The user's health information is: 5,000 steps, weight: 70 kg."

[1576] As described above, by using this system, users can reduce the effort required for managing ingredients, while receiving recipe suggestions that match their health information and preferences, and automatically ordering any ingredients they are missing.

[1577] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1578] Step 1:

[1579] The terminal (a camera inside the refrigeration equipment) captures the image.

[1580] Input: Visual data of the inside of the refrigerator

[1581] Output: Acquired image data

[1582] Specific operation: A camera inside the refrigerator captures images of the inside of the refrigerator at regular intervals or according to the user's instructions. These images are temporarily stored on the device and then uploaded to the server.

[1583] Step 2:

[1584] The image data acquired by the terminal is uploaded to the server.

[1585] Input: Acquired image data

[1586] Output: Uploaded image data

[1587] Specific operation: The device sends image data to the server via an Internet connection, and the server stores the received data in temporary storage.

[1588] Step 3:

[1589] The server processes the image data using an AI image analysis algorithm and automatically recognizes the ingredients inside.

[1590] Input: Uploaded image data

[1591] Output: List of recognized ingredients

[1592] How it works: Using AI image analysis algorithms (e.g., TensorFlow and OpenCV) running on the server, each ingredient in the image is identified. The recognized ingredient information is then stored in a database.

[1593] Step 4:

[1594] The server retrieves the user's health and preference information from the database and integrates it with food ingredient information.

[1595] Input: Recognized ingredient list, user health information, and preference information

[1596] Output: Integrated user and ingredient information

[1597] How it works: The server retrieves the user's health and preference information from the database and combines it with the recognized ingredient list. This combined data is used to generate recipes.

[1598] Step 5:

[1599] The server uses a generative AI model to generate recipes based on the integrated data.

[1600] Input: Integrated user and ingredient information

[1601] Output: The generated recipe

[1602] How it works: The server uses a generative AI model to generate an optimal recipe from the integrated data. This process may involve prompts.

[1603] Step 6:

[1604] The server transmits the generated recipe to the user terminal.

[1605] Input: Generated recipe

[1606] Output: Recipe information sent to the user's device

[1607] Specific operation: The server sends the generated recipe to the user device (smartphone app), which receives it and displays it to the user.

[1608] Step 7:

[1609] The user terminal checks the recipe and places an order for any missing ingredients.

[1610] Input: Recipe information sent to the user's device

[1611] Output: List of missing ingredients and order request

[1612] How it works: The user checks the recipe and checks for missing ingredients through the app. The missing ingredients are automatically recognized and an order request is sent to the food delivery service through the ordering method.

[1613] Through these processing steps, users can efficiently manage ingredients and receive appropriate recipe suggestions. In addition, ingredients that are in short supply are automatically ordered, improving the quality of life.

[1614] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1615] The present invention is a system that equips a refrigeration unit with acquisition means, recognition means, generation means, transmission means, and an emotion engine, and not only manages ingredients and suggests recipes for the user, but also suggests individual recipes that take the user's emotions into consideration.The system aims to suggest more appropriate recipes based on the user's health information, preference information, and current emotional state.

[1616] Overall system configuration

[1617] The system mainly consists of the following components:

[1618] 1. Camera inside the refrigeration equipment (acquisition method)

[1619] 2. Server that performs AI processing (recognition means, generation means, transmission means, emotion engine)

[1620] 3. User's smartphone app (user device)

[1621] System Operation Overview

[1622] 1. Camera inside the refrigeration equipment (acquisition method)

[1623] Terminal (refrigerator): A camera inside the refrigeration equipment captures images of the interior at regular intervals or according to user instructions.

[1624] Terminal (refrigerator): Uploads the acquired image data to the server.

[1625] 2. Image analysis and food ingredient recognition (recognition method)

[1626] Server: The server acquires the received image data and automatically recognizes the ingredients using an AI image analysis algorithm.

[1627] Server: Stores the recognized ingredient information in a database and updates the status inside the refrigeration equipment.

[1628] 3. Recipe generation (generation method)

[1629] Server: Generates optimal recipes based on the user's health information (number of steps, weight, age, etc.), preference information (favorite dishes and seasonings, etc.), and emotional information.

[1630] Server: For example, when a user is feeling stressed, the server suggests recipes that will help them relax.

[1631] 4. Emotion Recognition (Emotion Engine)

[1632] User: Emotions are captured from the user's voice, facial expressions, and text input via a smartphone app or a camera or microphone built into the refrigeration equipment display.

[1633] Server: Analyzes the acquired emotion data and identifies the user's current emotional state.

[1634] 5. Notification of information and recipe suggestions (transmission method)

[1635] Server: Sends the generated recipe to the user's smartphone app.

[1636] Users can view suggested recipes through a smartphone app.

[1637] Specific examples

[1638] For example, consider a case where cabbage, carrots, and chicken are stored in a refrigeration facility.

[1639] 1. Terminal (refrigerator): The camera takes pictures of the inside of the refrigerator and uploads the image data to the server.

[1640] 2. Server: Analyzes the uploaded image and automatically recognizes "cabbage," "carrot," and "chicken."

[1641] 3. User: Enters the following information into the smartphone app: "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food." At this time, the built-in camera and microphone determine that the user is feeling a little stressed.

[1642] 4. Server: Integrates the ingredient list, health information, preference information, and emotional information to suggest dishes like "teriyaki chicken" and "Japanese-style cabbage and carrot salad," as well as "herbal tea" for relaxation.

[1643] 5. User: Selects "Teriyaki Chicken" from the recipes suggested by the smartphone app and makes adjustments such as "making it stronger in seasoning."

[1644] 6. Server: Generates the fine-tuned recipe and sends the final cooking instructions and required seasoning list to the user's smartphone.

[1645] 7. User: Cooks food according to the recipe displayed on the smartphone app.

[1646] In this way, the present invention is a system that efficiently utilizes ingredients stored in refrigeration equipment and suggests optimal recipes based on the user's health information, preference information, and emotional state, helping to reduce food waste and solving daily meal problems.

[1647] The processing flow will be explained below.

[1648] Step 1:

[1649] Terminal (refrigerator): The camera inside the refrigerator takes pictures of the inside at a set time. For example, if it is set to take pictures every day at 10:00, the camera will automatically operate at that time.

[1650] Step 2:

[1651] Terminal (refrigerator): The acquired image data is sent to the server via the internal storage device of the refrigeration equipment or the network.

[1652] Step 3:

[1653] Server: The server takes the received image data and begins processing it using AI image analysis algorithms, specifically detecting objects in the image and identifying which ingredients they correspond to.

[1654] Step 4:

[1655] Server: Stores the recognized ingredient information in a database. Newly recognized ingredients and updated information on existing ingredients are reflected in the database.

[1656] Step 5:

[1657] User: Opens the smartphone app and enters health and preference information. For example, enter "Today's steps: 5,000 steps," "Weight: 70 kg," and "Preferences: Japanese food."

[1658] Step 6:

[1659] Terminal (smartphone): The entered health information and preference information is sent to the server.

[1660] Step 7:

[1661] User: Using a smartphone app and the built-in camera and microphone in the refrigerator display, emotional information is acquired from voice and facial expressions. For example, emotions such as "I'm tired" or "I want to relax" are analyzed.

[1662] Step 8:

[1663] Server: Using the emotion engine, analyze the acquired emotion data and identify the user's current emotional state.

[1664] Step 9:

[1665] Server: Integrates the acquired information on ingredients, health, preferences, and emotions to generate optimal recipes. For example, a user feeling stressed might be suggested a recipe such as "herbal tea" that has a relaxing effect.

[1666] Step 10:

[1667] Server: Sends the generated recipe to the user's smartphone.

[1668] Step 11:

[1669] User: Check the recipes displayed on the smartphone app and select the desired recipe. For example, select "Teriyaki Chicken."

[1670] Step 12:

[1671] User: Requests seasoning or portion adjustments for a selected recipe. For example, they might type, "I'd like it a little stronger today."

[1672] Step 13:

[1673] Server: Fine-tunes the recipe data based on the user's request and generates the final cooking steps and list of required seasonings.

[1674] Step 14:

[1675] Server: Sends the fine-tuned recipe and cooking instructions to the user's smartphone.

[1676] Step 15:

[1677] User: Cooks a dish by following the cooking instructions displayed on the smartphone app.

[1678] Step 16:

[1679] Server: Regularly checks the food database in the refrigeration facility to identify food items that are close to their expiration date.

[1680] Step 17:

[1681] Server: Generates recipes that prioritize ingredients with an approaching expiration date and sends notification information to the user's smartphone.

[1682] Step 18:

[1683] Terminal (refrigerator): Notifications regarding expiration dates are displayed on the refrigerator equipment display and on the user's smartphone app.

[1684] Step 19:

[1685] User: Check the notification, use the suggested recipes as a guide, and prioritize ingredients that are close to their expiration date.

[1686] Through these steps, users can efficiently utilize the food stored in their refrigerators, helping them manage their health and reduce food waste.

[1687] Example 2

[1688] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1689] Today, there is a demand for efficient use of ingredients in refrigeration equipment, reducing food waste, and individually optimized recipe suggestions based on the user's health information, preferences, and even emotional state. However, conventional systems do not adequately consider the user's emotional state when managing ingredients or suggesting recipes, resulting in insufficient improvement in user satisfaction. The present invention aims to solve these problems by suggesting optimal recipes based on the user's health, preferences, and emotional state.

[1690] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server is disposed inside the refrigeration equipment and includes an acquisition means for acquiring images, a recognition means for processing the acquired images and automatically recognizing ingredients, a generation means for generating recipes based on ingredient information recognized by the recognition means, an emotion recognition means for analyzing user emotion information and generating recipes based on the emotional state, and a transmission means for transmitting the recipes generated by the generation means to a user terminal. This makes it possible to efficiently utilize ingredients in the refrigeration equipment and propose individually optimized recipes based on the user's health, preferences, and emotional state.

[1691] "Acquisition means" is a general term for hardware such as cameras and sensors that are placed inside the refrigeration equipment to acquire image data, and the software that controls them.

[1692] "Recognition means" is a general term for AI algorithms and software processing that processes image data acquired by the acquisition means and automatically identifies ingredients.

[1693] The "generation means" is a general term for programs and algorithms for generating recipes based on the ingredient information recognized by the recognition means and other related data.

[1694] "Emotion recognition means" is a general term for hardware and software that acquires emotional data from a user's voice, facial expression, text input, etc., and analyzes it to identify the user's emotional state.

[1695] The "transmission means" is a general term for the communication function for transmitting the recipe and notification information generated by the generation means to the user's terminal and the program that controls it.

[1696] A "user terminal" is a device that a user operates and views, and includes a smartphone, tablet, PC, etc.

[1697] MODE FOR CARRYING OUT THE INVENTION

[1698] The present invention is a system for managing ingredients and suggesting recipes to users by equipping a refrigeration unit with an acquisition means, a recognition means, a generation means, a transmission means, and an emotion recognition means. This system uses hardware and software such as a camera inside the refrigeration unit as an acquisition means, an AI image analysis algorithm as a recognition means, a machine learning model and a rule-based algorithm as a generation means, a communication function as a transmission means, and a voice recognition and facial expression analysis algorithm as an emotion recognition means.

[1699] Hardware and Software Use

[1700] 1. Camera inside the refrigeration equipment (acquisition method)

[1701] Terminal (refrigerator): A camera installed inside the refrigeration equipment captures images of the inside periodically or at the user's command. For example, the camera saves the image data in JPEG format and uploads it to a server via Wi-Fi.

[1702] 2. Image analysis and food ingredient recognition (recognition method)

[1703] Server: The server processes the received image data using an AI image analysis algorithm using TensorFlow and OpenCV to automatically recognize ingredients. The recognized ingredient information is stored in a database in JSON format.

[1704] 3. Emotion recognition (emotion recognition means)

[1705] User: Emotional data is collected from facial expressions, voice, and text input using a smartphone app or a camera or microphone built into the refrigeration equipment display. For example, emotional data is input by answering questions within the app.

[1706] Server: The emotion recognition engine analyzes the acquired data using natural language processing algorithms to identify the user's current emotional state, and records the results in a database.

[1707] 4. Recipe generation (generation method)

[1708] Server: The server integrates the user's health information (e.g., number of steps and weight), preference information (e.g., favorite dishes and seasonings), and emotional information to generate optimal recipes using machine learning models and rule-based algorithms. For example, if the user is feeling stressed, it will suggest recipes with a relaxing effect.

[1709] 5. Notification of information and recipe suggestions (transmission method)

[1710] Server: The generated recipe is sent to the user's device in JSON or XML format.

[1711] User: Using the smartphone app, view the suggested recipes and make adjustments to finalize the recipe. Recipe details and cooking instructions are displayed in the app.

[1712] Examples of specific examples and prompts

[1713] For example, imagine a refrigerator containing cabbage, carrots, and chicken. The refrigerator's camera takes a picture of the interior at 8:00 AM and uploads it to the server as "fridge_image_20230315_0800.jpg." The server uses TensorFlow to recognize "cabbage," "carrot," and "chicken" and stores them in a database. The user enters "Today's steps: 5,000," "Weight: 70 kg," and "Preferences: Japanese food" into a smartphone app, and the built-in camera and microphone determine that the user is "feeling a little stressed." Based on this information, the server suggests "teriyaki chicken," "Japanese-style cabbage and carrot salad," and "herbal tea" for relaxation. The user then adjusts and confirms the recipe.

[1714] Prompt Sentence Examples

[1715] "The ingredients stored in the refrigerator are cabbage, carrots, and chicken. The user likes Japanese food, has walked 5,000 steps today, and weighs 70 kg. The user is also feeling a little stressed. Based on this information, please suggest a Japanese recipe that will have a relaxing effect."

[1716] As described above, the present invention is a system that efficiently manages ingredients in a refrigeration facility and provides individually optimized recipes that take into account the user's health information, preference information, and even emotional state.

[1717] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1718] Step 1: Acquire and upload images

[1719] Terminal (refrigerator):

[1720] A camera inside the refrigeration equipment periodically captures images of the interior, for example, at 8:00 a.m. every day. Users can also manually instruct the camera to take pictures.

[1721] Input: User instructions or set scheduled time

[1722] Output: JPEG format image file

[1723] The captured image is saved as "fridge_image_yyyyMMdd_HHmm.jpg" and uploaded to the server via Wi-Fi.

[1724] Step 2: Image analysis and ingredient recognition

[1725] server:

[1726] The image data received by the server is stored in " / images / received / ".

[1727] Input: Uploaded JPEG image file

[1728] Processing: Recognize ingredients in the image using image analysis algorithms based on TensorFlow and OpenCV

[1729] Output: Ingredient information in JSON format

[1730] The recognized ingredient information is stored in the "food_items" table. For example, "cabbage," "carrot," and "chicken" are identified.

[1731] Step 3: Emotion Recognition

[1732] User:

[1733] Facial expressions, voice, and text input are provided via a smartphone app or a camera and microphone built into the refrigeration equipment display.

[1734] Input: Voice data, facial expression images, text data

[1735] Output: Emotion data

[1736] server:

[1737] An emotion recognition engine analyzes the input data and uses natural language processing algorithms to identify the user's emotional state.

[1738] Input: Voice data, facial expression images, text data

[1739] Processing: Emotional data analysis

[1740] Output: Emotional state (e.g., "I'm feeling a little stressed")

[1741] The analysis results are recorded in the "user_emotions" table.

[1742] Step 4: Recipe Generation

[1743] server:

[1744] The system integrates the user's health information, preference information, emotional information, and information on ingredients in the refrigerator to generate optimal recipes.

[1745] Input: Health information (number of steps, weight), preference information, emotional information, information on ingredients in the refrigerator

[1746] Processing: Generate recipes using machine learning models and rule-based algorithms

[1747] Output: Suggested recipe

[1748] For example, if a user is feeling stressed, the app will suggest relaxing dishes such as "teriyaki chicken," "Japanese-style cabbage and carrot salad," and even "herbal tea."

[1749] Step 5: Information and recipe suggestions

[1750] server:

[1751] The generated recipe is sent to the user's device in JSON or XML format.

[1752] Input: The generated recipe and other relevant information

[1753] Output: Data sent to the user terminal

[1754] User:

[1755] The suggested recipe is viewed on a smartphone app, and adjustments are made as needed. Finally, the final recipe is confirmed.

[1756] Input: Recipe information sent from the server

[1757] Output: Finalized recipe and cooking instructions

[1758] For example, if a refrigerator contains cabbage, carrots, and chicken, the refrigerator's camera periodically captures images and uploads them to a server. The server then analyzes the images and recognizes the ingredients. The user provides health and emotional information through a smartphone app, and the server then suggests recipes with a relaxing effect. Finally, the user can view and adjust the recipe and cook the food.

[1759] Example prompt sentence:

[1760] "The ingredients stored in the refrigerator are cabbage, carrots, and chicken. The user likes Japanese food, has walked 5,000 steps today, and weighs 70 kg. The user is also feeling a little stressed. Based on this information, please suggest a Japanese recipe that will have a relaxing effect."

[1761] (Application example 2)

[1762] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1763] For modern consumers, managing ingredients in refrigerated facilities and proposing optimal recipes based on that information is a very time-consuming task. Managing expiration dates for ingredients, creating recipes based on that information, and quickly sourcing missing ingredients are also challenges. Furthermore, most existing systems do not offer recipe suggestions that take into account the user's health information or emotional state. This situation could exacerbate food waste and health management issues. The present invention aims to solve these challenges by providing a system that manages ingredients, proposes recipes, automatically orders ingredients, and provides virtual experiences.

[1764] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1765] In this invention, the server includes an acquisition means for acquiring images, a recognition means for processing the acquired images and automatically recognizing ingredients, a generation means for generating a recipe based on the ingredient information recognized by the recognition means, a transmission means for transmitting the recipe generated by the generation means to a user terminal, an ordering means for automatically ordering missing ingredients, and a virtual reality means for providing a virtual experience, thereby enabling the user to efficiently manage ingredients in the refrigeration equipment, receive optimal recipe suggestions based on health information and emotional state, quickly order missing ingredients, and enjoy an intuitive virtual shopping experience.

[1766] Definition of Terms

[1767] "Means for acquiring images" refers to cameras or sensors that take images of the inside of the refrigeration equipment and acquire them as digital data.

[1768] "Means for processing acquired images and automatically recognizing ingredients" refers to a device or software that analyzes acquired image data and identifies the ingredients contained therein using AI or machine learning algorithms.

[1769] "Generation means for generating a recipe based on the ingredient information recognized by the recognition means" refers to a device or software that creates an optimal recipe based on the ingredient information obtained by the recognition means, taking into consideration the user's health information, preference information, emotional state, etc.

[1770] "Transmission means for transmitting the recipe generated by the generation means to a user terminal" refers to a device or software that transmits the generated recipe to a terminal such as a user's smartphone or tablet via a network.

[1771] "Ordering means for automatically ordering missing ingredients" means a device or software that automatically orders missing ingredients online based on the generated recipe.

[1772] "Virtual reality means for providing a virtual experience" means a device or software that provides a virtual reality (VR) or augmented reality (AR) environment that can be intuitively operated by a user.

[1773] "Health information" refers to physical data such as the user's number of steps, weight, age, etc., and particularly includes data related to dietary habits.

[1774] "Preference information" refers to information about a user's favorite dishes and seasonings, and particularly includes data that reflects an individual's food preferences.

[1775] "Emotional state" refers to the mental state of the user and includes data for inferring emotions from voice, facial expressions, text input, etc.

[1776] MODE FOR CARRYING OUT THE INVENTION

[1777] This invention realizes a system that uses an AI model to recognize ingredients based on images captured by a camera inside the refrigeration equipment, and then generates and suggests recipes to the user. It also has the ability to suggest more personalized recipes and automatically order missing ingredients by taking into account the user's health information and emotional state. Below, we will explain each element of this system in detail.

[1778] Hardware and software used

[1779] Hardware: Cameras in refrigeration equipment, smartphones, server PCs, smart glasses

[1780] Software: Python, Flask, TensorFlow, OpenAI API

[1781] System configuration and operation

[1782] 1. Camera inside the refrigeration equipment (acquisition method)

[1783] Cameras placed inside the refrigeration equipment periodically or at the user's command capture images of the interior, which are then uploaded as digital data to a server.

[1784] 2. Image analysis and food ingredient recognition (recognition method)

[1785] The server analyzes the image data received from the acquisition means and automatically recognizes ingredients using an AI model trained using TensorFlow.

[1786] 3. Recipe generation (generation method)

[1787] The server generates the most suitable recipe based on the ingredient information identified by the recognition means, taking into consideration the user's health information (e.g., number of steps, weight) and preference information (e.g., favorite dishes, seasonings). Furthermore, to take the user's emotional state into account, it uses emotional information obtained by analyzing their voice and facial expressions. At this point, the generated recipe is sent to the smartphone app.

[1788] 4. Automatic ordering of missing ingredients (ordering method)

[1789] Based on the generated recipe, any missing ingredients are automatically ordered online, allowing users to quickly get the ingredients they need.

[1790] 5. Virtual Experience (Virtual Reality Methods)

[1791] Users can enjoy an intuitive virtual experience using smart glasses or a smartphone, making refrigerator food management and online shopping even more convenient.

[1792] Specific examples

[1793] For example, imagine a refrigerator containing cabbage, carrots, and chicken. A camera takes images of these ingredients and uploads them to a server. The server recognizes these ingredients and compares them with the user's health information (e.g., 5,000 steps today, weight 70 kg) and preference information (e.g., likes Japanese food) against the input data. It also takes into account the user's emotional state, as determined from their voice and facial expressions (e.g., feeling a little stressed), and generates the most suitable recipe (e.g., teriyaki chicken, Japanese-style cabbage and carrot salad, and relaxing herbal tea).

[1794] The generated recipe is sent to a smartphone app, where users can use it to create the dish, and if any ingredients are missing, the system will automatically order them online and have them available immediately.

[1795] Prompt Sentence Examples

[1796] "Ingredients in the refrigerator: [cabbage, carrots, chicken]. Health information: {"steps": 5000, "weight": 70kg}. Preference information: "I like Japanese food, and I like it strong." Emotional state: "I'm feeling a little stressed." Please suggest recipes based on this."

[1797] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1798] Processing steps of the system that realizes the application example

[1799] Step 1:

[1800] The terminal (a camera inside the refrigerator) periodically captures images of the inside of the refrigerator or when instructed by the user. This image data is captured and saved in digital format. The input is the current state of the inside of the refrigerator, and the output is the digital image data.

[1801] Step 2:

[1802] The image data acquired by the device is uploaded to the server. The input here is the image data acquired in step 1, and the output is the image data transferred to the server.

[1803] Step 3:

[1804] The server processes the uploaded image data and automatically recognizes ingredients using an AI image recognition algorithm. A model using TensorFlow analyzes the image. The image data is given as input, and the output is the recognized ingredient information.

[1805] Step 4:

[1806] The recognized ingredient information is stored in a database on the server, and the status inside the refrigeration equipment is updated. The input is the recognized ingredient information, and the output is the updated database information.

[1807] Step 5:

[1808] Users use a smartphone app to input health information (e.g., number of steps, weight) and preference information (e.g., favorite food). In addition, emotional information is acquired through the built-in camera and microphone. The user's health information, preference information, and emotional information are given as input, and the output is the integrated data.

[1809] Step 6:

[1810] The server generates the optimal recipe based on the integrated information (ingredient information, health information, preference information, and emotional information). In this process, recipe generation is performed based on prompts using OpenAI's API. The input is the integrated data, and the output is the generated recipe.

[1811] Step 7:

[1812] The generated recipe is sent to the user device (smartphone app). The input is the generated recipe, and the output is the recipe information displayed on the user device.

[1813] Step 8:

[1814] If necessary, the user can automatically order missing ingredients online based on the generated recipe. The input is the missing ingredient information, and the output is the ingredients obtained by online ordering.

[1815] Step 9:

[1816] Users use smart glasses or smartphones to control refrigeration equipment and enjoy virtual experiences, including ingredient management, recipe suggestions, and even virtual shopping. The input is the user's operation, and the output is the user experience.

[1817] Through these steps, the system will be able to provide comprehensive services, from ingredient management to recipe suggestions, online ordering of missing ingredients, and virtual experiences.

[1818] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1819] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1820] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1821] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1822] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1823] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1824] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1825] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1826] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1827] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1828] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1829] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1830] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1831] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1832] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1833] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1834] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1835] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1836] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1837] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1838] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1839] The following is further disclosed regarding the above embodiment.

[1840] (Claim 1)

[1841] an acquisition means disposed within the refrigeration equipment for acquiring an image;

[1842] a recognition means for processing the acquired image and automatically recognizing ingredients;

[1843] a generation means for generating a recipe based on the ingredient information recognized by the recognition means;

[1844] a transmitting means for transmitting the recipe generated by the generating means to a user terminal;

[1845] A system including:

[1846] (Claim 2)

[1847] 10. The system of claim 1, wherein the generating means further comprises means for adjusting the recipe based on health or preference information about the user.

[1848] (Claim 3)

[1849] 10. The system of claim 1, wherein the recognition means further comprises means for tracking expiration dates of ingredients in the refrigeration facility and generating recipes that prioritize ingredients with upcoming expiration dates.

[1850] "Example 1"

[1851] (Claim 1)

[1852] Acquisition means;

[1853] A recognition means;

[1854] generating means;

[1855] A transmitting means;

[1856] A user terminal;

[1857] A system including:

[1858] (Claim 2)

[1859] 2. The system of claim 1, wherein the recognition means is disposed within the refrigeration equipment and includes means for acquiring images, the means acquiring images at regular intervals or based on instructions from a user, and uploading the images to a server using an internet connection.

[1860] (Claim 3)

[1861] The system of claim 1, wherein the server includes means for receiving uploaded images, automatically recognizing ingredients in the images using an image analysis algorithm that uses deep learning, storing the recognized ingredient information in a database, and updating the internal status of the refrigeration equipment.

[1862] (Claim 4)

[1863] 2. The system according to claim 1, wherein the server includes means for receiving health information and preference information of a user and generating an optimal recipe based on the information.

[1864] (Claim 5)

[1865] The system of claim 1, wherein the generating means includes means for generating recipes using a generative AI model that takes into account a user's health information and ingredient information.

[1866] (Claim 6)

[1867] The system of claim 1 , wherein the server includes means for transmitting the generated recipe information to a user's smartphone, allowing the user to review the suggested recipe and adjust it if necessary.

[1868] (Claim 7)

[1869] 2. The system of claim 1, wherein the generating means further comprises means for tracking expiration dates of ingredients in the refrigeration facility and generating a recipe that prioritizes ingredients with upcoming expiration dates.

[1870] "Application Example 1"

[1871] (Claim 1)

[1872] an acquisition means disposed within the refrigeration equipment for acquiring an image;

[1873] a recognition means for processing the acquired image and automatically recognizing ingredients;

[1874] a generation means for generating a recipe based on the ingredient information recognized by the recognition means;

[1875] a transmitting means for transmitting the recipe generated by the generating means to a user terminal;

[1876] ordering means for automatically ordering the missing ingredients via the transmission means;

[1877] A system including:

[1878] (Claim 2)

[1879] 10. The system of claim 1, wherein the generating means further comprises means for adjusting the recipe based on health or preference information about the user.

[1880] (Claim 3)

[1881] 2. The system of claim 1, wherein the recognition means further comprises means for tracking expiration dates of ingredients in the refrigeration equipment and generating recipes that prioritize ingredients with upcoming expiration dates, and the ordering means further comprises means for automatically ordering ingredients that are in short supply.

[1882] "Example 2: Combining Emotion Engines"

[1883] (Claim 1)

[1884] an acquisition means disposed within the refrigeration equipment for acquiring an image;

[1885] a recognition means for processing the acquired image and automatically recognizing ingredients;

[1886] a generation means for generating a recipe based on the ingredient information recognized by the recognition means;

[1887] emotion recognition means for analyzing the user's emotion information and generating a recipe based on the user's emotion state;

[1888] a transmitting means for transmitting the recipe generated by the generating means to a user terminal;

[1889] A system including:

[1890] (Claim 2)

[1891] 10. The system of claim 1, wherein the generating means further comprises means for adjusting the recipe based on health or preference information about the user.

[1892] (Claim 3)

[1893] 10. The system of claim 1, wherein the recognition means further comprises means for tracking expiration dates of ingredients in the refrigeration facility and generating recipes that prioritize ingredients with upcoming expiration dates.

[1894] "Application example 2 when combining emotion engines"

[1895] Claims

[1896] (Claim 1)

[1897] an acquisition means for acquiring an image;

[1898] a recognition means for processing the acquired image and automatically recognizing ingredients;

[1899] a generation means for generating a recipe based on the ingredient information recognized by the recognition means;

[1900] a transmitting means for transmitting the recipe generated by the generating means to a user terminal;

[1901] an ordering means for automatically ordering missing ingredients;

[1902] a virtual reality means for providing a virtual experience;

[1903] A system including:

[1904] (Claim 2)

[1905] 10. The system of claim 1, wherein the generating means further comprises means for adjusting the recipe based on health or preference information and an emotional state of the user.

[1906] (Claim 3)

[1907] 10. The system of claim 1, wherein the recognition means further comprises means for tracking expiration dates of ingredients in the refrigeration facility and generating recipes that prioritize ingredients with upcoming expiration dates. [Explanation of symbols]

[1908] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. an acquisition means disposed within the refrigeration equipment for acquiring an image; a recognition means for processing the acquired image and automatically recognizing ingredients; a generation means for generating a recipe based on the ingredient information recognized by the recognition means; a transmitting means for transmitting the recipe generated by the generating means to a user terminal; A system including:

2. The system of claim 1 , wherein the generating means further comprises means for adjusting the recipe based on health or preference information about the user.

3. 10. The system of claim 1, wherein the recognition means further comprises means for tracking expiration dates of ingredients in the refrigeration facility and generating recipes that prioritize ingredients with upcoming expiration dates.

Images