System

Abstract

A system is provided.SOLUTION: A system includes means for capturing an image of an ingredient with a camera built in a refrigerator, means for transmitting image data of the ingredient to a server, means for analyzing the image data in the server and recognizing a type, an amount, a condition, and a use-by date of the ingredient, means for inputting a health condition by a user, means for generating a recipe based on the health condition and the ingredient data and transmitting the recipe to a user terminal, and means for displaying the generated recipe.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] While a large amount of food is wasted every year, many people continue to go hungry and lack access to healthy meals, creating serious social problems. Even at home, food waste and overlooking expiration dates on ingredients often result in unnecessary waste. It is also extremely difficult to create appropriate recipes that take into account each household's health status and dietary balance. Furthermore, when necessary ingredients are in short supply, the hassle of going out to buy them each time is a major burden. There is a need for a system that can solve these issues, reduce food waste, and support health maintenance. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a system that includes a means for taking images of ingredients using a camera built into the refrigerator, a means for sending image data of the ingredients to a server, a means for analyzing the image data in the server and recognizing the type, amount, condition and expiration date of the ingredients, a means for the user to input their health status, a means for generating a recipe based on the health status and ingredient data and sending it to the user's terminal, and a means for displaying the generated recipe.

[0006] Specifically, when ingredients are placed in the refrigerator, a camera automatically takes a picture and performs an initial analysis. The analysis results are sent via the internet to a server where a more detailed analysis is performed and the ingredient data is stored in a database. The user enters their health status through a smartphone app, and the AI ​​generates and suggests appropriate recipes based on the user's health status and the ingredient data in the refrigerator. Additionally, if any ingredients are missing, an order is automatically generated with a food delivery service and the necessary ingredients are delivered. This reduces food waste in the home and supports healthy eating habits.

[0007] The "camera" is a device installed inside the refrigerator that takes pictures of ingredients.

[0008] "Image data" is data that indicates image information of ingredients photographed by a camera.

[0009] A "server" is a computer system that receives image data via the Internet and performs analysis.

[0010] "Analysis" is the process of recognizing the type, quantity, condition and expiration date of ingredients based on image data.

[0011] "Food ingredient data" refers to information on the type, quantity, condition, and expiration date of food ingredients obtained through analysis.

[0012] A "database" is an information storage system for storing analyzed ingredient data.

[0013] A "user terminal" is a device such as a smartphone or tablet that a user uses to input their health status and receive and display recipes.

[0014] "Health condition" is information about physical condition and health entered by the user.

[0015] "Generative AI" is an artificial intelligence system that generates appropriate recipes based on health status and ingredient data.

[0016] A "recipe" is information including cooking steps and ingredient lists suggested by the generative AI based on health status and food ingredient data.

[0017] The "display means" is a function that displays the generated recipe on the user terminal.

[0018] A "food delivery service" is a service that delivers necessary food ingredients to homes.

[0019] "Order" is order information that automatically orders missing ingredients from a food delivery service. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention relates to a system that reduces food waste and supports household health maintenance by using a camera built into a refrigerator, a server, a generation AI, a user terminal, and a food delivery service. This system is specifically implemented as follows.

[0042] System Configuration

[0043] In-fridge camera: The camera is placed inside the refrigerator and is used to take pictures of the ingredients added.

[0044] User device: Consists of a smartphone, tablet, etc., and is used by users to input their health status and receive and display recipes.

[0045] Server: A cloud system that receives and analyzes image data. This is where food ingredient data is generated.

[0046] Generative AI: Artificial intelligence that generates appropriate recipes based on health and ingredient data.

[0047] Food delivery service: A service with an automatic ordering function to make up for missing ingredients.

[0048] Operation overview

[0049] Register ingredients and send data

[0050] Device: When a user places a new ingredient in the refrigerator, a camera takes a picture of it, which undergoes initial analysis and is then sent to the server.

[0051] Server: Receives image data and performs image analysis using a deep learning module. This allows the type, quantity, condition, and expiration date of ingredients to be recognized. The analysis results are stored in a database.

[0052] User health management

[0053] User: Using a smartphone app, the user enters their current health status, such as whether they are feeling unwell or on a diet.

[0054] Server: Updates the profile database based on the received health status data.

[0055] Recipe generation and distribution

[0056] Server: The AI ​​generates optimal recipes based on the stored ingredient data and the user's health status. If the user is not feeling well, it will generate recipes that are easy to digest, and under normal circumstances, it will generate recipes that prioritize ingredients that are close to their expiration date.

[0057] Server: Sends the generated recipe to the user's device.

[0058] User: A smartphone app displays recipe suggestions, such as "Today's recommended dish is cabbage and tomato salad."

[0059] Automatic ordering of missing ingredients

[0060] Server: Based on the proposed recipe, check the refrigerator for missing ingredients.

[0061] Server: Automatically generate and send orders to food delivery services to make up for missing ingredients.

[0062] User: Checks the status of a grocery delivery on a smartphone app, for example, with a notification that "cabbage and tomatoes are scheduled to be delivered tomorrow."

[0063] Specific examples

[0064] What to do when you are unwell

[0065] 1. User: The user types "I feel like I have a cold" into a smartphone app.

[0066] 2. Server: The server updates the health data and generates a porridge recipe suitable for the user's physical condition.

[0067] 3. Server: Sends the generated recipe to the user's device and notifies the user.

[0068] 4. User: The smartphone displays the message, "Let's make porridge today."

[0069] Utilizing ingredients that are close to their expiration date

[0070] 1. Terminal: The refrigerator camera takes a picture of the newly added milk and sends the image to the server.

[0071] 2. Server: Analyzes the image data and identifies the type of milk and its expiration date.

[0072] 3. Server: Updates the ingredient database and determines when the expiration date is approaching.

[0073] 4. Server: Generates recipes for dishes using milk (e.g., cream stew) and sends them to the user's device.

[0074] 5. User: The smartphone displays the message, "Let's make cream stew today."

[0075] The present invention makes it possible to reduce food waste and effectively support users in maintaining their health.

[0076] The processing flow will be explained below.

[0077] Specific processing of the program

[0078] Ingredient recognition and data transmission

[0079] Step 1:

[0080] Device: The camera built into the refrigerator takes pictures of the newly added ingredients by the user.

[0081] Step 2:

[0082] Terminal: Performs initial analysis of images captured by the camera and extracts features such as the shape, color, and tags of ingredients.

[0083] Step 3:

[0084] Terminal: Sends image data along with initial analysis data to a server via the internet.

[0085] Analysis of food ingredient data

[0086] Step 4:

[0087] Server: The received image data is input into a deep learning module to analyze the detailed type, quantity, and condition of the ingredients.

[0088] Step 5:

[0089] Server: Reads the expiration date on food packaging using OCR (optical character recognition) technology.

[0090] Step 6:

[0091] Server: Stores the analysis results in a food ingredient database. Stored data includes type, quantity, expiration date, and condition.

[0092] User Health Check

[0093] Step 7:

[0094] User: Launches the smartphone app and enters their current health and physical condition.

[0095] Step 8:

[0096] Server: Updates the received health status data in the user profile database.

[0097] Recipe generation and distribution

[0098] Step 9:

[0099] Server: Generative AI generates optimal recipes based on ingredient data and health status data.

[0100] Step 10:

[0101] Server: Sends the generated recipe to the user's device.

[0102] Step 11:

[0103] User: A smartphone app displays a suggested recipe, for example, "Today's recommended dish is cabbage and tomato salad."

[0104] Automatic ordering of missing ingredients

[0105] Step 12:

[0106] Server: Based on the proposed recipe, check the amount of ingredients currently in the refrigerator and check for any missing ingredients.

[0107] Step 13:

[0108] Server: Creates an order list to make up for any missing ingredients and generates order data for the food delivery service API.

[0109] Step 14:

[0110] User: Checks the status of grocery delivery on a smartphone app. For example, it shows "cabbage and tomatoes are scheduled to be delivered tomorrow."

[0111] Example 1

[0112] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0113] There is a need for efficient methods to reduce food waste in the home and provide healthy meals. However, current refrigerator management systems make it difficult to accurately grasp the status of ingredients and tend to waste ingredients that are approaching their expiration date. Furthermore, they do not suggest optimal recipes based on health status, which makes it difficult to adequately manage health at home.

[0114] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0115] In this invention, the server

[0116] A means for taking an image of the food using a camera built into the refrigerator;

[0117] means for transmitting image data of ingredients to a server;

[0118] A means for analyzing image data in the server and recognizing the type, amount, condition and expiration date of ingredients;

[0119] a means for a user to input a health status;

[0120] A means for generating a recipe by the artificial intelligence based on the health status and the ingredient data and transmitting the recipe to the user terminal;

[0121] A display means for the generated recipe;

[0122] Includes:

[0123] This will automate food ingredient management and suggest optimal recipes based on health status, reducing food waste and helping to maintain household health.

[0124] A "camera built into a refrigerator" is a device that is installed inside the refrigerator and takes pictures of ingredients.

[0125] "Food ingredients" is a general term for food stored in a refrigerator.

[0126] "Image data" is digital data that contains visual information of ingredients photographed by a camera.

[0127] The "server" is a computer system installed on the cloud that analyzes image data of ingredients and provides services based on the user's health condition.

[0128] "Analysis" is the process of recognizing the type, quantity, condition, and expiration date of ingredients based on image data.

[0129] A "deep learning module" is a type of artificial intelligence technology used to analyze images of ingredients, and utilizes a multi-layer neural network.

[0130] A "user terminal" is a device that allows a user to input their health status and receive notifications and recipes from the system, and includes smartphones, tablets, etc.

[0131] "Health condition data" is information about the current health condition entered by the user.

[0132] "Generative AI" is artificial intelligence that generates optimal recipes based on the user's health condition and ingredient data.

[0133] A "recipe" is a document that describes the steps for preparing a dish using specific ingredients.

[0134] A "food delivery service" is a commercial service that automatically orders and delivers food ingredients needed by users.

[0135] A "database" is a system for systematically storing analysis results and user health status data.

[0136] An "order" is request data for ordering the necessary ingredients from the ingredient delivery service.

[0137] The "profile database" is a database for storing a user's past health condition data and food consumption history.

[0138] This invention is a system for supporting household food waste reduction and health maintenance. The system's main components are a camera built into the refrigerator, a server, a generative AI model, a user terminal, and a food delivery service.

[0139] Refrigerator camera

[0140] The camera built into the refrigerator, which is the terminal, is a device that takes pictures of ingredients added to the refrigerator. A high-resolution camera is used and is positioned so that it covers the entire interior of the refrigerator. When a user puts new ingredients into the refrigerator, the camera automatically detects this and takes a picture. This picture is sent to a server on the cloud via Wi-Fi. A secure protocol (e.g., HTTPS) is used for transmission.

[0141] Server and Image Analysis

[0142] The server analyzes the received image data using a deep learning module. For example, it uses a deep learning library such as TensorFlow to identify the type, quantity, condition (e.g., fresh, spoiled), and expiration date of the ingredients. The results of this analysis are stored in a database (e.g., MySQL or MongoDB). The stored data includes details such as the ingredient name, quantity, expiration date, and the date and time of addition.

[0143] User health management

[0144] Using a smartphone app, users input their current health status (e.g., feeling a bit under the weather, dieting, high stress) using text fields and pull-down menus. This information is sent in real time to a server, which then updates the user's profile database. This profile also includes past health data and food consumption history.

[0145] Recipe generation and distribution using generative AI

[0146] The server retrieves the stored ingredient data and the user's health status data and generates optimal recipes using a generative AI model. Examples of generative AI models include OpenAI's GPT-3 and other custom models. The generative AI takes into account the user's health status and available ingredient information to create healthy recipes. For example, it might recommend porridge when feeling unwell or a low-calorie salad when on a diet. This generated recipe is sent to the user's device and displayed on a smartphone app.

[0147] Automatic ordering of missing ingredients

[0148] The server checks the refrigerator for missing ingredients based on the proposed recipe. If any ingredients are found to be missing, it automatically calls the food delivery service's API and generates an order. The order includes credit card information and delivery address. Delivery information from the food delivery service is sent to the user's device. The user can check the delivery status on their smartphone app.

[0149] Examples of concrete examples and prompts

[0150] As a concrete example, let's look at how to respond when you're feeling unwell. When a user enters "I'm feeling a bit sick" into a smartphone app, the server updates the health data and generates a porridge recipe suited to the user's physical condition. The generated recipe is sent to the user's device, and the smartphone displays a message saying, "Let's try making porridge today."

[0151] We will also give a concrete example of how to utilize ingredients that are approaching their expiration date. The camera in the refrigerator takes a picture of the newly added milk and sends the image to the server. The server analyzes the image data and recognizes the type of milk and its expiration date. It updates the ingredient database and determines that the expiration date is approaching. A recipe for a dish that uses the milk (for example, cream stew) is generated and sent to the user's device. The user is then prompted on their smartphone, "Let's try making cream stew today."

[0152] Examples of prompts:

[0153] "When you put new food in the refrigerator, the camera automatically takes a picture and sends it to the server."

[0154] "When a user inputs information about when they are feeling unwell into a smartphone app, an appropriate recipe is generated based on that information."

[0155] "A function that automatically orders missing ingredients from a delivery service based on the suggested recipe."

[0156] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0157] Step 1: Take a photo of the food

[0158] When a user adds new ingredients to the refrigerator, the device's built-in camera automatically takes a picture of the ingredients.

[0159] Input: New ingredients added to the fridge

[0160] Output: Image data of ingredients

[0161] The images are high resolution and the cameras are positioned to cover the entire interior of the refrigerator. Once the images are captured, the image data is used in the next step.

[0162] Step 2: Sending image data

[0163] The device sends the captured image data to a cloud server via Wi-Fi using a secure protocol (e.g., HTTPS).

[0164] Input: Image data of the photographed food

[0165] Output: Image data sent to the server

[0166] This allows the image data to reach the server for analysis.

[0167] Step 3: Analyzing the image data

[0168] The server analyzes the received image data using a deep learning module (e.g., TensorFlow) to identify the type, quantity, condition, and expiration date of the ingredients.

[0169] Input: Image data of the ingredients sent

[0170] Output: Type, quantity, condition and expiration date of recognized ingredients

[0171] The analysis results include details such as ingredient name, quantity, expiration date, and date and time of addition, which are stored in a database.

[0172] Step 4: Save the ingredients data

[0173] The server stores the analysis results in a database (e.g., MySQL or MongoDB).

[0174] Input: Detailed information about the recognized ingredients

[0175] Output: Saved ingredient data

[0176] Based on the stored data, the ingredients are managed in the next step.

[0177] Step 5: Enter your health status

[0178] Users enter their current health status using a smartphone app, which provides text fields and pull-down menus for input.

[0179] Input: User's health status information (e.g., feeling a bit under the weather, dieting, high stress)

[0180] Output: Health status data sent to the server

[0181] The submitted data is added to the user's profile database.

[0182] Step 6: Update your profile

[0183] The server then updates the user's profile database based on the received health status data, which includes past health status data and food consumption history.

[0184] Input: Health status data sent by the user

[0185] Output: Updated profile data

[0186] This lays the foundation for generating optimal recipes based on the user's health condition.

[0187] Step 7: Generative AI generates recipes

[0188] The server retrieves the stored ingredient data and the user's health status data and generates optimal recipes using generative AI models, including OpenAI's GPT-3 and other custom models.

[0189] Input: Stored food and health data

[0190] Output: The generated recipe

[0191] The generative AI uses this data to create recipes, taking into account the patient's health status and available ingredients. For example, it might recommend porridge if you're feeling unwell, or a low-calorie salad if you're on a diet.

[0192] Step 8: Submit your recipe

[0193] The server then sends the generated recipe to the user's device, which includes the ingredients, cooking instructions, and nutritional information.

[0194] Input: Generated recipe

[0195] Output: Recipe sent to user's device

[0196] This allows the user to receive and check the recipe.

[0197] Step 9: View the recipe

[0198] The user checks the suggested recipes on the smartphone app, which displays something like, "Today's recommended dish is cabbage and tomato salad."

[0199] Input: Recipe sent from the server

[0200] Output: Recipe displayed on smartphone app

[0201] This allows the user to easily check the suggested menu.

[0202] Step 10: Check for missing ingredients

[0203] The server checks the refrigerator for missing ingredients based on the proposed recipe.

[0204] Input: Proposed recipe and current ingredient data

[0205] Output: List of missing ingredients

[0206] If any ingredients are found to be missing, an automatic order will be placed in the next step.

[0207] Step 11: Automated Orders

[0208] The server automatically generates and sends an order to a food delivery service for any missing ingredients, along with credit card information and a delivery address.

[0209] Input: Missing ingredients list

[0210] Output: Order data sent to grocery delivery service

[0211] This will automatically replenish any missing ingredients.

[0212] Step 12: Delivery Information Notification

[0213] The server receives delivery information from the food delivery service and notifies the user terminal of the delivery information, including the scheduled delivery date and time and the delivery status.

[0214] Input: Delivery information from food delivery service

[0215] Output: Delivery information sent to the user's device

[0216] The user checks the status of their grocery delivery on a smartphone app, which displays a notification such as "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[0217] (Application example 1)

[0218] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0219] Efficient and accurate food management is essential to simultaneously reduce food waste at home and maintain good health. However, current systems make food management cumbersome, and they rarely suggest meals that are particularly suited to a person's health or automatically order ingredients that are in short supply. This leads to food waste and makes it difficult for users to maintain their health. Therefore, a system that effectively supports reducing food waste and maintaining good health is needed.

[0220] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0221] In this invention, the server includes means for taking images of ingredients with a camera built into the refrigerator, means for transmitting image data of ingredients to the server, means for analyzing the image data in the server and recognizing the type, amount, condition, and expiration date of ingredients, means for the user to input health status, means for generating recipes based on the health status and ingredient data and transmitting them to the user terminal, means for displaying the generated recipe, means for automatically ordering missing ingredients, means for notifying the user terminal of the delivery status of the automatically ordered ingredients, and means for using generation AI to generate optimal menus based on the health status and tracked ingredient information. This effectively reduces food waste and maintains health, making it easy for users to eat healthy meals.

[0222] A "camera built into a refrigerator" is a camera built into the refrigerator that takes pictures of food stored inside the refrigerator.

[0223] The "means for transmitting image data of ingredients to the server" refers to a communication device and protocol for transmitting image data captured by a camera built into the refrigerator to the server via the Internet.

[0224] "Means for analyzing image data and recognizing the type, quantity, condition and expiration date of ingredients" refers to a program and process that analyzes image data sent to the server and automatically determines the type, quantity, condition and expiration date of ingredients.

[0225] "Means for users to input their health status" refers to the interface and data input device that allows users to input and register their own health status.

[0226] The "means for generating a recipe and transmitting it to a user terminal" refers to a system and program for generating an appropriate recipe based on health status and ingredient data and transmitting the recipe to the user's terminal.

[0227] The "means for displaying the generated recipe" refers to a display and application for displaying the generated recipe on the user terminal.

[0228] The "means for automatically ordering missing ingredients" refers to a system and program that automatically orders missing ingredients from a food delivery service when ingredients required for a proposed recipe are missing.

[0229] The "means for notifying the user terminal of the delivery status of ordered ingredients" refers to a system and program for automatically notifying the user terminal of the delivery status of ordered ingredients.

[0230] "Generative AI" refers to models and algorithms for generating recipes using artificial intelligence based on specific tasks.

[0231] The "means for generating an optimal menu" refers to a system and program for generating an optimal menu for a user based on the user's health condition and tracked ingredient information.

[0232] This invention is a system used in a home refrigerator that simultaneously reduces food waste and maintains health. This system uses a program and a cloud server to manage ingredients, create recipes based on health information, and automatically order ingredients when they are in short supply. Specifically, the system is configured as follows:

[0233] System Configuration

[0234] 1. In-fridge camera:

[0235] A camera placed inside the refrigerator takes pictures of the ingredients, either a USB camera or a built-in camera.

[0236] 2. Server:

[0237] It functions as part of a cloud server that receives and analyzes image data of ingredients. The server is equipped with deep learning models (e.g., TensorFlow or PyTorch) that are used to recognize the type, quantity, condition, and expiration date of ingredients.

[0238] 3. User Device:

[0239] It consists of a smartphone or tablet that allows users to input their health status and receive and display recipes. The user interface includes a form for accepting health status input and a function for displaying generated recipes. Specifically, an iOS or Android application is used.

[0240] 4. Generation AI:

[0241] It is an artificial intelligence that generates appropriate recipes based on health and ingredient data, for example, GPT-4 and other generative models are used to provide recipes that suit the user's needs.

[0242] 5. Food delivery service:

[0243] This service has an automatic ordering function to make up for missing ingredients, and uses an API to generate orders and track the delivery status of ingredients.

[0244] Operation overview

[0245] 1. Register ingredients and send data:

[0246] The camera inside the refrigerator takes a picture of the new food item and sends the image to the server. The server receives the image data and analyzes it using a deep learning module. The analysis results include the type, quantity, condition, and expiration date of the food item, and these are stored in a database.

[0247] 2. User Health Management:

[0248] Users use a smartphone app to input their current health status, such as whether they are feeling unwell or on a diet. The server updates the profile database based on the received health status data.

[0249] 3. Recipe generation and distribution:

[0250] The server uses the AI ​​to generate optimal recipes based on the stored ingredient data and the user's health status data. The generated recipes are sent to the user's device, and the suggested recipes are displayed on the smartphone app.

[0251] 4. Automatic ordering of missing ingredients:

[0252] The server checks the refrigerator for missing ingredients based on the proposed recipe, automatically generates and sends an order to the food delivery service for the missing ingredients, and the user can check the status of the food delivery on their smartphone app.

[0253] Specific examples

[0254] What to do when you are unwell

[0255] 1. The user types "I feel like I have a cold" into a smartphone app.

[0256] 2. The server updates the health data and generates a porridge recipe suitable for the user's physical condition.

[0257] 3. The generated recipe is sent to the user's device and notified to the user.

[0258] 4. The user will see a message on their smartphone saying, "Let's try making porridge today."

[0259] Prompt Sentence Examples

[0260] The prompt is:

[0261] "The user's current health condition is as follows: poor health, desires easy-to-digest diet. The ingredients in the refrigerator are: tomatoes, cabbage, chicken. Please generate the optimal recipe based on these conditions."

[0262] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0263] Step 1:

[0264] A camera inside the refrigerator takes a picture of the newly added ingredient.

[0265] Input: Ingredients in the refrigerator

[0266] Output: Image data of ingredients

[0267] Specific operation: When the user puts ingredients into the refrigerator, the camera captures the moment and saves it as image data.

[0268] Step 2:

[0269] The server receives image data of the ingredients and analyzes the images to recognize the type, amount, condition, and expiration date of the ingredients.

[0270] Input: Image data of ingredients

[0271] Output: Analyzed food data (type, quantity, condition, expiration date)

[0272] How it works: Image data is sent over the internet to a server, which then analyzes the image using a deep learning model (e.g., TensorFlow). The analysis results are stored in a database.

[0273] Step 3:

[0274] Users input their own health status using a smartphone app.

[0275] Input: User's health status data (e.g., cold, desire for easy-to-digest meals, etc.)

[0276] Output: Updated health profile data

[0277] Specific operation: The user enters information such as "feeling a bit sick" or "on a diet" into the app. The entered data is sent to the server, and the profile database is updated.

[0278] Step 4:

[0279] Based on the ingredient data and health status data stored on the server, the generative AI generates the optimal recipe.

[0280] Input: Analyzed food ingredient data, health profile data

[0281] Output: The generated recipe

[0282] How it works: Generative AI (e.g., GPT-4) generates optimal recipes based on input data. For example, if you are feeling unwell, it will generate a recipe for easy-to-digest porridge.

[0283] Step 5:

[0284] The server sends the generated recipe to the user's device and displays it within the app.

[0285] Input: Generated recipe

[0286] Output: The recipe displayed on the user's terminal

[0287] Specific operation: The generated recipe is sent to the user's smartphone app and displayed as a notification within the app, such as "Today's recommended dish is porridge."

[0288] Step 6:

[0289] Based on the proposed recipe, the server checks the refrigerator for any missing ingredients and automatically generates and sends an order to the food delivery service.

[0290] Input: Generated recipe, analyzed food data in the refrigerator

[0291] Output: Order request to food delivery service

[0292] Specific operation: Checks the ingredients required for the generated recipe, and if there are any missing from the refrigerator, automatically sends an order request to the food delivery service. The order is generated using an API.

[0293] Step 7:

[0294] A user checks the status of their grocery delivery on a smartphone app.

[0295] Input: Delivery status data from food delivery service

[0296] Output: Delivery status displayed on the smartphone app (e.g., "Cabbage and tomatoes are scheduled for delivery tomorrow")

[0297] Specific operation: The server notifies the user's device of the delivery status obtained from the food delivery service and displays the status within the app.

[0298] By going through each step in this way, a system will be built that effectively reduces food waste and maintains health.

[0299] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0300] The present invention relates to a system that reduces food waste and proposes recipes that take into consideration the health of the household and the emotions of the user, using a camera built into a refrigerator, a server, a generative AI, a user terminal, an emotion engine, and a food delivery service. This system is specifically implemented as follows.

[0301] System Configuration

[0302] In-fridge camera: The camera is placed inside the refrigerator and is used to take pictures of the ingredients added.

[0303] User device: Consists of a smartphone, tablet, etc., and is used by users to input their health status and emotions, and receive and display recipes.

[0304] Server: A cloud system that receives and analyzes image data. This is where food ingredient data is generated.

[0305] Generative AI: An artificial intelligence system that generates appropriate recipes based on health status and ingredient data, as well as emotional data recognized by an emotion engine.

[0306] Emotion engine: A system that recognizes user emotions from user input data and other data sources and provides that information to a server.

[0307] Food delivery service: A service with an automatic ordering function to make up for missing ingredients.

[0308] Operation overview

[0309] Register ingredients and send data

[0310] Device: When a user places a new ingredient in the refrigerator, a camera takes a picture of it, which undergoes initial analysis and is then sent to the server.

[0311] Server: Receives image data and performs image analysis using a deep learning module. This allows the type, quantity, condition, and expiration date of ingredients to be recognized. The analysis results are stored in a database.

[0312] User health management and emotion recognition

[0313] Users: Use a smartphone app to input their current health status and emotions, including stress, anxiety, and joy.

[0314] Server: Analyzes the user's emotions from the input data using the emotion engine. The analysis results are stored in the user's emotion database.

[0315] Recipe generation and distribution

[0316] Server: The generative AI generates optimal recipes based on the stored ingredient data, the user's health status, and emotional data. For example, if the user is feeling stressed, it will suggest recipes for relaxing herbal teas or easy-to-make sweets.

[0317] Server: Sends the generated recipe to the user's device.

[0318] User: Viewing recipe suggestions on a smartphone app, such as "Today's recommended dish is cabbage and tomato salad."

[0319] Automatic ordering of missing ingredients

[0320] Server: Based on the proposed recipe, check the amount of ingredients currently in the refrigerator and check for any missing ingredients.

[0321] Server: Creates an order list to make up for any missing ingredients and generates order data for the food delivery service API.

[0322] User: Checks the status of a grocery delivery on a smartphone app. For example, it might say, "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[0323] Specific examples

[0324] Dealing with poor health and emotional frustration

[0325] 1. User: The user enters "I feel like I have a cold" and "I'm stressed" into a smartphone app.

[0326] 2. Server: The server updates the health and emotional data and generates recipes for porridge and herbal tea for stress relief that are suited to the user's physical condition.

[0327] 3. Server: Sends the generated recipe to the user's device and notifies the user.

[0328] 4. User: The smartphone displays the message, "Let's make porridge and herbal tea today."

[0329] Utilizing ingredients that are close to their expiration date and making suggestions based on emotions

[0330] 1. Terminal: The refrigerator camera takes a picture of the newly added milk and sends the image to the server.

[0331] 2. Server: Analyzes the image data and identifies the type of milk and its expiration date.

[0332] 3. Server: Updates the ingredient database and determines when the expiration date is approaching.

[0333] 4. Server: Based on the user's emotional data, it generates recipes for milk-based dishes (e.g., cream stew) and soothing desserts.

[0334] 5. Server: Sends the recipe to the user device.

[0335] 6. User: "Today, let's make cream stew and chocolate mousse" appears on the smartphone.

[0336] The present invention makes it possible to reduce food waste and effectively support users in maintaining their health and managing their emotions.

[0337] The processing flow will be explained below.

[0338] Specific processing of the program

[0339] Ingredient recognition and data transmission

[0340] Step 1:

[0341] Device: The camera built into the refrigerator takes pictures of the newly added ingredients by the user.

[0342] Step 2:

[0343] Terminal: Performs initial analysis of images captured by the camera and extracts features such as the shape, color, and tags of ingredients.

[0344] Step 3:

[0345] Terminal: Sends image data along with initial analysis data to a server via the internet.

[0346] Analysis of food ingredient data

[0347] Step 4:

[0348] Server: The received image data is input into a deep learning module to analyze the detailed type, quantity, and condition of the ingredients.

[0349] Step 5:

[0350] Server: Reads the expiration date on food packaging using OCR (optical character recognition) technology.

[0351] Step 6:

[0352] Server: Stores the analysis results in a food ingredient database. Stored data includes type, quantity, expiration date, and condition.

[0353] User health management and emotion recognition

[0354] Step 7:

[0355] User: Launches the smartphone app and inputs their current health condition and emotions. For example, they input that their physical condition is "feeling a bit like a cold" and their emotion is "stressed."

[0356] Step 8:

[0357] Server: Updates the received health status data in the profile database.

[0358] Step 9:

[0359] Server: Analyzes the user's emotions from the input data using the emotion engine. The analysis results are stored in the user's emotion database.

[0360] Recipe generation and distribution

[0361] Step 10:

[0362] Server: The generative AI generates optimal recipes based on the stored ingredient data, the user's health status, and emotional data. For example, if a user is feeling unwell and stressed, it will suggest a recipe for porridge and herbal tea that helps relieve stress.

[0363] Step 11:

[0364] Server: Sends the generated recipe to the user's device.

[0365] Step 12:

[0366] User: A smartphone app displays recipe suggestions, such as "Today, try making porridge and herbal tea."

[0367] Automatic ordering of missing ingredients

[0368] Step 13:

[0369] Server: Based on the proposed recipe, check the amount of ingredients currently in the refrigerator and check for any missing ingredients.

[0370] Step 14:

[0371] Server: Creates an order list to make up for any missing ingredients and generates order data for the food delivery service API.

[0372] Step 15:

[0373] User: Checks the status of a grocery delivery on a smartphone app. For example, it might say, "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[0374] Specific examples

[0375] Dealing with poor health and emotional frustration

[0376] Step 1:

[0377] User: The user types "feeling a bit under the weather" and "feeling stressed" into a smartphone app.

[0378] Step 2:

[0379] Server: The server updates the health and emotional data of the user and generates recipes for porridge and herbal tea for stress relief that are suited to the user's physical condition.

[0380] Step 3:

[0381] Server: Sends the generated recipe to the user's device and notifies the user.

[0382] Step 4:

[0383] User: "Let's make porridge and herbal tea today" appears on their phone.

[0384] Utilizing ingredients that are close to their expiration date and making suggestions based on emotions

[0385] Step 1:

[0386] Device: The camera in the refrigerator takes a picture of the newly added milk and sends the image to the server.

[0387] Step 2:

[0388] Server: Analyzes image data and identifies the type of milk and its expiration date.

[0389] Step 3:

[0390] Server: Updates the food ingredient database and determines when the expiration date is approaching.

[0391] Step 4:

[0392] User: The user types "I'm tired" into a smartphone app.

[0393] Step 5:

[0394] Server: Generates recipes for milk-based dishes (e.g., cream stew) and soothing desserts (e.g., chocolate mousse) based on ingredient data and emotion data.

[0395] Step 6:

[0396] Server: Sends recipes to user devices.

[0397] Step 7:

[0398] User: "Today, let's make cream stew and chocolate mousse" appears on their phone.

[0399] Example 2

[0400] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0401] In modern households, it is difficult to manage food ingredients, and many ingredients pass their expiration date. Health management is also difficult because meal suggestions do not take into account the user's health and emotional state. As a result, food waste and unhealthy eating habits have become problems.

[0402] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0403] In this invention, the server includes means for acquiring images of ingredients using an image acquisition device built into the refrigerator, means for transmitting image data of the ingredients to the server, means for analyzing the image data in the server and recognizing the type, amount, condition, and expiration date of the ingredients, means for the user to input their health condition and emotions, means for generating recipes using a generative AI model based on the health condition, emotion data, and ingredient data and transmitting the recipes to the user terminal, and means for displaying the generated recipes. This makes it possible to reduce food waste and suggest meals that take into consideration the user's health management and emotions.

[0404] An "image capture device built into a refrigerator" is a combination of hardware and software that is installed inside the refrigerator and is used to capture images of ingredients and acquire the data.

[0405] A "server" is a computer system that analyzes the received image data, health condition data, and emotion data and performs the necessary processing.

[0406] The "means for transmitting image data" is a communication means for transmitting image data of ingredients captured by the image capturing device in the refrigerator to the server.

[0407] "Means for analyzing image data" refers to software and algorithms that analyze the image data received by the server using a deep learning module or the like to recognize the type, quantity, condition, and expiration date of ingredients.

[0408] The "means for inputting health status" refers to a user terminal equipped with an interface for the user to input their own health status.

[0409] The "means for inputting emotions" refers to a user terminal equipped with an interface for the user to input their own emotions.

[0410] "Means for generating recipes using a generative artificial intelligence model" refers to software and algorithms for generating optimal recipes using a generative AI model based on health data, emotional data, and ingredient data.

[0411] A "user terminal" is a communication terminal device such as a smartphone or tablet that allows a user to check the generated recipe.

[0412] The "means for displaying the generated recipe" refers to an interface and software for visually displaying the generated recipe on the user terminal.

[0413] "Means for generating and sending orders to food delivery services" refers to software and communication means for automatically generating and sending orders using the food delivery service's API when necessary ingredients are in short supply.

[0414] "Initial processing means" refers to software and algorithms that extract features from the image data captured by the camera and perform preliminary analysis on the data before sending it to the server.

[0415] This invention is a system for reducing food waste, maintaining the user's health, and suggesting recipes based on emotions. This system is realized using an image capture device built into a refrigerator, a server, a generative AI model, a user terminal, an emotion engine, and a food delivery service. The operation of the system is as follows.

[0416] System Configuration

[0417] Refrigerator image capture device: This is a device that is installed inside the refrigerator to capture images of added ingredients.

[0418] User device: This consists of a smartphone or tablet, and is used by the user to input their health status and emotions, and to receive and display the generated recipes. The user device contains the interface and software.

[0419] Server: This is a computer system that receives and analyzes image data. The server uses deep learning modules (e.g., TensorFlow or PyTorch), databases (e.g., MySQL or PostgreSQL), and emotion engines (e.g., Emotion API).

[0420] Generative AI model: This is an artificial intelligence model that generates optimal recipes based on health status, emotional data, and ingredient data. For example, GPT-4 is used as a generative AI model.

[0421] Emotion engine: This is a system that recognizes the user's emotions and provides that information to the server. Specifically, it uses the Emotion API.

[0422] Food delivery service: A service that automatically generates order data when necessary ingredients are in short supply. For example, it uses APIs such as Amazon Fresh and Instacart.

[0423] Operation overview

[0424] This system operates in the following manner.

[0425] 1. Register ingredients and send data

[0426] When a user places new ingredients in the refrigerator, an image capture device inside the refrigerator takes a picture. The image data is sent to a server, which analyzes it using a deep learning module. This identifies the type, quantity, condition, and expiration date of the ingredients. The analysis results are then stored in a database.

[0427] 2. User Health Management and Emotion Recognition

[0428] The user inputs their current health status and emotions using a smartphone app. The server analyzes this data using an emotion engine and stores the results in an emotion database.

[0429] 3. Recipe generation and distribution

[0430] The server uses a generative AI model based on the stored ingredient data, the user's health status data, and emotional data to generate the optimal recipe. For example, the prompt text could read, "The user is feeling a bit under the weather and under a lot of stress. The ingredients available are cabbage, tomato, and chicken. Please suggest a dish that will have a relaxing effect using these ingredients."

[0431] 4. Automatic ordering of missing ingredients

[0432] Based on the proposed recipe, the system checks the amount of ingredients currently in the refrigerator and checks for any missing ingredients. If any ingredients are missing, it creates an order list and automatically sends the order data to the food delivery service's API.

[0433] Specific examples

[0434] 1. Dealing with poor health and emotional frustration

[0435] A user enters "feeling a bit like a cold" and "feeling stressed" into a smartphone app and submits the message.

[0436] Based on the data received by the server, a generative AI model is used to generate recipes for porridge and stress-relieving herbal tea.

[0437] The generated recipe is sent to the user terminal and notified to the user.

[0438] The user sees a message on their smartphone saying, "Let's make porridge and herbal tea today."

[0439] 2. Utilizing ingredients that are close to their expiration date and making suggestions based on emotions

[0440] The image capture device in the refrigerator takes a picture of the newly added milk and sends the image to the server.

[0441] The server analyzes the image data and identifies the type of milk and its expiration date.

[0442] The server updates the food ingredient database and determines when the expiration date is approaching.

[0443] The server generates recipes for cream stew using milk and a soothing dessert, for example, based on the user's emotional data.

[0444] The generated recipe is sent to the user terminal.

[0445] The user sees on their smartphone a message saying, "Today, let's make cream stew and chocolate mousse."

[0446] The present invention makes it possible to reduce food waste, maintain the user's health, and suggest meals that suit the user's emotions.

[0447] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0448] Step 1:

[0449] Food Registration

[0450] Terminal: When a user adds a new ingredient to the refrigerator, an image capture device in the refrigerator takes an image of the ingredient. The captured image data is sent from the terminal to the server.

[0451] Input: Image data of the newly added ingredient.

[0452] Output: Image data sent to the server.

[0453] Step 2:

[0454] Initial data analysis

[0455] Server: The received image data is analyzed using a deep learning module (e.g., TensorFlow or PyTorch). This analysis identifies the type, quantity, condition, and expiration date of the ingredients. The analysis results are stored in a database.

[0456] Input: Image data sent to the server.

[0457] Output: Data on the type, quantity, condition and expiry date of the recognized ingredients.

[0458] Step 3:

[0459] Health and Emotion Input

[0460] User: The user uses the smartphone app to input their own health condition and emotions. They select a condition such as "feeling a bit like a cold" or "feeling stressed" from the options and press the send button.

[0461] Input: Health and emotional information entered by the user into the app.

[0462] Output: Health and emotion data sent to the server.

[0463] Step 4:

[0464] Emotional Data Analysis

[0465] Server: Analyzes the received health status data and emotion data using an emotion engine (e.g., Emotion API). The analysis results are stored in an emotion database.

[0466] Input: User-entered health and emotion data.

[0467] Output: Stored sentiment analysis results.

[0468] Step 5:

[0469] Recipe Generation

[0470] Server: Generates optimal recipes using a generative AI model (such as GPT-4) based on the stored ingredient data, health status data, and emotion data. The prompt text is entered as follows: "The user is feeling a bit under the weather and is under a lot of stress. The ingredients available are cabbage, tomato, and chicken. Please suggest a dish that will have a relaxing effect using these ingredients."

[0471] Input: Food ingredient data, health condition data, emotion data, prompt sentence.

[0472] Output: The generated recipe data.

[0473] Step 6:

[0474] Recipe distribution

[0475] Server: Sends the generated recipe to the user's device.

[0476] Input: The generated recipe data.

[0477] Output: The recipe sent to the user's device.

[0478] Step 7:

[0479] Recipe display

[0480] User: The user checks the suggested recipes on the smartphone app. When the user opens the app, the suggested recipes are displayed.

[0481] Input: Recipe data sent from the server.

[0482] Output: The recipe displayed in the app.

[0483] Step 8:

[0484] Ingredients checking and automatic ordering

[0485] Server: Based on the proposed recipe, check the database for the amount of ingredients currently in the refrigerator. Check the required amount to see if any ingredients are missing.

[0486] Input: Current ingredient data and suggested recipe data.

[0487] Output: A list of missing ingredients.

[0488] Step 9:

[0489] Automatic Order Generation

[0490] Server: Based on the list of missing ingredients, order data is automatically generated and sent using the food delivery service's API.

[0491] Input: Missing ingredient list.

[0492] Output: Order data for food delivery service.

[0493] Step 10:

[0494] Delivery status notifications

[0495] Server: After the order is completed, check the delivery status and send the delivery information to the user's device.

[0496] Input: Delivery status information from a grocery delivery service.

[0497] Output: Delivery status notification sent to user device.

[0498] Step 11:

[0499] Delivery status display

[0500] User: The user checks the delivery status on their smartphone app, for example, "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[0501] Input: Delivery status notification.

[0502] Output: Delivery status information displayed in the app.

[0503] This will make it possible to reduce food waste, maintain health, and suggest recipes that respond to emotions.

[0504] (Application example 2)

[0505] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0506] Managing meals and maintaining one's health is important for passengers to stay comfortable in self-driving vehicles during long-distance travel. However, it is difficult to store appropriate ingredients, supply new ingredients, and suggest meals that take into account the passenger's health and emotions while traveling. Therefore, there is a need to provide a system that reduces food waste and makes meal suggestions that take into account the passenger's health and emotions while traveling in self-driving vehicles during long-distance travel.

[0507] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0508] In this invention, the server includes a means for capturing images of ingredients using an imaging device built into the refrigerator, a means for transmitting the ingredient information data to the server, a means for analyzing the information data in the server and recognizing the type, quantity, condition, and expiration date of the ingredients, a means for the user to input their health and emotional state, a means for generating suggestions based on the health and ingredient data and the emotional state data and transmitting the suggestions to the user terminal, and a means for displaying the generated suggestions. This automates meal management in an autonomous vehicle and enables meal suggestions based on the passenger's health and emotional state. Furthermore, if ingredients run low, new ingredients can be automatically replenished using a delivery service, allowing for comfortable meals to be served even while on the move.

[0509] An "imaging device" is a device that captures an image of a subject.

[0510] A "server" is a computer system that sends, receives, and processes data over a network.

[0511] "Information data" refers to data including images of ingredients and the user's health and emotional state.

[0512] "Analysis" is the process of breaking down input data and classifying, organizing, and evaluating it.

[0513] "Type of food ingredient" is information indicating the name and classification of food.

[0514] "Quantity" is information indicating the quantity or volume of ingredients.

[0515] "Status" is information indicating the freshness and storage state of the ingredients.

[0516] The "use by" date is information indicating the period during which the food ingredient can be safely used.

[0517] A "user terminal" is a device used by a user, such as a smartphone or tablet.

[0518] "Health status" is information relating to the user's physical condition and health.

[0519] "Emotional state" is information about the user's emotional and mental state.

[0520] "Suggestions" are recipes and meal recommendations generated based on the analysis results.

[0521] A "display means" is a method or device that visually shows the generated suggestions to the user.

[0522] "Delivery service" refers to a service that delivers food ingredients ordered online to a specified address.

[0523] The present invention provides a system for optimizing the dietary management and health status of passengers during long-distance travel in an autonomous vehicle and for suggesting recipes that take into consideration their emotions. This system is specifically implemented as follows.

[0524] System Configuration

[0525] 1. Imaging device (camera)

[0526] The system uses an imaging device built into the refrigerator inside the autonomous vehicle, which periodically captures images of the food items in the refrigerator and acquires the image data.

[0527] 2. Server

[0528] The image data is sent to a server where it is analyzed and the server uses deep learning techniques (e.g., TensorFlow, PyTorch) to identify the type, quantity, condition, and expiration date of the ingredients.

[0529] 3. User Device

[0530] Passengers use a user device such as a smartphone or tablet to access an application that inputs their health and emotional status. The application was developed using React Native.

[0531] 4. Emotion Engine

[0532] The health and emotional state data entered by the user into the device is analyzed by an emotion engine running on a cloud server (e.g., Microsoft Azure Cognitive Services).

[0533] 5. Generation AI

[0534] The generative AI (e.g., OpenAI GPT-4) on the server generates appropriate recipes based on the stored ingredient data and the user's health and emotional state.

[0535] 6. Display means

[0536] The generated recipe is sent to the user's smartphone or tablet in real time and displayed.

[0537] 7. Delivery Services

[0538] A delivery service will be used to automatically order and replenish missing ingredients at stops where the autonomous vehicle stops.

[0539] Operation overview

[0540] Program processing

[0541] The server receives the image data sent from the imaging device and analyzes the data using deep learning. As a result of the analysis, the type, quantity, condition, and expiration date of the ingredients are confirmed and stored in a database. The user inputs their health and emotional state using their user terminal. This input data is analyzed by the emotion engine and stored in the user's emotion database. The generation AI generates an optimal recipe based on the stored ingredient data and the user's health and emotional state, and sends it to the user terminal. The recipe is displayed in real time on the user terminal as a display method. If necessary ingredients are in short supply, an order is automatically issued to a delivery service, and they are replenished at the stop.

[0542] Specific examples

[0543] 1. User Input

[0544] A user types into a smartphone app, "I'm tired and would like a simple meal."

[0545] 2. Server Analysis

[0546] The server updates the health and emotion data and generates a recipe for "chicken and vegetable soup" that suits the user's physical condition.

[0547] 3. Sending and viewing recipes

[0548] The generated recipe is sent to the user terminal and displayed.

[0549] Generative AI model and prompts

[0550] Here's an example prompt that uses a generative AI model to generate a recipe:

[0551] User's current health status: Fatigue

[0552] User's current emotion: Wanting to relax

[0553] Usable ingredients: cabbage, carrots, chicken

[0554] Suggest a recipe to make:

[0555] This will enable passengers to enjoy comfortable meals and maintain their health while inside self-driving vehicles.

[0556] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0557] Step 1:

[0558] Photographing ingredients using an imaging device

[0559] The image capture device periodically captures images of food items in the refrigerator. The input is an image of the food items in the refrigerator, and the output is the captured image data. This image data is necessary to analyze the type, quantity, condition, and expiration date of the food items.

[0560] Step 2:

[0561] Sending image data to the server

[0562] Image data captured by an imaging device is sent to a server. The input is the captured image data, and the output is the image data uploaded to the server. This transmission prepares the server for subsequent analysis.

[0563] Step 3:

[0564] Image data analysis

[0565] The server analyzes the transmitted image data using deep learning technology (e.g., TensorFlow, PyTorch). The input is the image data uploaded to the server, and the output is the analysis results showing the type, quantity, condition, and expiration date of the ingredients. This analysis allows specific information about the ingredients to be registered in a database.

[0566] Step 4:

[0567] Input of the user's health and emotional state

[0568] Users use a smartphone or tablet to input their health and emotional status into the application. The input is information about the user's health and emotional status, and the output is the transmission of the input information. This allows data to be collected on the server for analysis by the emotion engine.

[0569] Step 5:

[0570] Analysis by emotion engine

[0571] The server uses an emotion engine (e.g., Microsoft Azure Cognitive Services) to analyze the user's health and emotional state. The input is the user's health and emotional state information, and the output is the analyzed emotion data. This data is stored in the user's emotion database.

[0572] Step 6:

[0573] Recipe generation using generative AI

[0574] The server's generative AI (e.g., OpenAI GPT-4) generates optimal recipes based on the stored ingredient data and the user's health and emotional state. The input is the stored ingredient data and the user's health and emotional state information, and the output is the generated recipe. This process suggests appropriate meals based on the user's state.

[0575] Step 7:

[0576] Sending and displaying recipes to user devices

[0577] The generated recipe is sent from the server to the user's smartphone or tablet. The input is the generated recipe, and the output is the recipe display on the user's device. The user can then cook a dish based on this recipe.

[0578] Step 8:

[0579] Use of delivery services

[0580] If a required ingredient is missing, the server automatically orders the missing ingredient from the delivery service. The input is the current ingredient status and the generated recipe, and the output is the order submission and replenishment of ingredients. This allows necessary ingredients to be replenished in a timely manner even while on the move, enabling comfortable meals to be served.

[0581] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0582] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0583] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0584] [Second embodiment]

[0585] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0586] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0587] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0588] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0589] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0590] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0591] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0592] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0593] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0594] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0595] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0596] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0597] The present invention relates to a system that reduces food waste and supports household health maintenance by using a camera built into a refrigerator, a server, a generation AI, a user terminal, and a food delivery service. This system is specifically implemented as follows.

[0598] System Configuration

[0599] In-fridge camera: The camera is placed inside the refrigerator and is used to take pictures of the ingredients added.

[0600] User device: Consists of a smartphone, tablet, etc., and is used by users to input their health status and receive and display recipes.

[0601] Server: A cloud system that receives and analyzes image data. This is where food ingredient data is generated.

[0602] Generative AI: Artificial intelligence that generates appropriate recipes based on health and ingredient data.

[0603] Food delivery service: A service with an automatic ordering function to make up for missing ingredients.

[0604] Operation overview

[0605] Register ingredients and send data

[0606] Device: When a user places a new ingredient in the refrigerator, a camera takes a picture of it, which undergoes initial analysis and is then sent to the server.

[0607] Server: Receives image data and performs image analysis using a deep learning module. This allows the type, quantity, condition, and expiration date of ingredients to be recognized. The analysis results are stored in a database.

[0608] User health management

[0609] User: Using a smartphone app, the user inputs their current health status, such as whether they are feeling unwell or on a diet.

[0610] Server: Updates the profile database based on the received health status data.

[0611] Recipe generation and distribution

[0612] Server: The AI ​​generates optimal recipes based on the stored ingredient data and the user's health status. If the user is not feeling well, it will generate recipes that are easy to digest, and under normal circumstances, it will generate recipes that prioritize ingredients that are close to their expiration date.

[0613] Server: Sends the generated recipe to the user's device.

[0614] User: A smartphone app displays recipe suggestions, such as "Today's recommended dish is cabbage and tomato salad."

[0615] Automatic ordering of missing ingredients

[0616] Server: Based on the proposed recipe, check the refrigerator for missing ingredients.

[0617] Server: Automatically generate and send orders to food delivery services to make up for missing ingredients.

[0618] User: Checks the status of a grocery delivery on a smartphone app, for example, with a notification that "cabbage and tomatoes are scheduled to be delivered tomorrow."

[0619] Specific examples

[0620] What to do when you are unwell

[0621] 1. User: The user types "I feel like I have a cold" into a smartphone app.

[0622] 2. Server: The server updates the health data and generates a porridge recipe suitable for the user's physical condition.

[0623] 3. Server: Sends the generated recipe to the user's device and notifies the user.

[0624] 4. User: The smartphone displays the message, "Let's make porridge today."

[0625] Utilizing ingredients that are close to their expiration date

[0626] 1. Terminal: The refrigerator camera takes a picture of the newly added milk and sends the image to the server.

[0627] 2. Server: Analyzes the image data and identifies the type of milk and its expiration date.

[0628] 3. Server: Updates the ingredient database and determines when the expiration date is approaching.

[0629] 4. Server: Generates recipes for dishes using milk (e.g., cream stew) and sends them to the user's device.

[0630] 5. User: The smartphone displays the message, "Let's make cream stew today."

[0631] The present invention makes it possible to reduce food waste and effectively support users in maintaining their health.

[0632] The processing flow will be explained below.

[0633] Specific processing of the program

[0634] Ingredient recognition and data transmission

[0635] Step 1:

[0636] Device: The camera built into the refrigerator takes pictures of the newly added ingredients by the user.

[0637] Step 2:

[0638] Terminal: Performs initial analysis of images captured by the camera and extracts features such as the shape, color, and tags of ingredients.

[0639] Step 3:

[0640] Terminal: Sends image data along with initial analysis data to a server via the internet.

[0641] Analysis of food ingredient data

[0642] Step 4:

[0643] Server: The received image data is input into a deep learning module to analyze the detailed type, quantity, and condition of the ingredients.

[0644] Step 5:

[0645] Server: Reads the expiration date printed on the food packaging using OCR (optical character recognition) technology.

[0646] Step 6:

[0647] Server: Stores the analysis results in a food ingredient database. Stored data includes type, quantity, expiration date, and condition.

[0648] User Health Check

[0649] Step 7:

[0650] User: Launches the smartphone app and enters their current health and physical condition.

[0651] Step 8:

[0652] Server: Updates the received health status data in the user profile database.

[0653] Recipe generation and distribution

[0654] Step 9:

[0655] Server: Generative AI generates optimal recipes based on ingredient data and health status data.

[0656] Step 10:

[0657] Server: Sends the generated recipe to the user's device.

[0658] Step 11:

[0659] User: A smartphone app displays a suggested recipe, for example, "Today's recommended dish is cabbage and tomato salad."

[0660] Automatic ordering of missing ingredients

[0661] Step 12:

[0662] Server: Based on the proposed recipe, check the amount of ingredients currently in the refrigerator and check for any missing ingredients.

[0663] Step 13:

[0664] Server: Creates an order list to make up for any missing ingredients and generates order data for the food delivery service API.

[0665] Step 14:

[0666] User: Checks the status of grocery delivery on a smartphone app. For example, it shows "cabbage and tomatoes are scheduled to be delivered tomorrow."

[0667] Example 1

[0668] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0669] There is a need for efficient methods to reduce food waste in the home and provide healthy meals. However, current refrigerator management systems make it difficult to accurately grasp the status of ingredients and tend to waste ingredients that are approaching their expiration date. Furthermore, they do not suggest optimal recipes based on health status, which makes it difficult to adequately manage health at home.

[0670] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0671] In this invention, the server

[0672] A means for taking an image of the food using a camera built into the refrigerator;

[0673] means for transmitting image data of ingredients to a server;

[0674] A means for analyzing image data in the server and recognizing the type, amount, condition and expiration date of ingredients;

[0675] a means for a user to input a health status;

[0676] A means for generating a recipe by the artificial intelligence based on the health status and the ingredient data and transmitting the recipe to the user terminal;

[0677] A display means for the generated recipe;

[0678] Includes.

[0679] This will automate food ingredient management and suggest optimal recipes based on health status, reducing food waste and helping to maintain household health.

[0680] A "camera built into a refrigerator" is a device that is installed inside the refrigerator and takes pictures of ingredients.

[0681] "Food ingredients" is a general term for food stored in a refrigerator.

[0682] "Image data" is digital data that contains visual information of ingredients photographed by a camera.

[0683] The "server" is a computer system installed on the cloud that analyzes image data of ingredients and provides services based on the user's health condition.

[0684] "Analysis" is the process of recognizing the type, quantity, condition, and expiration date of ingredients based on image data.

[0685] A "deep learning module" is a type of artificial intelligence technology used to analyze images of ingredients, and utilizes a multi-layer neural network.

[0686] A "user terminal" is a device that allows a user to input their health status and receive notifications and recipes from the system, and includes smartphones, tablets, etc.

[0687] "Health condition data" is information about the current health condition entered by the user.

[0688] "Generative AI" is artificial intelligence that generates optimal recipes based on the user's health condition and ingredient data.

[0689] A "recipe" is a document that describes the steps for preparing a dish using specific ingredients.

[0690] A "food delivery service" is a commercial service that automatically orders and delivers food ingredients needed by users.

[0691] A "database" is a system for systematically storing analysis results and user health status data.

[0692] An "order" is request data for ordering the necessary ingredients from the ingredient delivery service.

[0693] The "profile database" is a database for storing a user's past health condition data and food consumption history.

[0694] This invention is a system for supporting household food waste reduction and health maintenance. The system's main components are a camera built into the refrigerator, a server, a generative AI model, a user terminal, and a food delivery service.

[0695] Refrigerator camera

[0696] The camera built into the refrigerator, which is the terminal, is a device that takes pictures of ingredients added to the refrigerator. A high-resolution camera is used and is positioned so that it covers the entire interior of the refrigerator. When a user puts new ingredients into the refrigerator, the camera automatically detects this and takes a picture. This picture is sent to a server on the cloud via Wi-Fi. A secure protocol (e.g., HTTPS) is used for transmission.

[0697] Server and Image Analysis

[0698] The server analyzes the received image data using a deep learning module. For example, it uses a deep learning library such as TensorFlow to identify the type, quantity, condition (e.g., fresh, spoiled), and expiration date of the ingredients. The results of this analysis are stored in a database (e.g., MySQL or MongoDB). The stored data includes details such as the ingredient name, quantity, expiration date, and the date and time of addition.

[0699] User health management

[0700] Using a smartphone app, users input their current health status (e.g., feeling a bit under the weather, dieting, high stress) using text fields and pull-down menus. This information is sent in real time to a server, which then updates the user's profile database. This profile also includes past health data and food consumption history.

[0701] Recipe generation and distribution using generative AI

[0702] The server retrieves the stored ingredient data and the user's health status data and generates optimal recipes using a generative AI model. Examples of generative AI models include OpenAI's GPT-3 and other custom models. The generative AI takes into account the user's health status and available ingredient information to create healthy recipes. For example, it might recommend porridge when feeling unwell or a low-calorie salad when on a diet. This generated recipe is sent to the user's device and displayed on a smartphone app.

[0703] Automatic ordering of missing ingredients

[0704] The server checks the refrigerator for missing ingredients based on the proposed recipe. If any ingredients are found to be missing, it automatically calls the food delivery service's API and generates an order. The order includes credit card information and delivery address. Delivery information from the food delivery service is sent to the user's device. The user can check the delivery status on their smartphone app.

[0705] Examples of concrete examples and prompts

[0706] As a concrete example, let's look at how to respond when you're feeling unwell. When a user enters "I'm feeling a bit sick" into a smartphone app, the server updates the health data and generates a porridge recipe suited to the user's physical condition. The generated recipe is sent to the user's device, and the smartphone displays a message saying, "Let's try making porridge today."

[0707] We will also give a concrete example of how to utilize ingredients that are approaching their expiration date. The camera in the refrigerator takes a picture of the newly added milk and sends the image to the server. The server analyzes the image data and recognizes the type of milk and its expiration date. It updates the ingredient database and determines that the expiration date is approaching. A recipe for a dish that uses the milk (for example, cream stew) is generated and sent to the user's device. The user is then prompted on their smartphone, "Let's try making cream stew today."

[0708] Examples of prompts:

[0709] "When you put new food in the refrigerator, the camera automatically takes a picture and sends it to the server."

[0710] "When a user inputs information about when they are feeling unwell into a smartphone app, an appropriate recipe is generated based on that information."

[0711] "A function that automatically orders missing ingredients from a delivery service based on the suggested recipe."

[0712] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0713] Step 1: Take a photo of the food

[0714] When a user adds new ingredients to the refrigerator, the device's built-in camera automatically takes a picture of the ingredients.

[0715] Input: New ingredients added to the fridge

[0716] Output: Image data of ingredients

[0717] The images are high resolution and the cameras are positioned to cover the entire interior of the refrigerator. Once the images are captured, the image data is used in the next step.

[0718] Step 2: Sending image data

[0719] The device sends the captured image data to a cloud server via Wi-Fi using a secure protocol (e.g., HTTPS).

[0720] Input: Image data of the photographed food

[0721] Output: Image data sent to the server

[0722] This allows the image data to reach the server for analysis.

[0723] Step 3: Analyzing the image data

[0724] The server analyzes the received image data using a deep learning module (e.g., TensorFlow) to identify the type, quantity, condition, and expiration date of the ingredients.

[0725] Input: Image data of the ingredients sent

[0726] Output: Type, quantity, condition and expiration date of recognized ingredients

[0727] The analysis results include details such as ingredient name, quantity, expiration date, and date and time of addition, which are stored in a database.

[0728] Step 4: Save the ingredients data

[0729] The server stores the analysis results in a database (e.g., MySQL or MongoDB).

[0730] Input: Detailed information about the recognized ingredients

[0731] Output: Saved ingredient data

[0732] Based on the stored data, the ingredients are managed in the next step.

[0733] Step 5: Enter your health status

[0734] Users enter their current health status using a smartphone app, which provides text fields and pull-down menus for input.

[0735] Input: User's health status information (e.g., feeling a bit under the weather, dieting, high stress)

[0736] Output: Health status data sent to the server

[0737] The submitted data is added to the user's profile database.

[0738] Step 6: Update your profile

[0739] The server then updates the user's profile database based on the received health status data, which includes past health status data and food consumption history.

[0740] Input: Health status data sent by the user

[0741] Output: Updated profile data

[0742] This lays the foundation for generating optimal recipes based on the user's health condition.

[0743] Step 7: Generative AI generates recipes

[0744] The server retrieves the stored ingredient data and the user's health status data and generates optimal recipes using generative AI models, including OpenAI's GPT-3 and other custom models.

[0745] Input: Stored food and health data

[0746] Output: The generated recipe

[0747] The generative AI uses this data to create recipes, taking into account the patient's health status and available ingredients. For example, it might recommend porridge if you're feeling unwell, or a low-calorie salad if you're on a diet.

[0748] Step 8: Submit your recipe

[0749] The server then sends the generated recipe to the user's device, which includes the ingredients, cooking instructions, and nutritional information.

[0750] Input: Generated recipe

[0751] Output: Recipe sent to user's device

[0752] This allows the user to receive and check the recipe.

[0753] Step 9: View the recipe

[0754] The user checks the suggested recipes on the smartphone app, which displays something like, "Today's recommended dish is cabbage and tomato salad."

[0755] Input: Recipe sent from the server

[0756] Output: Recipe displayed on smartphone app

[0757] This allows the user to easily check the suggested menu.

[0758] Step 10: Check for missing ingredients

[0759] The server checks the refrigerator for missing ingredients based on the proposed recipe.

[0760] Input: Proposed recipe and current ingredient data

[0761] Output: List of missing ingredients

[0762] If any ingredients are found to be missing, an automatic order will be placed in the next step.

[0763] Step 11: Automated Orders

[0764] The server automatically generates and sends an order to a food delivery service for any missing ingredients, along with credit card information and a delivery address.

[0765] Input: Missing ingredients list

[0766] Output: Order data sent to grocery delivery service

[0767] This will automatically replenish any missing ingredients.

[0768] Step 12: Delivery Information Notification

[0769] The server receives delivery information from the food delivery service and notifies the user terminal of the delivery information, including the scheduled delivery date and time and the delivery status.

[0770] Input: Delivery information from food delivery service

[0771] Output: Delivery information sent to the user's device

[0772] The user checks the status of their grocery delivery on a smartphone app, which displays a notification such as "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[0773] (Application example 1)

[0774] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0775] Efficient and accurate food management is essential to simultaneously reduce food waste at home and maintain good health. However, current systems make food management cumbersome, and they rarely suggest meals that are particularly suited to a person's health or automatically order ingredients that are in short supply. This leads to food waste and makes it difficult for users to maintain their health. Therefore, a system that effectively supports reducing food waste and maintaining good health is needed.

[0776] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0777] In this invention, the server includes means for taking images of ingredients with a camera built into the refrigerator, means for transmitting image data of ingredients to the server, means for analyzing the image data in the server and recognizing the type, amount, condition, and expiration date of ingredients, means for the user to input health status, means for generating recipes based on the health status and ingredient data and transmitting them to the user terminal, means for displaying the generated recipe, means for automatically ordering missing ingredients, means for notifying the user terminal of the delivery status of the automatically ordered ingredients, and means for using generation AI to generate optimal menus based on the health status and tracked ingredient information. This effectively reduces food waste and maintains health, making it easy for users to eat healthy meals.

[0778] A "camera built into a refrigerator" is a camera built into the refrigerator that takes pictures of food stored inside the refrigerator.

[0779] The "means for transmitting image data of ingredients to the server" refers to a communication device and protocol for transmitting image data captured by a camera built into the refrigerator to the server via the Internet.

[0780] "Means for analyzing image data and recognizing the type, quantity, condition and expiration date of ingredients" refers to a program and process that analyzes image data sent to the server and automatically determines the type, quantity, condition and expiration date of ingredients.

[0781] "Means for users to input their health status" refers to the interface and data input device that allows users to input and register their own health status.

[0782] The "means for generating a recipe and transmitting it to a user terminal" refers to a system and program for generating an appropriate recipe based on health status and ingredient data and transmitting the recipe to the user's terminal.

[0783] The "means for displaying the generated recipe" refers to a display and application for displaying the generated recipe on the user terminal.

[0784] The "means for automatically ordering missing ingredients" refers to a system and program that automatically orders missing ingredients from a food delivery service when ingredients required for a proposed recipe are missing.

[0785] The "means for notifying the user terminal of the delivery status of ordered ingredients" refers to a system and program for automatically notifying the user terminal of the delivery status of ordered ingredients.

[0786] "Generative AI" refers to models and algorithms for generating recipes using artificial intelligence based on specific tasks.

[0787] The "means for generating an optimal menu" refers to a system and program for generating an optimal menu for a user based on the user's health condition and tracked ingredient information.

[0788] This invention is a system used in a home refrigerator that simultaneously reduces food waste and maintains health. This system uses a program and a cloud server to manage ingredients, create recipes based on health information, and automatically order ingredients when they are in short supply. Specifically, the system is configured as follows:

[0789] System Configuration

[0790] 1. In-fridge camera:

[0791] A camera placed inside the refrigerator takes pictures of the ingredients, either a USB camera or a built-in camera.

[0792] 2. Server:

[0793] It functions as part of a cloud server that receives and analyzes image data of ingredients. The server is equipped with deep learning models (e.g., TensorFlow or PyTorch) that are used to recognize the type, quantity, condition, and expiration date of ingredients.

[0794] 3. User Device:

[0795] It consists of a smartphone or tablet that allows users to input their health status and receive and display recipes. The user interface includes a form for accepting health status input and a function for displaying generated recipes. Specifically, an iOS or Android application is used.

[0796] 4. Generation AI:

[0797] It is an artificial intelligence that generates appropriate recipes based on health and ingredient data, for example, GPT-4 and other generative models are used to provide recipes that suit the user's needs.

[0798] 5. Food delivery service:

[0799] This service has an automatic ordering function to make up for missing ingredients, and uses an API to generate orders and track the delivery status of ingredients.

[0800] Operation overview

[0801] 1. Register ingredients and send data:

[0802] The camera inside the refrigerator takes a picture of the new food item and sends the image to the server. The server receives the image data and analyzes it using a deep learning module. The analysis results include the type, quantity, condition, and expiration date of the food item, and these are stored in a database.

[0803] 2. User Health Management:

[0804] Users use a smartphone app to input their current health status, such as whether they are feeling unwell or on a diet. The server updates the profile database based on the received health status data.

[0805] 3. Recipe generation and distribution:

[0806] The server uses the AI ​​to generate optimal recipes based on the stored ingredient data and the user's health status data. The generated recipes are sent to the user's device, and the suggested recipes are displayed on the smartphone app.

[0807] 4. Automatic ordering of missing ingredients:

[0808] The server checks the refrigerator for missing ingredients based on the proposed recipe, automatically generates and sends an order to the food delivery service for the missing ingredients, and the user can check the status of the food delivery on their smartphone app.

[0809] Specific examples

[0810] What to do when you are unwell

[0811] 1. The user types "I feel like I have a cold" into a smartphone app.

[0812] 2. The server updates the health data and generates a porridge recipe suitable for the user's physical condition.

[0813] 3. The generated recipe is sent to the user's device and notified to the user.

[0814] 4. The user will see a message on their smartphone saying, "Let's try making porridge today."

[0815] Prompt Sentence Examples

[0816] The prompt is:

[0817] "The user's current health condition is as follows: poor health, desires easy-to-digest diet. The ingredients in the refrigerator are: tomatoes, cabbage, chicken. Please generate the optimal recipe based on these conditions."

[0818] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0819] Step 1:

[0820] A camera inside the refrigerator takes a picture of the newly added ingredient.

[0821] Input: Ingredients in the refrigerator

[0822] Output: Image data of ingredients

[0823] Specific operation: When the user puts ingredients into the refrigerator, the camera captures the moment and saves it as image data.

[0824] Step 2:

[0825] The server receives image data of the ingredients and analyzes the images to recognize the type, amount, condition, and expiration date of the ingredients.

[0826] Input: Image data of ingredients

[0827] Output: Analyzed food data (type, quantity, condition, expiration date)

[0828] How it works: Image data is sent over the internet to a server, which then analyzes the image using a deep learning model (e.g., TensorFlow). The analysis results are stored in a database.

[0829] Step 3:

[0830] Users input their own health status using a smartphone app.

[0831] Input: User's health status data (e.g., cold, desire for easy-to-digest meals, etc.)

[0832] Output: Updated health profile data

[0833] Specific operation: The user enters information such as "feeling a bit sick" or "on a diet" into the app. The entered data is sent to the server, and the profile database is updated.

[0834] Step 4:

[0835] Based on the ingredient data and health status data stored on the server, the generative AI generates the optimal recipe.

[0836] Input: Analyzed food ingredient data, health profile data

[0837] Output: The generated recipe

[0838] How it works: Generative AI (e.g., GPT-4) generates optimal recipes based on input data. For example, if you are feeling unwell, it will generate a recipe for easy-to-digest porridge.

[0839] Step 5:

[0840] The server sends the generated recipe to the user's device and displays it within the app.

[0841] Input: Generated recipe

[0842] Output: The recipe displayed on the user's terminal

[0843] Specific operation: The generated recipe is sent to the user's smartphone app and displayed as a notification within the app, such as "Today's recommended dish is porridge."

[0844] Step 6:

[0845] Based on the proposed recipe, the server checks the refrigerator for any missing ingredients and automatically generates and sends an order to the food delivery service.

[0846] Input: Generated recipe, analyzed food data in the refrigerator

[0847] Output: Order request to food delivery service

[0848] Specific operation: Checks the ingredients required for the generated recipe, and if there are any missing from the refrigerator, automatically sends an order request to the food delivery service. The order is generated using an API.

[0849] Step 7:

[0850] A user checks the status of their grocery delivery on a smartphone app.

[0851] Input: Delivery status data from food delivery service

[0852] Output: Delivery status displayed on the smartphone app (e.g., "Cabbage and tomatoes are scheduled for delivery tomorrow")

[0853] Specific operation: The server notifies the user's device of the delivery status obtained from the food delivery service and displays the status within the app.

[0854] By going through each step in this way, a system will be built that effectively reduces food waste and maintains health.

[0855] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0856] The present invention relates to a system that reduces food waste and proposes recipes that take into consideration the health of the household and the emotions of the user, using a camera built into a refrigerator, a server, a generative AI, a user terminal, an emotion engine, and a food delivery service. This system is specifically implemented as follows.

[0857] System Configuration

[0858] In-fridge camera: The camera is placed inside the refrigerator and is used to take pictures of the ingredients added.

[0859] User device: Consists of a smartphone, tablet, etc., and is used by users to input their health status and emotions, and receive and display recipes.

[0860] Server: A cloud system that receives and analyzes image data. This is where food ingredient data is generated.

[0861] Generative AI: An artificial intelligence system that generates appropriate recipes based on health status and ingredient data, as well as emotional data recognized by an emotion engine.

[0862] Emotion engine: A system that recognizes user emotions from user input data and other data sources and provides that information to a server.

[0863] Food delivery service: A service with an automatic ordering function to make up for missing ingredients.

[0864] Operation overview

[0865] Register ingredients and send data

[0866] Device: When a user places a new ingredient in the refrigerator, a camera takes a picture of it, which undergoes initial analysis and is then sent to the server.

[0867] Server: Receives image data and performs image analysis using a deep learning module. This allows the type, quantity, condition, and expiration date of ingredients to be recognized. The analysis results are stored in a database.

[0868] User health management and emotion recognition

[0869] Users: Use a smartphone app to input their current health status and emotions, including stress, anxiety, and joy.

[0870] Server: Analyzes the user's emotions from the input data using the emotion engine. The analysis results are stored in the user's emotion database.

[0871] Recipe generation and distribution

[0872] Server: The generative AI generates optimal recipes based on the stored ingredient data, the user's health status, and emotional data. For example, if the user is feeling stressed, it will suggest recipes for relaxing herbal teas or easy-to-make sweets.

[0873] Server: Sends the generated recipe to the user's device.

[0874] User: Viewing recipe suggestions on a smartphone app, such as "Today's recommended dish is cabbage and tomato salad."

[0875] Automatic ordering of missing ingredients

[0876] Server: Based on the proposed recipe, check the amount of ingredients currently in the refrigerator and check for any missing ingredients.

[0877] Server: Creates an order list to make up for any missing ingredients and generates order data for the food delivery service API.

[0878] User: Checks the status of a grocery delivery on a smartphone app. For example, it might say, "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[0879] Specific examples

[0880] Dealing with poor health and emotional frustration

[0881] 1. User: The user enters "I feel like I have a cold" and "I'm stressed" into a smartphone app.

[0882] 2. Server: The server updates the health and emotional data and generates recipes for porridge and herbal tea for stress relief that are suited to the user's physical condition.

[0883] 3. Server: Sends the generated recipe to the user's device and notifies the user.

[0884] 4. User: The smartphone displays the message, "Let's make porridge and herbal tea today."

[0885] Utilizing ingredients that are close to their expiration date and making suggestions based on emotions

[0886] 1. Terminal: The refrigerator camera takes a picture of the newly added milk and sends the image to the server.

[0887] 2. Server: Analyzes the image data and identifies the type of milk and its expiration date.

[0888] 3. Server: Updates the ingredient database and determines when the expiration date is approaching.

[0889] 4. Server: Based on the user's emotional data, it generates recipes for milk-based dishes (e.g., cream stew) and soothing desserts.

[0890] 5. Server: Sends the recipe to the user device.

[0891] 6. User: "Today, let's make cream stew and chocolate mousse" appears on the smartphone.

[0892] The present invention makes it possible to reduce food waste and effectively support users in maintaining their health and managing their emotions.

[0893] The processing flow will be explained below.

[0894] Specific processing of the program

[0895] Ingredient recognition and data transmission

[0896] Step 1:

[0897] Device: The camera built into the refrigerator takes pictures of the newly added ingredients by the user.

[0898] Step 2:

[0899] Terminal: Performs initial analysis of images captured by the camera and extracts features such as the shape, color, and tags of ingredients.

[0900] Step 3:

[0901] Terminal: Sends image data along with initial analysis data to a server via the internet.

[0902] Analysis of food ingredient data

[0903] Step 4:

[0904] Server: The received image data is input into a deep learning module to analyze the detailed type, quantity, and condition of the ingredients.

[0905] Step 5:

[0906] Server: Reads the expiration date on food packaging using OCR (optical character recognition) technology.

[0907] Step 6:

[0908] Server: Stores the analysis results in a food ingredient database. Stored data includes type, quantity, expiration date, and condition.

[0909] User health management and emotion recognition

[0910] Step 7:

[0911] User: Launches the smartphone app and inputs their current health condition and emotions. For example, they input that their physical condition is "feeling a bit like a cold" and their emotion is "stressed."

[0912] Step 8:

[0913] Server: Updates the received health status data in the profile database.

[0914] Step 9:

[0915] Server: Analyzes the user's emotions from the input data using the emotion engine. The analysis results are stored in the user's emotion database.

[0916] Recipe generation and distribution

[0917] Step 10:

[0918] Server: The generative AI generates optimal recipes based on the stored ingredient data, the user's health status, and emotional data. For example, if a user is feeling unwell and stressed, it will suggest a recipe for porridge and herbal tea that helps relieve stress.

[0919] Step 11:

[0920] Server: Sends the generated recipe to the user's device.

[0921] Step 12:

[0922] User: A smartphone app displays recipe suggestions, such as "Today, try making porridge and herbal tea."

[0923] Automatic ordering of missing ingredients

[0924] Step 13:

[0925] Server: Based on the proposed recipe, check the amount of ingredients currently in the refrigerator and check for any missing ingredients.

[0926] Step 14:

[0927] Server: Creates an order list to make up for any missing ingredients and generates order data for the food delivery service API.

[0928] Step 15:

[0929] User: Checks the status of a grocery delivery on a smartphone app. For example, it might say, "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[0930] Specific examples

[0931] Dealing with poor health and emotional frustration

[0932] Step 1:

[0933] User: The user types "feeling a bit under the weather" and "feeling stressed" into a smartphone app.

[0934] Step 2:

[0935] Server: The server updates the health and emotional data of the user and generates recipes for porridge and herbal tea for stress relief that are suited to the user's physical condition.

[0936] Step 3:

[0937] Server: Sends the generated recipe to the user's device and notifies the user.

[0938] Step 4:

[0939] User: "Let's make porridge and herbal tea today" appears on their phone.

[0940] Utilizing ingredients that are close to their expiration date and making suggestions based on emotions

[0941] Step 1:

[0942] Device: The camera in the refrigerator takes a picture of the newly added milk and sends the image to the server.

[0943] Step 2:

[0944] Server: Analyzes image data and identifies the type of milk and its expiration date.

[0945] Step 3:

[0946] Server: Updates the food ingredient database and determines when the expiration date is approaching.

[0947] Step 4:

[0948] User: The user types "I'm tired" into a smartphone app.

[0949] Step 5:

[0950] Server: Generates recipes for milk-based dishes (e.g., cream stew) and soothing desserts (e.g., chocolate mousse) based on ingredient data and emotion data.

[0951] Step 6:

[0952] Server: Sends recipes to user devices.

[0953] Step 7:

[0954] User: "Today, let's make cream stew and chocolate mousse" appears on their phone.

[0955] Example 2

[0956] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0957] In modern households, it is difficult to manage food ingredients, and many ingredients pass their expiration date. Health management is also difficult because meal suggestions do not take into account the user's health and emotional state. As a result, food waste and unhealthy eating habits have become problems.

[0958] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0959] In this invention, the server includes means for acquiring images of ingredients using an image acquisition device built into the refrigerator, means for transmitting image data of the ingredients to the server, means for analyzing the image data in the server and recognizing the type, amount, condition, and expiration date of the ingredients, means for the user to input their health condition and emotions, means for generating recipes using a generative AI model based on the health condition, emotion data, and ingredient data and transmitting the recipes to the user terminal, and means for displaying the generated recipes. This makes it possible to reduce food waste and suggest meals that take into consideration the user's health management and emotions.

[0960] An "image capture device built into a refrigerator" is a combination of hardware and software that is installed inside the refrigerator and is used to capture images of ingredients and acquire the data.

[0961] A "server" is a computer system that analyzes the received image data, health condition data, and emotion data and performs the necessary processing.

[0962] The "means for transmitting image data" is a communication means for transmitting image data of ingredients captured by the image capturing device in the refrigerator to the server.

[0963] "Means for analyzing image data" refers to software and algorithms that analyze the image data received by the server using a deep learning module or the like to recognize the type, quantity, condition, and expiration date of ingredients.

[0964] The "means for inputting health status" refers to a user terminal equipped with an interface for the user to input their own health status.

[0965] The "means for inputting emotions" refers to a user terminal equipped with an interface for the user to input their own emotions.

[0966] "Means for generating recipes using a generative artificial intelligence model" refers to software and algorithms for generating optimal recipes using a generative AI model based on health data, emotional data, and ingredient data.

[0967] A "user terminal" is a communication terminal device such as a smartphone or tablet that allows a user to check the generated recipe.

[0968] The "means for displaying the generated recipe" refers to an interface and software for visually displaying the generated recipe on the user terminal.

[0969] "Means for generating and sending orders to food delivery services" refers to software and communication means for automatically generating and sending orders using the food delivery service's API when necessary ingredients are in short supply.

[0970] "Initial processing means" refers to software and algorithms that extract features from the image data captured by the camera and perform preliminary analysis on the data before sending it to the server.

[0971] This invention is a system for reducing food waste, maintaining the user's health, and suggesting recipes based on emotions. This system is realized using an image capture device built into a refrigerator, a server, a generative AI model, a user terminal, an emotion engine, and a food delivery service. The operation of the system is as follows.

[0972] System Configuration

[0973] Refrigerator image capture device: This is a device that is installed inside the refrigerator to capture images of added ingredients.

[0974] User device: This consists of a smartphone or tablet, and is used by the user to input their health status and emotions, and to receive and display the generated recipes. The user device contains the interface and software.

[0975] Server: This is a computer system that receives and analyzes image data. The server uses deep learning modules (e.g., TensorFlow or PyTorch), databases (e.g., MySQL or PostgreSQL), and emotion engines (e.g., Emotion API).

[0976] Generative AI model: This is an artificial intelligence model that generates optimal recipes based on health status, emotional data, and ingredient data. For example, GPT-4 is used as a generative AI model.

[0977] Emotion engine: This is a system that recognizes the user's emotions and provides that information to the server. Specifically, it uses the Emotion API.

[0978] Food delivery service: A service that automatically generates order data when necessary ingredients are in short supply. For example, it uses APIs such as Amazon Fresh and Instacart.

[0979] Operation overview

[0980] This system operates in the following manner.

[0981] 1. Register ingredients and send data

[0982] When a user places new ingredients in the refrigerator, an image capture device inside the refrigerator takes a picture. The image data is sent to a server, which analyzes it using a deep learning module. This identifies the type, quantity, condition, and expiration date of the ingredients. The analysis results are then stored in a database.

[0983] 2. User Health Management and Emotion Recognition

[0984] The user inputs their current health status and emotions using a smartphone app. The server analyzes this data using an emotion engine and stores the results in an emotion database.

[0985] 3. Recipe generation and distribution

[0986] The server uses a generative AI model based on the stored ingredient data, the user's health status data, and emotional data to generate the optimal recipe. For example, the prompt text could read, "The user is feeling a bit under the weather and under a lot of stress. The ingredients available are cabbage, tomato, and chicken. Please suggest a dish that will have a relaxing effect using these ingredients."

[0987] 4. Automatic ordering of missing ingredients

[0988] Based on the proposed recipe, the system checks the amount of ingredients currently in the refrigerator and checks for any missing ingredients. If any ingredients are missing, it creates an order list and automatically sends the order data to the food delivery service's API.

[0989] Specific examples

[0990] 1. Dealing with poor health and emotional frustration

[0991] A user enters "feeling a bit like a cold" and "feeling stressed" into a smartphone app and submits the message.

[0992] Based on the data received by the server, a generative AI model is used to generate recipes for porridge and stress-relieving herbal tea.

[0993] The generated recipe is sent to the user terminal and notified to the user.

[0994] The user sees a message on their smartphone saying, "Let's make porridge and herbal tea today."

[0995] 2. Utilizing ingredients that are close to their expiration date and making suggestions based on emotions

[0996] The image capture device in the refrigerator takes a picture of the newly added milk and sends the image to the server.

[0997] The server analyzes the image data and identifies the type of milk and its expiration date.

[0998] The server updates the food ingredient database and determines when the expiration date is approaching.

[0999] The server generates recipes for cream stew using milk and a soothing dessert, for example, based on the user's emotional data.

[1000] The generated recipe is sent to the user terminal.

[1001] The user sees on their smartphone a message saying, "Today, let's make cream stew and chocolate mousse."

[1002] The present invention makes it possible to reduce food waste, maintain the user's health, and suggest meals that suit the user's emotions.

[1003] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1004] Step 1:

[1005] Food Registration

[1006] Terminal: When a user adds a new ingredient to the refrigerator, an image capture device in the refrigerator takes an image of the ingredient. The captured image data is sent from the terminal to the server.

[1007] Input: Image data of the newly added ingredient.

[1008] Output: Image data sent to the server.

[1009] Step 2:

[1010] Initial data analysis

[1011] Server: The received image data is analyzed using a deep learning module (e.g., TensorFlow or PyTorch). This analysis identifies the type, quantity, condition, and expiration date of the ingredients. The analysis results are stored in a database.

[1012] Input: Image data sent to the server.

[1013] Output: Data on the type, quantity, condition and expiry date of the recognized ingredients.

[1014] Step 3:

[1015] Health and Emotion Input

[1016] User: The user uses the smartphone app to input their own health condition and emotions. They select a condition such as "feeling a bit like a cold" or "feeling stressed" from the options and press the send button.

[1017] Input: Health and emotional information entered by the user into the app.

[1018] Output: Health and emotion data sent to the server.

[1019] Step 4:

[1020] Emotional Data Analysis

[1021] Server: Analyzes the received health status data and emotion data using an emotion engine (e.g., Emotion API). The analysis results are stored in an emotion database.

[1022] Input: User-entered health and emotion data.

[1023] Output: Stored sentiment analysis results.

[1024] Step 5:

[1025] Recipe Generation

[1026] Server: Generates optimal recipes using a generative AI model (such as GPT-4) based on the stored ingredient data, health status data, and emotion data. The prompt text is entered as follows: "The user is feeling a bit under the weather and is under a lot of stress. The ingredients available are cabbage, tomato, and chicken. Please suggest a dish that will have a relaxing effect using these ingredients."

[1027] Input: Food ingredient data, health condition data, emotion data, prompt sentence.

[1028] Output: The generated recipe data.

[1029] Step 6:

[1030] Recipe distribution

[1031] Server: Sends the generated recipe to the user's device.

[1032] Input: The generated recipe data.

[1033] Output: The recipe sent to the user's device.

[1034] Step 7:

[1035] Recipe display

[1036] User: The user checks the suggested recipes on the smartphone app. When the user opens the app, the suggested recipes are displayed.

[1037] Input: Recipe data sent from the server.

[1038] Output: The recipe displayed in the app.

[1039] Step 8:

[1040] Ingredients checking and automatic ordering

[1041] Server: Based on the proposed recipe, check the database for the amount of ingredients currently in the refrigerator. Check the required amount to see if any ingredients are missing.

[1042] Input: Current ingredient data and suggested recipe data.

[1043] Output: A list of missing ingredients.

[1044] Step 9:

[1045] Automatic Order Generation

[1046] Server: Based on the list of missing ingredients, order data is automatically generated and sent using the food delivery service's API.

[1047] Input: Missing ingredient list.

[1048] Output: Order data for food delivery service.

[1049] Step 10:

[1050] Delivery status notifications

[1051] Server: After the order is completed, check the delivery status and send the delivery information to the user's device.

[1052] Input: Delivery status information from a grocery delivery service.

[1053] Output: Delivery status notification sent to user device.

[1054] Step 11:

[1055] Delivery status display

[1056] User: The user checks the delivery status on their smartphone app, for example, "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[1057] Input: Delivery status notification.

[1058] Output: Delivery status information displayed in the app.

[1059] This will make it possible to reduce food waste, maintain health, and suggest recipes that respond to emotions.

[1060] (Application example 2)

[1061] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1062] Managing meals and maintaining one's health is important for passengers to stay comfortable in self-driving vehicles during long-distance travel. However, it is difficult to store appropriate ingredients, supply new ingredients, and suggest meals that take into account the passenger's health and emotions while traveling. Therefore, there is a need to provide a system that reduces food waste and makes meal suggestions that take into account the passenger's health and emotions while traveling in self-driving vehicles during long-distance travel.

[1063] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1064] In this invention, the server includes a means for capturing images of ingredients using an imaging device built into the refrigerator, a means for transmitting the ingredient information data to the server, a means for analyzing the information data in the server and recognizing the type, quantity, condition, and expiration date of the ingredients, a means for the user to input their health and emotional state, a means for generating suggestions based on the health and ingredient data and the emotional state data and transmitting the suggestions to the user terminal, and a means for displaying the generated suggestions. This automates meal management in an autonomous vehicle and enables meal suggestions based on the passenger's health and emotional state. Furthermore, if ingredients run low, new ingredients can be automatically replenished using a delivery service, allowing for comfortable meals to be served even while on the move.

[1065] An "imaging device" is a device that captures an image of a subject.

[1066] A "server" is a computer system that sends, receives, and processes data over a network.

[1067] "Information data" refers to data including images of ingredients and the user's health and emotional state.

[1068] "Analysis" is the process of breaking down input data and classifying, organizing, and evaluating it.

[1069] "Type of food ingredient" is information indicating the name and classification of food.

[1070] "Quantity" is information indicating the quantity or volume of ingredients.

[1071] "Status" is information indicating the freshness and storage state of the ingredients.

[1072] The "use by" date is information indicating the period during which the food ingredient can be safely used.

[1073] A "user terminal" is a device used by a user, such as a smartphone or tablet.

[1074] "Health status" is information relating to the user's physical condition and health.

[1075] "Emotional state" is information about the user's emotional and mental state.

[1076] "Suggestions" are recipes and meal recommendations generated based on the analysis results.

[1077] A "display means" is a method or device that visually shows the generated suggestions to the user.

[1078] "Delivery service" refers to a service that delivers food ingredients ordered online to a specified address.

[1079] The present invention provides a system for optimizing the dietary management and health status of passengers during long-distance travel in an autonomous vehicle and for suggesting recipes that take into consideration their emotions. This system is specifically implemented as follows.

[1080] System Configuration

[1081] 1. Imaging device (camera)

[1082] The system uses an imaging device built into the refrigerator inside the autonomous vehicle, which periodically captures images of the food items in the refrigerator and acquires the image data.

[1083] 2. Server

[1084] The image data is sent to a server where it is analyzed and the server uses deep learning techniques (e.g., TensorFlow, PyTorch) to identify the type, quantity, condition, and expiration date of the ingredients.

[1085] 3. User Device

[1086] Passengers use a user device such as a smartphone or tablet to access an application that inputs their health and emotional status. The application was developed using React Native.

[1087] 4. Emotion Engine

[1088] The health and emotional state data entered by the user into the device is analyzed by an emotion engine running on a cloud server (e.g., Microsoft Azure Cognitive Services).

[1089] 5. Generation AI

[1090] The generative AI (e.g., OpenAI GPT-4) on the server generates appropriate recipes based on the stored ingredient data and the user's health and emotional state.

[1091] 6. Display means

[1092] The generated recipe is sent to the user's smartphone or tablet in real time and displayed.

[1093] 7. Delivery Services

[1094] A delivery service will be used to automatically order and replenish missing ingredients at stops where the autonomous vehicle stops.

[1095] Operation overview

[1096] Program processing

[1097] The server receives the image data sent from the imaging device and analyzes the data using deep learning. As a result of the analysis, the type, quantity, condition, and expiration date of the ingredients are confirmed and stored in a database. The user inputs their health and emotional state using their user terminal. This input data is analyzed by the emotion engine and stored in the user's emotion database. The generation AI generates an optimal recipe based on the stored ingredient data and the user's health and emotional state, and sends it to the user terminal. The recipe is displayed in real time on the user terminal as a display method. If necessary ingredients are in short supply, an order is automatically issued to a delivery service, and they are replenished at the stop.

[1098] Specific examples

[1099] 1. User Input

[1100] A user types into a smartphone app, "I'm tired and would like a simple meal."

[1101] 2. Server Analysis

[1102] The server updates the health and emotion data and generates a recipe for "chicken and vegetable soup" that suits the user's physical condition.

[1103] 3. Sending and viewing recipes

[1104] The generated recipe is sent to the user terminal and displayed.

[1105] Generative AI model and prompts

[1106] Here's an example prompt that uses a generative AI model to generate a recipe:

[1107] User's current health status: Fatigue

[1108] User's current emotion: Wanting to relax

[1109] Usable ingredients: cabbage, carrots, chicken

[1110] Suggest a recipe to make:

[1111] This will enable passengers to enjoy comfortable meals and maintain their health while inside self-driving vehicles.

[1112] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1113] Step 1:

[1114] Photographing ingredients using an imaging device

[1115] The image capture device periodically captures images of food items in the refrigerator. The input is an image of the food items in the refrigerator, and the output is the captured image data. This image data is necessary to analyze the type, quantity, condition, and expiration date of the food items.

[1116] Step 2:

[1117] Sending image data to the server

[1118] Image data captured by an imaging device is sent to a server. The input is the captured image data, and the output is the image data uploaded to the server. This transmission prepares the server for subsequent analysis.

[1119] Step 3:

[1120] Image data analysis

[1121] The server analyzes the transmitted image data using deep learning technology (e.g., TensorFlow, PyTorch). The input is the image data uploaded to the server, and the output is the analysis results showing the type, quantity, condition, and expiration date of the ingredients. This analysis allows specific information about the ingredients to be registered in a database.

[1122] Step 4:

[1123] Input of the user's health and emotional state

[1124] Users use a smartphone or tablet to input their health and emotional status into the application. The input is information about the user's health and emotional status, and the output is the transmission of the input information. This allows data to be collected on the server for analysis by the emotion engine.

[1125] Step 5:

[1126] Analysis by emotion engine

[1127] The server uses an emotion engine (e.g., Microsoft Azure Cognitive Services) to analyze the user's health and emotional state. The input is the user's health and emotional state information, and the output is the analyzed emotion data. This data is stored in the user's emotion database.

[1128] Step 6:

[1129] Recipe generation using generative AI

[1130] The server's generative AI (e.g., OpenAI GPT-4) generates optimal recipes based on the stored ingredient data and the user's health and emotional state. The input is the stored ingredient data and the user's health and emotional state information, and the output is the generated recipe. This process suggests appropriate meals based on the user's state.

[1131] Step 7:

[1132] Sending and displaying recipes to user devices

[1133] The generated recipe is sent from the server to the user's smartphone or tablet. The input is the generated recipe, and the output is the recipe display on the user's device. The user can then cook a dish based on this recipe.

[1134] Step 8:

[1135] Use of delivery services

[1136] If a required ingredient is missing, the server automatically orders the missing ingredient from the delivery service. The input is the current ingredient status and the generated recipe, and the output is the order submission and replenishment of ingredients. This allows necessary ingredients to be replenished in a timely manner even while on the move, enabling comfortable meals to be served.

[1137] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1138] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1139] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1140] [Third embodiment]

[1141] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1142] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1143] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1144] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1145] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1146] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1147] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1148] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1149] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1150] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1151] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1152] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1153] The present invention relates to a system that reduces food waste and supports household health maintenance by using a camera built into a refrigerator, a server, a generation AI, a user terminal, and a food delivery service. This system is specifically implemented as follows.

[1154] System Configuration

[1155] In-fridge camera: The camera is placed inside the refrigerator and is used to take pictures of the ingredients added.

[1156] User device: Consists of a smartphone, tablet, etc., and is used by users to input their health status and receive and display recipes.

[1157] Server: A cloud system that receives and analyzes image data. This is where food ingredient data is generated.

[1158] Generative AI: Artificial intelligence that generates appropriate recipes based on health and ingredient data.

[1159] Food delivery service: A service with an automatic ordering function to make up for missing ingredients.

[1160] Operation overview

[1161] Register ingredients and send data

[1162] Device: When a user places a new ingredient in the refrigerator, a camera takes a picture of it, which undergoes initial analysis and is then sent to the server.

[1163] Server: Receives image data and performs image analysis using a deep learning module. This allows the type, quantity, condition, and expiration date of ingredients to be recognized. The analysis results are stored in a database.

[1164] User health management

[1165] User: Using a smartphone app, the user inputs their current health status, such as whether they are feeling unwell or on a diet.

[1166] Server: Updates the profile database based on the received health status data.

[1167] Recipe generation and distribution

[1168] Server: The AI ​​generates optimal recipes based on the stored ingredient data and the user's health status. If the user is not feeling well, it will generate recipes that are easy to digest, and under normal circumstances, it will generate recipes that prioritize ingredients that are close to their expiration date.

[1169] Server: Sends the generated recipe to the user's device.

[1170] User: A smartphone app displays recipe suggestions, such as "Today's recommended dish is cabbage and tomato salad."

[1171] Automatic ordering of missing ingredients

[1172] Server: Based on the proposed recipe, check the refrigerator for missing ingredients.

[1173] Server: Automatically generate and send orders to food delivery services to make up for missing ingredients.

[1174] User: Checks the status of a grocery delivery on a smartphone app, for example, with a notification that "cabbage and tomatoes are scheduled to be delivered tomorrow."

[1175] Specific examples

[1176] What to do when you are unwell

[1177] 1. User: The user types "I feel like I have a cold" into a smartphone app.

[1178] 2. Server: The server updates the health data and generates a porridge recipe suitable for the user's physical condition.

[1179] 3. Server: Sends the generated recipe to the user's device and notifies the user.

[1180] 4. User: The smartphone displays the message, "Let's make porridge today."

[1181] Utilizing ingredients that are close to their expiration date

[1182] 1. Terminal: The refrigerator camera takes a picture of the newly added milk and sends the image to the server.

[1183] 2. Server: Analyzes the image data and identifies the type of milk and its expiration date.

[1184] 3. Server: Updates the ingredient database and determines when the expiration date is approaching.

[1185] 4. Server: Generates recipes for dishes using milk (e.g., cream stew) and sends them to the user's device.

[1186] 5. User: The smartphone displays the message, "Let's make cream stew today."

[1187] The present invention makes it possible to reduce food waste and effectively support users in maintaining their health.

[1188] The processing flow will be explained below.

[1189] Specific processing of the program

[1190] Ingredient recognition and data transmission

[1191] Step 1:

[1192] Device: The camera built into the refrigerator takes pictures of the newly added ingredients by the user.

[1193] Step 2:

[1194] Terminal: Performs initial analysis of images captured by the camera and extracts features such as the shape, color, and tags of ingredients.

[1195] Step 3:

[1196] Terminal: Sends image data along with initial analysis data to a server via the internet.

[1197] Analysis of food ingredient data

[1198] Step 4:

[1199] Server: The received image data is input into a deep learning module to analyze the detailed type, quantity, and condition of the ingredients.

[1200] Step 5:

[1201] Server: Reads the expiration date printed on the food packaging using OCR (optical character recognition) technology.

[1202] Step 6:

[1203] Server: Stores the analysis results in a food ingredient database. Stored data includes type, quantity, expiration date, and condition.

[1204] User Health Check

[1205] Step 7:

[1206] User: Launches the smartphone app and enters their current health and physical condition.

[1207] Step 8:

[1208] Server: Updates the received health status data in the user profile database.

[1209] Recipe generation and distribution

[1210] Step 9:

[1211] Server: Generative AI generates optimal recipes based on ingredient data and health status data.

[1212] Step 10:

[1213] Server: Sends the generated recipe to the user's device.

[1214] Step 11:

[1215] User: A smartphone app displays a suggested recipe, for example, "Today's recommended dish is cabbage and tomato salad."

[1216] Automatic ordering of missing ingredients

[1217] Step 12:

[1218] Server: Based on the proposed recipe, check the amount of ingredients currently in the refrigerator and check for any missing ingredients.

[1219] Step 13:

[1220] Server: Creates an order list to make up for any missing ingredients and generates order data for the food delivery service API.

[1221] Step 14:

[1222] User: Checks the status of grocery delivery on a smartphone app. For example, it shows "cabbage and tomatoes are scheduled to be delivered tomorrow."

[1223] Example 1

[1224] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1225] There is a need for efficient methods to reduce food waste in the home and provide healthy meals. However, current refrigerator management systems make it difficult to accurately grasp the status of ingredients and tend to waste ingredients that are approaching their expiration date. Furthermore, they do not suggest optimal recipes based on health status, which makes it difficult to adequately manage health at home.

[1226] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1227] In this invention, the server

[1228] A means for taking an image of the food using a camera built into the refrigerator;

[1229] means for transmitting image data of ingredients to a server;

[1230] A means for analyzing image data in the server and recognizing the type, amount, condition and expiration date of ingredients;

[1231] a means for a user to input a health status;

[1232] A means for generating a recipe by the artificial intelligence based on the health status and the ingredient data and transmitting the recipe to the user terminal;

[1233] A display means for the generated recipe;

[1234] Includes.

[1235] This will automate food ingredient management and suggest optimal recipes based on health status, reducing food waste and helping to maintain household health.

[1236] A "camera built into a refrigerator" is a device that is installed inside the refrigerator and takes pictures of ingredients.

[1237] "Food ingredients" is a general term for food stored in a refrigerator.

[1238] "Image data" is digital data that contains visual information of ingredients photographed by a camera.

[1239] The "server" is a computer system installed on the cloud that analyzes image data of ingredients and provides services based on the user's health condition.

[1240] "Analysis" is the process of recognizing the type, quantity, condition, and expiration date of ingredients based on image data.

[1241] A "deep learning module" is a type of artificial intelligence technology used to analyze images of ingredients, and utilizes a multi-layer neural network.

[1242] A "user terminal" is a device that allows a user to input their health status and receive notifications and recipes from the system, and includes smartphones, tablets, etc.

[1243] "Health condition data" is information about the current health condition entered by the user.

[1244] "Generative AI" is artificial intelligence that generates optimal recipes based on the user's health condition and ingredient data.

[1245] A "recipe" is a document that describes the steps for preparing a dish using specific ingredients.

[1246] A "food delivery service" is a commercial service that automatically orders and delivers food ingredients needed by users.

[1247] A "database" is a system for systematically storing analysis results and user health status data.

[1248] An "order" is request data for ordering the necessary ingredients from the ingredient delivery service.

[1249] The "profile database" is a database for storing a user's past health condition data and food consumption history.

[1250] This invention is a system for supporting household food waste reduction and health maintenance. The system's main components are a camera built into the refrigerator, a server, a generative AI model, a user terminal, and a food delivery service.

[1251] Refrigerator camera

[1252] The camera built into the refrigerator, which is the terminal, is a device that takes pictures of ingredients added to the refrigerator. A high-resolution camera is used and is positioned so that it covers the entire interior of the refrigerator. When a user puts new ingredients into the refrigerator, the camera automatically detects this and takes a picture. This picture is sent to a server on the cloud via Wi-Fi. A secure protocol (e.g., HTTPS) is used for transmission.

[1253] Server and Image Analysis

[1254] The server analyzes the received image data using a deep learning module. For example, it uses a deep learning library such as TensorFlow to identify the type, quantity, condition (e.g., fresh, spoiled), and expiration date of the ingredients. The results of this analysis are stored in a database (e.g., MySQL or MongoDB). The stored data includes details such as the ingredient name, quantity, expiration date, and the date and time of addition.

[1255] User health management

[1256] Using a smartphone app, users input their current health status (e.g., feeling a bit under the weather, dieting, high stress) using text fields and pull-down menus. This information is sent in real time to a server, which then updates the user's profile database. This profile also includes past health data and food consumption history.

[1257] Recipe generation and distribution using generative AI

[1258] The server retrieves the stored ingredient data and the user's health status data and generates optimal recipes using a generative AI model. Examples of generative AI models include OpenAI's GPT-3 and other custom models. The generative AI takes into account the user's health status and available ingredient information to create healthy recipes. For example, it might recommend porridge when feeling unwell or a low-calorie salad when on a diet. This generated recipe is sent to the user's device and displayed on a smartphone app.

[1259] Automatic ordering of missing ingredients

[1260] The server checks the refrigerator for missing ingredients based on the proposed recipe. If any ingredients are found to be missing, it automatically calls the food delivery service's API and generates an order. The order includes credit card information and delivery address. Delivery information from the food delivery service is sent to the user's device. The user can check the delivery status on their smartphone app.

[1261] Examples of concrete examples and prompts

[1262] As a concrete example, let's look at how to respond when you're feeling unwell. When a user enters "I'm feeling a bit sick" into a smartphone app, the server updates the health data and generates a porridge recipe suited to the user's physical condition. The generated recipe is sent to the user's device, and the smartphone displays a message saying, "Let's try making porridge today."

[1263] We will also give a concrete example of how to utilize ingredients that are approaching their expiration date. The camera in the refrigerator takes a picture of the newly added milk and sends the image to the server. The server analyzes the image data and recognizes the type of milk and its expiration date. It updates the ingredient database and determines that the expiration date is approaching. A recipe for a dish that uses the milk (for example, cream stew) is generated and sent to the user's device. The user is then prompted on their smartphone, "Let's try making cream stew today."

[1264] Examples of prompts:

[1265] "When you put new food in the refrigerator, the camera automatically takes a picture and sends it to the server."

[1266] "When a user inputs information about when they are feeling unwell into a smartphone app, an appropriate recipe is generated based on that information."

[1267] "A function that automatically orders missing ingredients from a delivery service based on the suggested recipe."

[1268] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1269] Step 1: Take a photo of the food

[1270] When a user adds new ingredients to the refrigerator, the device's built-in camera automatically takes a picture of the ingredients.

[1271] Input: New ingredients added to the fridge

[1272] Output: Image data of ingredients

[1273] The images are high resolution and the cameras are positioned to cover the entire interior of the refrigerator. Once the images are captured, the image data is used in the next step.

[1274] Step 2: Sending image data

[1275] The device sends the captured image data to a cloud server via Wi-Fi using a secure protocol (e.g., HTTPS).

[1276] Input: Image data of the photographed food

[1277] Output: Image data sent to the server

[1278] This allows the image data to reach the server for analysis.

[1279] Step 3: Analyzing the image data

[1280] The server analyzes the received image data using a deep learning module (e.g., TensorFlow) to identify the type, quantity, condition, and expiration date of the ingredients.

[1281] Input: Image data of the ingredients sent

[1282] Output: Type, quantity, condition and expiration date of recognized ingredients

[1283] The analysis results include details such as ingredient name, quantity, expiration date, and date and time of addition, which are stored in a database.

[1284] Step 4: Save the ingredients data

[1285] The server stores the analysis results in a database (e.g., MySQL or MongoDB).

[1286] Input: Detailed information about the recognized ingredients

[1287] Output: Saved ingredient data

[1288] Based on the stored data, the ingredients are managed in the next step.

[1289] Step 5: Enter your health status

[1290] Users enter their current health status using a smartphone app, which provides text fields and pull-down menus for input.

[1291] Input: User's health status information (e.g., feeling a bit under the weather, dieting, high stress)

[1292] Output: Health status data sent to the server

[1293] The submitted data is added to the user's profile database.

[1294] Step 6: Update your profile

[1295] The server then updates the user's profile database based on the received health status data, which includes past health status data and food consumption history.

[1296] Input: Health status data sent by the user

[1297] Output: Updated profile data

[1298] This lays the foundation for generating optimal recipes based on the user's health condition.

[1299] Step 7: Generative AI generates recipes

[1300] The server retrieves the stored ingredient data and the user's health status data and generates optimal recipes using generative AI models, including OpenAI's GPT-3 and other custom models.

[1301] Input: Stored food and health data

[1302] Output: The generated recipe

[1303] The generative AI uses this data to create recipes, taking into account the patient's health status and available ingredients. For example, it might recommend porridge if you're feeling unwell, or a low-calorie salad if you're on a diet.

[1304] Step 8: Submit your recipe

[1305] The server then sends the generated recipe to the user's device, which includes the ingredients, cooking instructions, and nutritional information.

[1306] Input: Generated recipe

[1307] Output: Recipe sent to user's device

[1308] This allows the user to receive and check the recipe.

[1309] Step 9: View the recipe

[1310] The user checks the suggested recipes on the smartphone app, which displays something like, "Today's recommended dish is cabbage and tomato salad."

[1311] Input: Recipe sent from the server

[1312] Output: Recipe displayed on smartphone app

[1313] This allows the user to easily check the suggested menu.

[1314] Step 10: Check for missing ingredients

[1315] The server checks the refrigerator for missing ingredients based on the proposed recipe.

[1316] Input: Proposed recipe and current ingredient data

[1317] Output: List of missing ingredients

[1318] If any ingredients are found to be missing, an automatic order will be placed in the next step.

[1319] Step 11: Automated Orders

[1320] The server automatically generates and sends an order to a food delivery service for any missing ingredients, along with credit card information and a delivery address.

[1321] Input: Missing ingredients list

[1322] Output: Order data sent to grocery delivery service

[1323] This will automatically replenish any missing ingredients.

[1324] Step 12: Delivery Information Notification

[1325] The server receives delivery information from the food delivery service and notifies the user terminal of the delivery information, including the scheduled delivery date and time and the delivery status.

[1326] Input: Delivery information from food delivery service

[1327] Output: Delivery information sent to the user's device

[1328] The user checks the status of their grocery delivery on a smartphone app, which displays a notification such as "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[1329] (Application example 1)

[1330] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1331] Efficient and accurate food management is essential to simultaneously reduce food waste at home and maintain good health. However, current systems make food management cumbersome, and they rarely suggest meals that are particularly suited to a person's health or automatically order ingredients that are in short supply. This leads to food waste and makes it difficult for users to maintain their health. Therefore, a system that effectively supports reducing food waste and maintaining good health is needed.

[1332] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1333] In this invention, the server includes means for taking images of ingredients with a camera built into the refrigerator, means for transmitting image data of ingredients to the server, means for analyzing the image data in the server and recognizing the type, amount, condition, and expiration date of ingredients, means for the user to input health status, means for generating recipes based on the health status and ingredient data and transmitting them to the user terminal, means for displaying the generated recipe, means for automatically ordering missing ingredients, means for notifying the user terminal of the delivery status of the automatically ordered ingredients, and means for using generation AI to generate optimal menus based on the health status and tracked ingredient information. This effectively reduces food waste and maintains health, making it easy for users to eat healthy meals.

[1334] A "camera built into a refrigerator" is a camera built into the refrigerator that takes pictures of food stored inside the refrigerator.

[1335] The "means for transmitting image data of ingredients to the server" refers to a communication device and protocol for transmitting image data captured by a camera built into the refrigerator to the server via the Internet.

[1336] "Means for analyzing image data and recognizing the type, quantity, condition and expiration date of ingredients" refers to a program and process that analyzes image data sent to the server and automatically determines the type, quantity, condition and expiration date of ingredients.

[1337] "Means for users to input their health status" refers to the interface and data input device that allows users to input and register their own health status.

[1338] The "means for generating a recipe and transmitting it to a user terminal" refers to a system and program for generating an appropriate recipe based on health status and ingredient data and transmitting the recipe to the user's terminal.

[1339] The "means for displaying the generated recipe" refers to a display and application for displaying the generated recipe on the user terminal.

[1340] The "means for automatically ordering missing ingredients" refers to a system and program that automatically orders missing ingredients from a food delivery service when ingredients required for a proposed recipe are missing.

[1341] The "means for notifying the user terminal of the delivery status of ordered ingredients" refers to a system and program for automatically notifying the user terminal of the delivery status of ordered ingredients.

[1342] "Generative AI" refers to models and algorithms for generating recipes using artificial intelligence based on specific tasks.

[1343] The "means for generating an optimal menu" refers to a system and program for generating an optimal menu for a user based on the user's health condition and tracked ingredient information.

[1344] This invention is a system used in a home refrigerator that simultaneously reduces food waste and maintains health. This system uses a program and a cloud server to manage ingredients, create recipes based on health information, and automatically order ingredients when they are in short supply. Specifically, the system is configured as follows:

[1345] System Configuration

[1346] 1. In-fridge camera:

[1347] A camera placed inside the refrigerator takes pictures of the ingredients, either a USB camera or a built-in camera.

[1348] 2. Server:

[1349] It functions as part of a cloud server that receives and analyzes image data of ingredients. The server is equipped with deep learning models (e.g., TensorFlow or PyTorch) that are used to recognize the type, quantity, condition, and expiration date of ingredients.

[1350] 3. User Device:

[1351] It consists of a smartphone or tablet that allows users to input their health status and receive and display recipes. The user interface includes a form for accepting health status input and a function for displaying generated recipes. Specifically, an iOS or Android application is used.

[1352] 4. Generation AI:

[1353] It is an artificial intelligence that generates appropriate recipes based on health and ingredient data, for example, GPT-4 and other generative models are used to provide recipes that suit the user's needs.

[1354] 5. Food delivery services:

[1355] This service has an automatic ordering function to make up for missing ingredients, and uses an API to generate orders and track the delivery status of ingredients.

[1356] Operation overview

[1357] 1. Register ingredients and send data:

[1358] The camera inside the refrigerator takes a picture of the new food item and sends the image to the server. The server receives the image data and analyzes it using a deep learning module. The analysis results include the type, quantity, condition, and expiration date of the food item, and these are stored in a database.

[1359] 2. User Health Management:

[1360] Users use a smartphone app to input their current health status, such as whether they are feeling unwell or on a diet. The server updates the profile database based on the received health status data.

[1361] 3. Recipe generation and distribution:

[1362] The server uses the AI ​​to generate optimal recipes based on the stored ingredient data and the user's health status data. The generated recipes are sent to the user's device, and the suggested recipes are displayed on the smartphone app.

[1363] 4. Automatic ordering of missing ingredients:

[1364] The server checks the refrigerator for missing ingredients based on the proposed recipe, automatically generates and sends an order to the food delivery service for the missing ingredients, and the user can check the status of the food delivery on their smartphone app.

[1365] Specific examples

[1366] What to do when you are unwell

[1367] 1. The user types "I feel like I have a cold" into a smartphone app.

[1368] 2. The server updates the health data and generates a porridge recipe suitable for the user's physical condition.

[1369] 3. The generated recipe is sent to the user's device and notified to the user.

[1370] 4. The user will see a message on their smartphone saying, "Let's make porridge today."

[1371] Prompt Sentence Examples

[1372] The prompt is:

[1373] "The user's current health condition is as follows: poor health, desires easy-to-digest diet. The ingredients in the refrigerator are: tomatoes, cabbage, chicken. Please generate the optimal recipe based on these conditions."

[1374] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1375] Step 1:

[1376] A camera inside the refrigerator takes a picture of the newly added ingredient.

[1377] Input: Ingredients in the refrigerator

[1378] Output: Image data of ingredients

[1379] Specific operation: When the user puts ingredients into the refrigerator, the camera captures the moment and saves it as image data.

[1380] Step 2:

[1381] The server receives image data of the ingredients and analyzes the images to recognize the type, amount, condition, and expiration date of the ingredients.

[1382] Input: Image data of ingredients

[1383] Output: Analyzed food data (type, quantity, condition, expiration date)

[1384] How it works: Image data is sent over the internet to a server, which then analyzes the image using a deep learning model (e.g., TensorFlow). The analysis results are stored in a database.

[1385] Step 3:

[1386] Users input their own health status using a smartphone app.

[1387] Input: User's health status data (e.g., cold, desire for easy-to-digest meals, etc.)

[1388] Output: Updated health profile data

[1389] Specific operation: The user enters information such as "feeling a bit sick" or "on a diet" into the app. The entered data is sent to the server, and the profile database is updated.

[1390] Step 4:

[1391] Based on the ingredient data and health status data stored on the server, the generative AI generates the optimal recipe.

[1392] Input: Analyzed food ingredient data, health profile data

[1393] Output: The generated recipe

[1394] How it works: Generative AI (e.g., GPT-4) generates optimal recipes based on input data. For example, if you are feeling unwell, it will generate a recipe for easy-to-digest porridge.

[1395] Step 5:

[1396] The server sends the generated recipe to the user's device and displays it within the app.

[1397] Input: Generated recipe

[1398] Output: The recipe displayed on the user's terminal

[1399] Specific operation: The generated recipe is sent to the user's smartphone app and displayed as a notification within the app, such as "Today's recommended dish is porridge."

[1400] Step 6:

[1401] Based on the proposed recipe, the server checks the refrigerator for any missing ingredients and automatically generates and sends an order to the food delivery service.

[1402] Input: Generated recipe, analyzed food data in the refrigerator

[1403] Output: Order request to food delivery service

[1404] Specific operation: Checks the ingredients required for the generated recipe, and if there are any missing from the refrigerator, automatically sends an order request to the food delivery service. The order is generated using an API.

[1405] Step 7:

[1406] A user checks the status of their grocery delivery on a smartphone app.

[1407] Input: Delivery status data from food delivery service

[1408] Output: Delivery status displayed on the smartphone app (e.g., "Cabbage and tomatoes are scheduled for delivery tomorrow")

[1409] Specific operation: The server notifies the user's device of the delivery status obtained from the food delivery service and displays the status within the app.

[1410] By going through each step in this way, a system will be built that effectively reduces food waste and maintains health.

[1411] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1412] The present invention relates to a system that reduces food waste and proposes recipes that take into consideration the health of the household and the emotions of the user, using a camera built into a refrigerator, a server, a generative AI, a user terminal, an emotion engine, and a food delivery service. This system is specifically implemented as follows.

[1413] System Configuration

[1414] In-fridge camera: The camera is placed inside the refrigerator and is used to take pictures of the ingredients added.

[1415] User device: Consists of a smartphone, tablet, etc., and is used by users to input their health status and emotions, and receive and display recipes.

[1416] Server: A cloud system that receives and analyzes image data. This is where food ingredient data is generated.

[1417] Generative AI: An artificial intelligence system that generates appropriate recipes based on health status and ingredient data, as well as emotional data recognized by an emotion engine.

[1418] Emotion engine: A system that recognizes user emotions from user input data and other data sources and provides that information to a server.

[1419] Food delivery service: A service with an automatic ordering function to make up for missing ingredients.

[1420] Operation overview

[1421] Register ingredients and send data

[1422] Device: When a user places a new ingredient in the refrigerator, a camera takes a picture of it, which undergoes initial analysis and is then sent to the server.

[1423] Server: Receives image data and performs image analysis using a deep learning module. This allows the type, quantity, condition, and expiration date of ingredients to be recognized. The analysis results are stored in a database.

[1424] User health management and emotion recognition

[1425] Users: Use a smartphone app to input their current health status and emotions, including stress, anxiety, and joy.

[1426] Server: Analyzes the user's emotions from the input data using the emotion engine. The analysis results are stored in the user's emotion database.

[1427] Recipe generation and distribution

[1428] Server: The generative AI generates optimal recipes based on the stored ingredient data, the user's health status, and emotional data. For example, if the user is feeling stressed, it will suggest recipes for relaxing herbal teas or easy-to-make sweets.

[1429] Server: Sends the generated recipe to the user's device.

[1430] User: Viewing recipe suggestions on a smartphone app, such as "Today's recommended dish is cabbage and tomato salad."

[1431] Automatic ordering of missing ingredients

[1432] Server: Based on the proposed recipe, check the amount of ingredients currently in the refrigerator and check for any missing ingredients.

[1433] Server: Creates an order list to make up for any missing ingredients and generates order data for the food delivery service API.

[1434] User: Checks the status of a grocery delivery on a smartphone app. For example, it might say, "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[1435] Specific examples

[1436] Dealing with poor health and emotional frustration

[1437] 1. User: The user enters "I feel like I have a cold" and "I'm stressed" into a smartphone app.

[1438] 2. Server: The server updates the health and emotional data and generates recipes for porridge and herbal tea for stress relief that are suited to the user's physical condition.

[1439] 3. Server: Sends the generated recipe to the user's device and notifies the user.

[1440] 4. User: The smartphone displays the message, "Let's make porridge and herbal tea today."

[1441] Utilizing ingredients that are close to their expiration date and making suggestions based on emotions

[1442] 1. Terminal: The refrigerator camera takes a picture of the newly added milk and sends the image to the server.

[1443] 2. Server: Analyzes the image data and identifies the type of milk and its expiration date.

[1444] 3. Server: Updates the ingredient database and determines when the expiration date is approaching.

[1445] 4. Server: Based on the user's emotional data, it generates recipes for milk-based dishes (e.g., cream stew) and soothing desserts.

[1446] 5. Server: Sends the recipe to the user device.

[1447] 6. User: "Today, let's make cream stew and chocolate mousse" appears on the smartphone.

[1448] The present invention makes it possible to reduce food waste and effectively support users in maintaining their health and managing their emotions.

[1449] The processing flow will be explained below.

[1450] Specific processing of the program

[1451] Ingredient recognition and data transmission

[1452] Step 1:

[1453] Device: The camera built into the refrigerator takes pictures of the newly added ingredients by the user.

[1454] Step 2:

[1455] Terminal: Performs initial analysis of images captured by the camera and extracts features such as the shape, color, and tags of ingredients.

[1456] Step 3:

[1457] Terminal: Sends image data along with initial analysis data to a server via the internet.

[1458] Analysis of food ingredient data

[1459] Step 4:

[1460] Server: The received image data is input into a deep learning module to analyze the detailed type, quantity, and condition of the ingredients.

[1461] Step 5:

[1462] Server: Reads the expiration date printed on the food packaging using OCR (optical character recognition) technology.

[1463] Step 6:

[1464] Server: Stores the analysis results in a food ingredient database. Stored data includes type, quantity, expiration date, and condition.

[1465] User health management and emotion recognition

[1466] Step 7:

[1467] User: Launches the smartphone app and inputs their current health condition and emotions. For example, they input that their physical condition is "feeling a bit like a cold" and their emotion is "stressed."

[1468] Step 8:

[1469] Server: Updates the received health status data in the profile database.

[1470] Step 9:

[1471] Server: Analyzes the user's emotions from the input data using the emotion engine. The analysis results are stored in the user's emotion database.

[1472] Recipe generation and distribution

[1473] Step 10:

[1474] Server: The generative AI generates optimal recipes based on the stored ingredient data, the user's health status, and emotional data. For example, if a user is feeling unwell and stressed, it will suggest a recipe for porridge and herbal tea that helps relieve stress.

[1475] Step 11:

[1476] Server: Sends the generated recipe to the user's device.

[1477] Step 12:

[1478] User: A smartphone app displays recipe suggestions, such as "Today, try making porridge and herbal tea."

[1479] Automatic ordering of missing ingredients

[1480] Step 13:

[1481] Server: Based on the proposed recipe, check the amount of ingredients currently in the refrigerator and check for any missing ingredients.

[1482] Step 14:

[1483] Server: Creates an order list to make up for any missing ingredients and generates order data for the food delivery service API.

[1484] Step 15:

[1485] User: Checks the status of a grocery delivery on a smartphone app. For example, it might say, "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[1486] Specific examples

[1487] Dealing with poor health and emotional frustration

[1488] Step 1:

[1489] User: The user types "feeling a bit under the weather" and "feeling stressed" into a smartphone app.

[1490] Step 2:

[1491] Server: The server updates the health and emotional data of the user and generates recipes for porridge and herbal tea for stress relief that are suited to the user's physical condition.

[1492] Step 3:

[1493] Server: Sends the generated recipe to the user's device and notifies the user.

[1494] Step 4:

[1495] User: "Let's make porridge and herbal tea today" appears on their phone.

[1496] Utilizing ingredients that are close to their expiration date and making suggestions based on emotions

[1497] Step 1:

[1498] Device: The camera in the refrigerator takes a picture of the newly added milk and sends the image to the server.

[1499] Step 2:

[1500] Server: Analyzes image data and identifies the type of milk and its expiration date.

[1501] Step 3:

[1502] Server: Updates the food ingredient database and determines when the expiration date is approaching.

[1503] Step 4:

[1504] User: The user types "I'm tired" into a smartphone app.

[1505] Step 5:

[1506] Server: Generates recipes for milk-based dishes (e.g., cream stew) and soothing desserts (e.g., chocolate mousse) based on ingredient data and emotion data.

[1507] Step 6:

[1508] Server: Sends recipes to user devices.

[1509] Step 7:

[1510] User: "Today, let's make cream stew and chocolate mousse" appears on their phone.

[1511] Example 2

[1512] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1513] In modern households, it is difficult to manage food ingredients, and many ingredients pass their expiration date. Health management is also difficult because meal suggestions do not take into account the user's health and emotional state. As a result, food waste and unhealthy eating habits have become problems.

[1514] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1515] In this invention, the server includes means for acquiring images of ingredients using an image acquisition device built into the refrigerator, means for transmitting image data of the ingredients to the server, means for analyzing the image data in the server and recognizing the type, amount, condition, and expiration date of the ingredients, means for the user to input their health condition and emotions, means for generating recipes using a generative AI model based on the health condition, emotion data, and ingredient data and transmitting the recipes to the user terminal, and means for displaying the generated recipes. This makes it possible to reduce food waste and suggest meals that take into consideration the user's health management and emotions.

[1516] An "image capture device built into a refrigerator" is a combination of hardware and software that is installed inside the refrigerator and is used to capture images of ingredients and acquire the data.

[1517] A "server" is a computer system that analyzes the received image data, health condition data, and emotion data and performs the necessary processing.

[1518] The "means for transmitting image data" is a communication means for transmitting image data of ingredients captured by the image capturing device in the refrigerator to the server.

[1519] "Means for analyzing image data" refers to software and algorithms that analyze the image data received by the server using a deep learning module or the like to recognize the type, quantity, condition, and expiration date of ingredients.

[1520] The "means for inputting health status" refers to a user terminal equipped with an interface for the user to input their own health status.

[1521] The "means for inputting emotions" refers to a user terminal equipped with an interface for the user to input their own emotions.

[1522] "Means for generating recipes using a generative artificial intelligence model" refers to software and algorithms for generating optimal recipes using a generative AI model based on health data, emotional data, and ingredient data.

[1523] A "user terminal" is a communication terminal device such as a smartphone or tablet that allows a user to check the generated recipe.

[1524] The "means for displaying the generated recipe" refers to an interface and software for visually displaying the generated recipe on the user terminal.

[1525] "Means for generating and sending orders to food delivery services" refers to software and communication means for automatically generating and sending orders using the food delivery service's API when necessary ingredients are in short supply.

[1526] "Initial processing means" refers to software and algorithms that extract features from the image data captured by the camera and perform preliminary analysis on the data before sending it to the server.

[1527] This invention is a system for reducing food waste, maintaining the user's health, and suggesting recipes based on emotions. This system is realized using an image capture device built into a refrigerator, a server, a generative AI model, a user terminal, an emotion engine, and a food delivery service. The operation of the system is as follows.

[1528] System Configuration

[1529] Refrigerator image capture device: This is a device that is installed inside the refrigerator to capture images of added ingredients.

[1530] User device: This consists of a smartphone or tablet, and is used by the user to input their health status and emotions, and to receive and display the generated recipes. The user device contains the interface and software.

[1531] Server: This is a computer system that receives and analyzes image data. The server uses deep learning modules (e.g., TensorFlow or PyTorch), databases (e.g., MySQL or PostgreSQL), and emotion engines (e.g., Emotion API).

[1532] Generative AI model: This is an artificial intelligence model that generates optimal recipes based on health status, emotional data, and ingredient data. For example, GPT-4 is used as a generative AI model.

[1533] Emotion engine: This is a system that recognizes the user's emotions and provides that information to the server. Specifically, it uses the Emotion API.

[1534] Food delivery service: A service that automatically generates order data when necessary ingredients are in short supply. For example, it uses APIs such as Amazon Fresh and Instacart.

[1535] Operation overview

[1536] This system operates in the following manner.

[1537] 1. Register ingredients and send data

[1538] When a user places new ingredients in the refrigerator, an image capture device inside the refrigerator takes a picture. The image data is sent to a server, which analyzes it using a deep learning module. This identifies the type, quantity, condition, and expiration date of the ingredients. The analysis results are then stored in a database.

[1539] 2. User Health Management and Emotion Recognition

[1540] The user inputs their current health status and emotions using a smartphone app. The server analyzes this data using an emotion engine and stores the results in an emotion database.

[1541] 3. Recipe generation and distribution

[1542] The server uses a generative AI model based on the stored ingredient data, the user's health status data, and emotional data to generate the optimal recipe. For example, the prompt text could read, "The user is feeling a bit under the weather and under a lot of stress. The ingredients available are cabbage, tomato, and chicken. Please suggest a dish that will have a relaxing effect using these ingredients."

[1543] 4. Automatic ordering of missing ingredients

[1544] Based on the proposed recipe, the system checks the amount of ingredients currently in the refrigerator and checks for any missing ingredients. If any ingredients are missing, it creates an order list and automatically sends the order data to the food delivery service's API.

[1545] Specific examples

[1546] 1. Dealing with poor health and emotional frustration

[1547] A user enters "feeling a bit like a cold" and "feeling stressed" into a smartphone app and submits the message.

[1548] Based on the data received by the server, a generative AI model is used to generate recipes for porridge and stress-relieving herbal tea.

[1549] The generated recipe is sent to the user terminal and notified to the user.

[1550] The user sees a message on their smartphone saying, "Let's make porridge and herbal tea today."

[1551] 2. Utilizing ingredients that are close to their expiration date and making suggestions based on emotions

[1552] The image capture device in the refrigerator takes a picture of the newly added milk and sends the image to the server.

[1553] The server analyzes the image data and identifies the type of milk and its expiration date.

[1554] The server updates the food ingredient database and determines when the expiration date is approaching.

[1555] The server generates recipes for cream stew using milk and a soothing dessert, for example, based on the user's emotional data.

[1556] The generated recipe is sent to the user terminal.

[1557] The user sees on their smartphone a message saying, "Today, let's make cream stew and chocolate mousse."

[1558] The present invention makes it possible to reduce food waste, maintain the user's health, and suggest meals that suit the user's emotions.

[1559] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1560] Step 1:

[1561] Food Registration

[1562] Terminal: When a user adds a new ingredient to the refrigerator, an image capture device in the refrigerator takes an image of the ingredient. The captured image data is sent from the terminal to the server.

[1563] Input: Image data of the newly added ingredient.

[1564] Output: Image data sent to the server.

[1565] Step 2:

[1566] Initial data analysis

[1567] Server: The received image data is analyzed using a deep learning module (e.g., TensorFlow or PyTorch). This analysis identifies the type, quantity, condition, and expiration date of the ingredients. The analysis results are stored in a database.

[1568] Input: Image data sent to the server.

[1569] Output: Data on the type, quantity, condition and expiry date of the recognized ingredients.

[1570] Step 3:

[1571] Health and Emotion Input

[1572] User: The user uses the smartphone app to input their own health condition and emotions. They select a condition such as "feeling a bit like a cold" or "feeling stressed" from the options and press the send button.

[1573] Input: Health and emotional information entered by the user into the app.

[1574] Output: Health and emotion data sent to the server.

[1575] Step 4:

[1576] Emotional Data Analysis

[1577] Server: Analyzes the received health status data and emotion data using an emotion engine (e.g., Emotion API). The analysis results are stored in an emotion database.

[1578] Input: User-entered health and emotion data.

[1579] Output: Stored sentiment analysis results.

[1580] Step 5:

[1581] Recipe Generation

[1582] Server: Generates optimal recipes using a generative AI model (such as GPT-4) based on the stored ingredient data, health status data, and emotion data. The prompt text is entered as follows: "The user is feeling a bit under the weather and is under a lot of stress. The ingredients available are cabbage, tomato, and chicken. Please suggest a dish that will have a relaxing effect using these ingredients."

[1583] Input: Food ingredient data, health condition data, emotion data, prompt sentence.

[1584] Output: The generated recipe data.

[1585] Step 6:

[1586] Recipe distribution

[1587] Server: Sends the generated recipe to the user's device.

[1588] Input: The generated recipe data.

[1589] Output: The recipe sent to the user's device.

[1590] Step 7:

[1591] Recipe Display

[1592] User: The user checks the suggested recipes on the smartphone app. When the user opens the app, the suggested recipes are displayed.

[1593] Input: Recipe data sent from the server.

[1594] Output: The recipe displayed in the app.

[1595] Step 8:

[1596] Ingredients checking and automatic ordering

[1597] Server: Based on the proposed recipe, check the database for the amount of ingredients currently in the refrigerator. Check the required amount to see if any ingredients are missing.

[1598] Input: Current ingredient data and suggested recipe data.

[1599] Output: A list of missing ingredients.

[1600] Step 9:

[1601] Automatic Order Generation

[1602] Server: Based on the list of missing ingredients, order data is automatically generated and sent using the food delivery service's API.

[1603] Input: Missing ingredient list.

[1604] Output: Order data for food delivery service.

[1605] Step 10:

[1606] Delivery status notifications

[1607] Server: After the order is completed, check the delivery status and send the delivery information to the user's device.

[1608] Input: Delivery status information from a grocery delivery service.

[1609] Output: Delivery status notification sent to user device.

[1610] Step 11:

[1611] Delivery status display

[1612] User: The user checks the delivery status on their smartphone app, for example, "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[1613] Input: Delivery status notification.

[1614] Output: Delivery status information displayed in the app.

[1615] This will make it possible to reduce food waste, maintain health, and suggest recipes that respond to emotions.

[1616] (Application example 2)

[1617] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1618] Managing meals and maintaining one's health is important for passengers to stay comfortable in self-driving vehicles during long-distance travel. However, it is difficult to store appropriate ingredients, supply new ingredients, and suggest meals that take into account the passenger's health and emotions while traveling. Therefore, there is a need to provide a system that reduces food waste and makes meal suggestions that take into account the passenger's health and emotions while traveling in self-driving vehicles during long-distance travel.

[1619] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1620] In this invention, the server includes a means for capturing images of ingredients using an imaging device built into the refrigerator, a means for transmitting the ingredient information data to the server, a means for analyzing the information data in the server and recognizing the type, quantity, condition, and expiration date of the ingredients, a means for the user to input their health and emotional state, a means for generating suggestions based on the health and ingredient data and the emotional state data and transmitting the suggestions to the user terminal, and a means for displaying the generated suggestions. This automates meal management in an autonomous vehicle and enables meal suggestions based on the passenger's health and emotional state. Furthermore, if ingredients run low, new ingredients can be automatically replenished using a delivery service, allowing for comfortable meals to be served even while on the move.

[1621] An "imaging device" is a device that captures an image of a subject.

[1622] A "server" is a computer system that sends, receives, and processes data over a network.

[1623] "Information data" refers to data including images of ingredients and the user's health and emotional state.

[1624] "Analysis" is the process of breaking down input data and classifying, organizing, and evaluating it.

[1625] "Type of food ingredient" is information indicating the name and classification of food.

[1626] "Quantity" is information indicating the quantity or volume of ingredients.

[1627] "Status" is information indicating the freshness and storage state of the ingredients.

[1628] The "use by" date is information indicating the period during which the food ingredient can be safely used.

[1629] A "user terminal" is a device used by a user, such as a smartphone or tablet.

[1630] "Health status" is information relating to the user's physical condition and health.

[1631] "Emotional state" is information about the user's emotional and mental state.

[1632] "Suggestions" are recipes and meal recommendations generated based on the analysis results.

[1633] A "display means" is a method or device that visually shows the generated suggestions to the user.

[1634] "Delivery service" refers to a service that delivers food ingredients ordered online to a specified address.

[1635] The present invention provides a system for optimizing the dietary management and health status of passengers during long-distance travel in an autonomous vehicle and for suggesting recipes that take into consideration their emotions. This system is specifically implemented as follows.

[1636] System Configuration

[1637] 1. Imaging device (camera)

[1638] The system uses an imaging device built into the refrigerator inside the autonomous vehicle, which periodically captures images of the food items in the refrigerator and acquires the image data.

[1639] 2. Server

[1640] The image data is sent to a server where it is analyzed and the server uses deep learning techniques (e.g., TensorFlow, PyTorch) to identify the type, quantity, condition, and expiration date of the ingredients.

[1641] 3. User Device

[1642] Passengers use a user device such as a smartphone or tablet to access an application that inputs their health and emotional status. The application was developed using React Native.

[1643] 4. Emotion Engine

[1644] The health and emotional state data entered by the user into the device is analyzed by an emotion engine running on a cloud server (e.g., Microsoft Azure Cognitive Services).

[1645] 5. Generation AI

[1646] The generative AI (e.g., OpenAI GPT-4) on the server generates appropriate recipes based on the stored ingredient data and the user's health and emotional state.

[1647] 6. Display means

[1648] The generated recipe is sent to the user's smartphone or tablet in real time and displayed.

[1649] 7. Delivery Services

[1650] A delivery service will be used to automatically order and replenish missing ingredients at stops where the autonomous vehicle stops.

[1651] Operation overview

[1652] Program processing

[1653] The server receives the image data sent from the imaging device and analyzes the data using deep learning. As a result of the analysis, the type, quantity, condition, and expiration date of the ingredients are confirmed and stored in a database. The user inputs their health and emotional state using their user terminal. This input data is analyzed by the emotion engine and stored in the user's emotion database. The generation AI generates an optimal recipe based on the stored ingredient data and the user's health and emotional state, and sends it to the user terminal. The recipe is displayed in real time on the user terminal as a display method. If necessary ingredients are in short supply, an order is automatically issued to a delivery service, and they are replenished at the stop.

[1654] Specific examples

[1655] 1. User Input

[1656] A user types into a smartphone app, "I'm tired and would like a simple meal."

[1657] 2. Server Analysis

[1658] The server updates the health and emotion data and generates a recipe for "chicken and vegetable soup" that suits the user's physical condition.

[1659] 3. Sending and viewing recipes

[1660] The generated recipe is sent to the user terminal and displayed.

[1661] Generative AI model and prompts

[1662] Here's an example prompt that uses a generative AI model to generate a recipe:

[1663] User's current health status: Fatigue

[1664] User's current emotion: Wanting to relax

[1665] Usable ingredients: cabbage, carrots, chicken

[1666] Suggest a recipe to make:

[1667] This will enable passengers to enjoy comfortable meals and maintain their health while inside self-driving vehicles.

[1668] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1669] Step 1:

[1670] Photographing ingredients using an imaging device

[1671] The image capture device periodically captures images of food items in the refrigerator. The input is an image of the food items in the refrigerator, and the output is the captured image data. This image data is necessary to analyze the type, quantity, condition, and expiration date of the food items.

[1672] Step 2:

[1673] Sending image data to the server

[1674] Image data captured by an imaging device is sent to a server. The input is the captured image data, and the output is the image data uploaded to the server. This transmission prepares the server for subsequent analysis.

[1675] Step 3:

[1676] Image data analysis

[1677] The server analyzes the transmitted image data using deep learning technology (e.g., TensorFlow, PyTorch). The input is the image data uploaded to the server, and the output is the analysis results showing the type, quantity, condition, and expiration date of the ingredients. This analysis allows specific information about the ingredients to be registered in a database.

[1678] Step 4:

[1679] Input of the user's health and emotional state

[1680] Users use a smartphone or tablet to input their health and emotional status into the application. The input is information about the user's health and emotional status, and the output is the transmission of the input information. This allows data to be collected on the server for analysis by the emotion engine.

[1681] Step 5:

[1682] Analysis by emotion engine

[1683] The server uses an emotion engine (e.g., Microsoft Azure Cognitive Services) to analyze the user's health and emotional state. The input is the user's health and emotional state information, and the output is the analyzed emotion data. This data is stored in the user's emotion database.

[1684] Step 6:

[1685] Recipe generation using generative AI

[1686] The server's generative AI (e.g., OpenAI GPT-4) generates optimal recipes based on the stored ingredient data and the user's health and emotional state. The input is the stored ingredient data and the user's health and emotional state information, and the output is the generated recipe. This process suggests appropriate meals based on the user's state.

[1687] Step 7:

[1688] Sending and displaying recipes to user devices

[1689] The generated recipe is sent from the server to the user's smartphone or tablet. The input is the generated recipe, and the output is the recipe display on the user's device. The user can then cook a dish based on this recipe.

[1690] Step 8:

[1691] Use of delivery services

[1692] If a required ingredient is missing, the server automatically orders the missing ingredient from the delivery service. The input is the current ingredient status and the generated recipe, and the output is the order submission and replenishment of ingredients. This allows necessary ingredients to be replenished in a timely manner even while on the move, enabling comfortable meals to be served.

[1693] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1694] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1695] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1696] [Fourth embodiment]

[1697] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1698] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1699] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1700] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1701] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1702] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1703] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1704] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1705] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1706] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1707] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1708] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1709] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1710] The present invention relates to a system that reduces food waste and supports household health maintenance by using a camera built into a refrigerator, a server, a generation AI, a user terminal, and a food delivery service. This system is specifically implemented as follows.

[1711] System Configuration

[1712] In-fridge camera: The camera is placed inside the refrigerator and is used to take pictures of the ingredients added.

[1713] User device: Consists of a smartphone, tablet, etc., and is used by users to input their health status and receive and display recipes.

[1714] Server: A cloud system that receives and analyzes image data. This is where food ingredient data is generated.

[1715] Generative AI: Artificial intelligence that generates appropriate recipes based on health and ingredient data.

[1716] Food delivery service: A service with an automatic ordering function to make up for missing ingredients.

[1717] Operation overview

[1718] Register ingredients and send data

[1719] Device: When a user places a new ingredient in the refrigerator, a camera takes a picture of it, which undergoes initial analysis and is then sent to the server.

[1720] Server: Receives image data and performs image analysis using a deep learning module. This allows the type, quantity, condition, and expiration date of ingredients to be recognized. The analysis results are stored in a database.

[1721] User health management

[1722] User: Using a smartphone app, the user inputs their current health status, such as whether they are feeling unwell or on a diet.

[1723] Server: Updates the profile database based on the received health status data.

[1724] Recipe generation and distribution

[1725] Server: The AI ​​generates optimal recipes based on the stored ingredient data and the user's health status. If the user is not feeling well, it will generate recipes that are easy to digest, and under normal circumstances, it will generate recipes that prioritize ingredients that are close to their expiration date.

[1726] Server: Sends the generated recipe to the user's device.

[1727] User: A smartphone app displays recipe suggestions, such as "Today's recommended dish is cabbage and tomato salad."

[1728] Automatic ordering of missing ingredients

[1729] Server: Based on the proposed recipe, check the refrigerator for missing ingredients.

[1730] Server: Automatically generate and send orders to food delivery services to make up for missing ingredients.

[1731] User: Checks the status of a grocery delivery on a smartphone app, for example, with a notification that "cabbage and tomatoes are scheduled to be delivered tomorrow."

[1732] Specific examples

[1733] What to do when you are unwell

[1734] 1. User: The user types "I feel like I have a cold" into a smartphone app.

[1735] 2. Server: The server updates the health data and generates a porridge recipe suitable for the user's physical condition.

[1736] 3. Server: Sends the generated recipe to the user's device and notifies the user.

[1737] 4. User: The smartphone displays the message, "Let's make porridge today."

[1738] Utilizing ingredients that are close to their expiration date

[1739] 1. Terminal: The refrigerator camera takes a picture of the newly added milk and sends the image to the server.

[1740] 2. Server: Analyzes the image data and identifies the type of milk and its expiration date.

[1741] 3. Server: Updates the ingredient database and determines when the expiration date is approaching.

[1742] 4. Server: Generates recipes for dishes using milk (e.g., cream stew) and sends them to the user's device.

[1743] 5. User: The smartphone displays the message, "Let's make cream stew today."

[1744] The present invention makes it possible to reduce food waste and effectively support users in maintaining their health.

[1745] The processing flow will be explained below.

[1746] Specific processing of the program

[1747] Ingredient recognition and data transmission

[1748] Step 1:

[1749] Device: The camera built into the refrigerator takes pictures of the newly added ingredients by the user.

[1750] Step 2:

[1751] Terminal: Performs initial analysis of images captured by the camera and extracts features such as the shape, color, and tags of ingredients.

[1752] Step 3:

[1753] Terminal: Sends image data along with initial analysis data to a server via the internet.

[1754] Analysis of food ingredient data

[1755] Step 4:

[1756] Server: The received image data is input into a deep learning module to analyze the detailed type, quantity, and condition of the ingredients.

[1757] Step 5:

[1758] Server: Reads the expiration date printed on the food packaging using OCR (optical character recognition) technology.

[1759] Step 6:

[1760] Server: Stores the analysis results in a food ingredient database. Stored data includes type, quantity, expiration date, and condition.

[1761] User Health Check

[1762] Step 7:

[1763] User: Launches the smartphone app and enters their current health and physical condition.

[1764] Step 8:

[1765] Server: Updates the received health status data in the user profile database.

[1766] Recipe generation and distribution

[1767] Step 9:

[1768] Server: Generative AI generates optimal recipes based on ingredient data and health status data.

[1769] Step 10:

[1770] Server: Sends the generated recipe to the user's device.

[1771] Step 11:

[1772] User: A smartphone app displays a suggested recipe, for example, "Today's recommended dish is cabbage and tomato salad."

[1773] Automatic ordering of missing ingredients

[1774] Step 12:

[1775] Server: Based on the proposed recipe, check the amount of ingredients currently in the refrigerator and check for any missing ingredients.

[1776] Step 13:

[1777] Server: Creates an order list to make up for any missing ingredients and generates order data for the food delivery service API.

[1778] Step 14:

[1779] User: Checks the status of grocery delivery on a smartphone app. For example, it shows "cabbage and tomatoes are scheduled to be delivered tomorrow."

[1780] Example 1

[1781] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1782] There is a need for efficient methods to reduce food waste in the home and provide healthy meals. However, current refrigerator management systems make it difficult to accurately grasp the status of ingredients and tend to waste ingredients that are approaching their expiration date. Furthermore, they do not suggest optimal recipes based on health status, which makes it difficult to adequately manage health at home.

[1783] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1784] In this invention, the server

[1785] A means for taking an image of the food using a camera built into the refrigerator;

[1786] means for transmitting image data of ingredients to a server;

[1787] A means for analyzing image data in the server and recognizing the type, amount, condition and expiration date of ingredients;

[1788] a means for a user to input a health status;

[1789] A means for generating a recipe by the artificial intelligence based on the health status and the ingredient data and transmitting the recipe to the user terminal;

[1790] A display means for the generated recipe;

[1791] Includes.

[1792] This will automate food ingredient management and suggest optimal recipes based on health status, reducing food waste and helping to maintain household health.

[1793] A "camera built into a refrigerator" is a device that is installed inside the refrigerator and takes pictures of ingredients.

[1794] "Food ingredients" is a general term for food stored in a refrigerator.

[1795] "Image data" is digital data that contains visual information of ingredients photographed by a camera.

[1796] The "server" is a computer system installed on the cloud that analyzes image data of ingredients and provides services based on the user's health condition.

[1797] "Analysis" is the process of recognizing the type, quantity, condition, and expiration date of ingredients based on image data.

[1798] A "deep learning module" is a type of artificial intelligence technology used to analyze images of ingredients, and utilizes a multi-layer neural network.

[1799] A "user terminal" is a device that allows a user to input their health status and receive notifications and recipes from the system, and includes smartphones, tablets, etc.

[1800] "Health condition data" is information about the current health condition entered by the user.

[1801] "Generative AI" is artificial intelligence that generates optimal recipes based on the user's health condition and ingredient data.

[1802] A "recipe" is a document that describes the steps for preparing a dish using specific ingredients.

[1803] A "food delivery service" is a commercial service that automatically orders and delivers food ingredients needed by users.

[1804] A "database" is a system for systematically storing analysis results and user health status data.

[1805] An "order" is request data for ordering the necessary ingredients from the ingredient delivery service.

[1806] The "profile database" is a database for storing a user's past health condition data and food consumption history.

[1807] This invention is a system for supporting household food waste reduction and health maintenance. The system's main components are a camera built into the refrigerator, a server, a generative AI model, a user terminal, and a food delivery service.

[1808] Refrigerator camera

[1809] The camera built into the refrigerator, which is the terminal, is a device that takes pictures of ingredients added to the refrigerator. A high-resolution camera is used and is positioned so that it covers the entire interior of the refrigerator. When a user puts new ingredients into the refrigerator, the camera automatically detects this and takes a picture. This picture is sent to a server on the cloud via Wi-Fi. A secure protocol (e.g., HTTPS) is used for transmission.

[1810] Server and Image Analysis

[1811] The server analyzes the received image data using a deep learning module. For example, it uses a deep learning library such as TensorFlow to identify the type, quantity, condition (e.g., fresh, spoiled), and expiration date of the ingredients. The results of this analysis are stored in a database (e.g., MySQL or MongoDB). The stored data includes details such as the ingredient name, quantity, expiration date, and the date and time of addition.

[1812] User health management

[1813] Using a smartphone app, users input their current health status (e.g., feeling a bit under the weather, dieting, high stress) using text fields and pull-down menus. This information is sent in real time to a server, which then updates the user's profile database. This profile also includes past health data and food consumption history.

[1814] Recipe generation and distribution using generative AI

[1815] The server retrieves the stored ingredient data and the user's health status data and generates optimal recipes using a generative AI model. Examples of generative AI models include OpenAI's GPT-3 and other custom models. The generative AI takes into account the user's health status and available ingredient information to create healthy recipes. For example, it might recommend porridge when feeling unwell or a low-calorie salad when on a diet. This generated recipe is sent to the user's device and displayed on a smartphone app.

[1816] Automatic ordering of missing ingredients

[1817] The server checks the refrigerator for missing ingredients based on the proposed recipe. If any ingredients are found to be missing, it automatically calls the food delivery service's API and generates an order. The order includes credit card information and delivery address. Delivery information from the food delivery service is sent to the user's device. The user can check the delivery status on their smartphone app.

[1818] Examples of concrete examples and prompts

[1819] As a concrete example, let's look at how to respond when you're feeling unwell. When a user enters "I'm feeling a bit sick" into a smartphone app, the server updates the health data and generates a porridge recipe suited to the user's physical condition. The generated recipe is sent to the user's device, and the smartphone displays a message saying, "Let's try making porridge today."

[1820] We will also give a concrete example of how to utilize ingredients that are approaching their expiration date. The camera in the refrigerator takes a picture of the newly added milk and sends the image to the server. The server analyzes the image data and recognizes the type of milk and its expiration date. It updates the ingredient database and determines that the expiration date is approaching. A recipe for a dish that uses the milk (for example, cream stew) is generated and sent to the user's device. The user is then prompted on their smartphone, "Let's try making cream stew today."

[1821] Examples of prompts:

[1822] "When you put new food in the refrigerator, the camera automatically takes a picture and sends it to the server."

[1823] "When a user inputs information about when they are feeling unwell into a smartphone app, an appropriate recipe is generated based on that information."

[1824] "A function that automatically orders missing ingredients from a delivery service based on the suggested recipe."

[1825] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1826] Step 1: Take a photo of the food

[1827] When a user adds new ingredients to the refrigerator, the device's built-in camera automatically takes a picture of the ingredients.

[1828] Input: New ingredients added to the fridge

[1829] Output: Image data of ingredients

[1830] The images are high resolution and the cameras are positioned to cover the entire interior of the refrigerator. Once the images are captured, the image data is used in the next step.

[1831] Step 2: Sending image data

[1832] The device sends the captured image data to a cloud server via Wi-Fi using a secure protocol (e.g., HTTPS).

[1833] Input: Image data of the photographed food

[1834] Output: Image data sent to the server

[1835] This allows the image data to reach the server for analysis.

[1836] Step 3: Analyzing the image data

[1837] The server analyzes the received image data using a deep learning module (e.g., TensorFlow) to identify the type, quantity, condition, and expiration date of the ingredients.

[1838] Input: Image data of the ingredients sent

[1839] Output: Type, quantity, condition and expiration date of recognized ingredients

[1840] The analysis results include details such as ingredient name, quantity, expiration date, and date and time of addition, which are stored in a database.

[1841] Step 4: Save the ingredients data

[1842] The server stores the analysis results in a database (e.g., MySQL or MongoDB).

[1843] Input: Detailed information about the recognized ingredients

[1844] Output: Saved ingredient data

[1845] Based on the stored data, the ingredients are managed in the next step.

[1846] Step 5: Enter your health status

[1847] Users enter their current health status using a smartphone app, which provides text fields and pull-down menus for input.

[1848] Input: User's health status information (e.g., feeling a bit under the weather, dieting, high stress)

[1849] Output: Health status data sent to the server

[1850] The submitted data is added to the user's profile database.

[1851] Step 6: Update your profile

[1852] The server then updates the user's profile database based on the received health status data, which includes past health status data and food consumption history.

[1853] Input: Health status data sent by the user

[1854] Output: Updated profile data

[1855] This lays the foundation for generating optimal recipes based on the user's health condition.

[1856] Step 7: Generative AI generates recipes

[1857] The server retrieves the stored ingredient data and the user's health status data and generates optimal recipes using generative AI models, including OpenAI's GPT-3 and other custom models.

[1858] Input: Stored food and health data

[1859] Output: The generated recipe

[1860] The generative AI uses this data to create recipes, taking into account the patient's health status and available ingredients. For example, it might recommend porridge if you're feeling unwell, or a low-calorie salad if you're on a diet.

[1861] Step 8: Submit your recipe

[1862] The server then sends the generated recipe to the user's device, which includes the ingredients, cooking instructions, and nutritional information.

[1863] Input: Generated recipe

[1864] Output: Recipe sent to user's device

[1865] This allows the user to receive and check the recipe.

[1866] Step 9: View the recipe

[1867] The user checks the suggested recipes on the smartphone app, which displays something like, "Today's recommended dish is cabbage and tomato salad."

[1868] Input: Recipe sent from the server

[1869] Output: Recipe displayed on smartphone app

[1870] This allows the user to easily check the suggested menu.

[1871] Step 10: Check for missing ingredients

[1872] The server checks the refrigerator for missing ingredients based on the proposed recipe.

[1873] Input: Proposed recipe and current ingredient data

[1874] Output: List of missing ingredients

[1875] If any ingredients are found to be missing, an automatic order will be placed in the next step.

[1876] Step 11: Automated Orders

[1877] The server automatically generates and sends an order to a food delivery service for any missing ingredients, along with credit card information and a delivery address.

[1878] Input: Missing ingredients list

[1879] Output: Order data sent to grocery delivery service

[1880] This will automatically replenish any missing ingredients.

[1881] Step 12: Delivery Information Notification

[1882] The server receives delivery information from the food delivery service and notifies the user terminal of the delivery information, including the scheduled delivery date and time and the delivery status.

[1883] Input: Delivery information from food delivery service

[1884] Output: Delivery information sent to the user's device

[1885] The user checks the status of their grocery delivery on a smartphone app, which displays a notification such as "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[1886] (Application example 1)

[1887] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1888] Efficient and accurate food management is essential to simultaneously reduce food waste at home and maintain good health. However, current systems make food management cumbersome, and they rarely suggest meals that are particularly suited to a person's health or automatically order ingredients that are in short supply. This leads to food waste and makes it difficult for users to maintain their health. Therefore, a system that effectively supports reducing food waste and maintaining good health is needed.

[1889] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1890] In this invention, the server includes means for taking images of ingredients with a camera built into the refrigerator, means for transmitting image data of ingredients to the server, means for analyzing the image data in the server and recognizing the type, amount, condition, and expiration date of ingredients, means for the user to input health status, means for generating recipes based on the health status and ingredient data and transmitting them to the user terminal, means for displaying the generated recipe, means for automatically ordering missing ingredients, means for notifying the user terminal of the delivery status of the automatically ordered ingredients, and means for using generation AI to generate optimal menus based on the health status and tracked ingredient information. This effectively reduces food waste and maintains health, making it easy for users to eat healthy meals.

[1891] A "camera built into a refrigerator" is a camera built into the refrigerator that takes pictures of food stored inside the refrigerator.

[1892] The "means for transmitting image data of ingredients to the server" refers to a communication device and protocol for transmitting image data captured by a camera built into the refrigerator to the server via the Internet.

[1893] The "means for analyzing image data and recognizing the type, quantity, condition and expiration date of ingredients" refers to a program and process that analyzes image data sent to the server and automatically determines the type, quantity, condition and expiration date of ingredients.

[1894] "Means for users to input their health status" refers to the interface and data input device that allows users to input and register their own health status.

[1895] The "means for generating a recipe and transmitting it to a user terminal" refers to a system and program for generating an appropriate recipe based on health status and ingredient data and transmitting the recipe to the user's terminal.

[1896] The "means for displaying the generated recipe" refers to a display and application for displaying the generated recipe on the user terminal.

[1897] The "means for automatically ordering missing ingredients" refers to a system and program that automatically orders missing ingredients from a food delivery service when ingredients required for a proposed recipe are missing.

[1898] The "means for notifying the user terminal of the delivery status of ordered ingredients" refers to a system and program for automatically notifying the user terminal of the delivery status of ordered ingredients.

[1899] "Generative AI" refers to models and algorithms for generating recipes using artificial intelligence based on specific tasks.

[1900] The "means for generating an optimal menu" refers to a system and program for generating an optimal menu for a user based on the user's health condition and tracked ingredient information.

[1901] This invention is a system used in a home refrigerator that simultaneously reduces food waste and maintains health. This system uses a program and a cloud server to manage ingredients, create recipes based on health information, and automatically order ingredients when they are in short supply. Specifically, the system is configured as follows:

[1902] System Configuration

[1903] 1. In-fridge camera:

[1904] A camera placed inside the refrigerator takes pictures of the ingredients, either a USB camera or a built-in camera.

[1905] 2. Server:

[1906] It functions as part of a cloud server that receives and analyzes image data of ingredients. The server is equipped with deep learning models (e.g., TensorFlow or PyTorch) that are used to recognize the type, quantity, condition, and expiration date of ingredients.

[1907] 3. User Device:

[1908] It consists of a smartphone or tablet that allows users to input their health status and receive and display recipes. The user interface includes a form for accepting health status input and a function for displaying generated recipes. Specifically, an iOS or Android application is used.

[1909] 4. Generation AI:

[1910] It is an artificial intelligence that generates appropriate recipes based on health and ingredient data, for example, GPT-4 and other generative models are used to provide recipes that suit the user's needs.

[1911] 5. Food delivery service:

[1912] This service has an automatic ordering function to make up for missing ingredients, and uses an API to generate orders and track the delivery status of ingredients.

[1913] Operation overview

[1914] 1. Register ingredients and send data:

[1915] The camera inside the refrigerator takes a picture of the new food item and sends the image to the server. The server receives the image data and analyzes it using a deep learning module. The analysis results include the type, quantity, condition, and expiration date of the food item, and these are stored in a database.

[1916] 2. User Health Management:

[1917] Users use a smartphone app to input their current health status, such as whether they are feeling unwell or on a diet. The server updates the profile database based on the received health status data.

[1918] 3. Recipe generation and distribution:

[1919] The server uses the AI ​​to generate optimal recipes based on the stored ingredient data and the user's health status data. The generated recipes are sent to the user's device, and the suggested recipes are displayed on the smartphone app.

[1920] 4. Automatic ordering of missing ingredients:

[1921] The server checks the refrigerator for missing ingredients based on the proposed recipe, automatically generates and sends an order to the food delivery service for the missing ingredients, and the user can check the status of the food delivery on their smartphone app.

[1922] Specific examples

[1923] What to do when you are unwell

[1924] 1. The user types "I feel like I have a cold" into a smartphone app.

[1925] 2. The server updates the health data and generates a porridge recipe suitable for the user's physical condition.

[1926] 3. The generated recipe is sent to the user's device and notified to the user.

[1927] 4. The user will see a message on their smartphone saying, "Let's make porridge today."

[1928] Prompt Sentence Examples

[1929] The prompt is:

[1930] "The user's current health condition is as follows: poor health, desires easy-to-digest diet. The ingredients in the refrigerator are: tomatoes, cabbage, chicken. Please generate the optimal recipe based on these conditions."

[1931] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1932] Step 1:

[1933] A camera inside the refrigerator takes a picture of the newly added ingredient.

[1934] Input: Ingredients in the refrigerator

[1935] Output: Image data of ingredients

[1936] Specific operation: When the user puts ingredients into the refrigerator, the camera captures the moment and saves it as image data.

[1937] Step 2:

[1938] The server receives image data of the ingredients and analyzes the images to recognize the type, amount, condition, and expiration date of the ingredients.

[1939] Input: Image data of ingredients

[1940] Output: Analyzed food data (type, quantity, condition, expiration date)

[1941] How it works: Image data is sent over the internet to a server, which then analyzes the image using a deep learning model (e.g., TensorFlow). The analysis results are stored in a database.

[1942] Step 3:

[1943] Users input their own health status using a smartphone app.

[1944] Input: User's health status data (e.g., cold, desire for easy-to-digest meals, etc.)

[1945] Output: Updated health profile data

[1946] Specific operation: The user enters information such as "feeling a bit sick" or "on a diet" into the app. The entered data is sent to the server, and the profile database is updated.

[1947] Step 4:

[1948] Based on the ingredient data and health status data stored on the server, the generative AI generates the optimal recipe.

[1949] Input: Analyzed food ingredient data, health profile data

[1950] Output: The generated recipe

[1951] How it works: Generative AI (e.g., GPT-4) generates optimal recipes based on input data. For example, if you are feeling unwell, it will generate a recipe for easy-to-digest porridge.

[1952] Step 5:

[1953] The server sends the generated recipe to the user's device and displays it within the app.

[1954] Input: Generated recipe

[1955] Output: The recipe displayed on the user's terminal

[1956] Specific operation: The generated recipe is sent to the user's smartphone app and displayed as a notification within the app, such as "Today's recommended dish is porridge."

[1957] Step 6:

[1958] Based on the proposed recipe, the server checks the refrigerator for any missing ingredients and automatically generates and sends an order to the food delivery service.

[1959] Input: Generated recipe, analyzed food data in the refrigerator

[1960] Output: Order request to food delivery service

[1961] Specific operation: Checks the ingredients required for the generated recipe, and if there are any missing from the refrigerator, automatically sends an order request to the food delivery service. The order is generated using an API.

[1962] Step 7:

[1963] A user checks the status of their grocery delivery on a smartphone app.

[1964] Input: Delivery status data from food delivery service

[1965] Output: Delivery status displayed on the smartphone app (e.g., "Cabbage and tomatoes are scheduled for delivery tomorrow")

[1966] Specific operation: The server notifies the user's device of the delivery status obtained from the food delivery service and displays the status within the app.

[1967] By going through each step in this way, a system will be built that effectively reduces food waste and maintains health.

[1968] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1969] The present invention relates to a system that reduces food waste and proposes recipes that take into consideration the health of the household and the emotions of the user, using a camera built into a refrigerator, a server, a generative AI, a user terminal, an emotion engine, and a food delivery service. This system is specifically implemented as follows.

[1970] System Configuration

[1971] In-fridge camera: The camera is placed inside the refrigerator and is used to take pictures of the ingredients added.

[1972] User device: Consists of a smartphone, tablet, etc., and is used by users to input their health status and emotions, and receive and display recipes.

[1973] Server: A cloud system that receives and analyzes image data. This is where food ingredient data is generated.

[1974] Generative AI: An artificial intelligence system that generates appropriate recipes based on health status and ingredient data, as well as emotional data recognized by an emotion engine.

[1975] Emotion engine: A system that recognizes user emotions from user input data and other data sources and provides that information to a server.

[1976] Food delivery service: A service with an automatic ordering function to make up for missing ingredients.

[1977] Operation overview

[1978] Register ingredients and send data

[1979] Device: When a user places a new ingredient in the refrigerator, a camera takes a picture of it, which undergoes initial analysis and is then sent to the server.

[1980] Server: Receives image data and performs image analysis using a deep learning module. This allows the type, quantity, condition, and expiration date of ingredients to be recognized. The analysis results are stored in a database.

[1981] User health management and emotion recognition

[1982] Users: Use a smartphone app to input their current health status and emotions, including stress, anxiety, and joy.

[1983] Server: Analyzes the user's emotions from the input data using the emotion engine. The analysis results are stored in the user's emotion database.

[1984] Recipe generation and distribution

[1985] Server: The generative AI generates optimal recipes based on the stored ingredient data, the user's health status, and emotional data. For example, if the user is feeling stressed, it will suggest recipes for relaxing herbal teas or easy-to-make sweets.

[1986] Server: Sends the generated recipe to the user's device.

[1987] User: Viewing recipe suggestions on a smartphone app, such as "Today's recommended dish is cabbage and tomato salad."

[1988] Automatic ordering of missing ingredients

[1989] Server: Based on the proposed recipe, check the amount of ingredients currently in the refrigerator and check for any missing ingredients.

[1990] Server: Creates an order list to make up for any missing ingredients and generates order data for the food delivery service API.

[1991] User: Checks the status of a grocery delivery on a smartphone app. For example, it might say, "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[1992] Specific examples

[1993] Dealing with poor health and emotional frustration

[1994] 1. User: The user enters "I feel like I have a cold" and "I'm stressed" into a smartphone app.

[1995] 2. Server: The server updates the health and emotional data and generates recipes for porridge and herbal tea for stress relief that are suited to the user's physical condition.

[1996] 3. Server: Sends the generated recipe to the user's device and notifies the user.

[1997] 4. User: The smartphone displays the message, "Let's make porridge and herbal tea today."

[1998] Utilizing ingredients that are close to their expiration date and making suggestions based on emotions

[1999] 1. Terminal: The refrigerator camera takes a picture of the newly added milk and sends the image to the server.

[2000] 2. Server: Analyzes the image data and identifies the type of milk and its expiration date.

[2001] 3. Server: Updates the ingredient database and determines when the expiration date is approaching.

[2002] 4. Server: Based on the user's emotional data, it generates recipes for milk-based dishes (e.g., cream stew) and soothing desserts.

[2003] 5. Server: Sends the recipe to the user device.

[2004] 6. User: "Today, let's make cream stew and chocolate mousse" appears on the smartphone.

[2005] The present invention makes it possible to reduce food waste and effectively support users in maintaining their health and managing their emotions.

[2006] The processing flow will be explained below.

[2007] Specific processing of the program

[2008] Ingredient recognition and data transmission

[2009] Step 1:

[2010] Device: The camera built into the refrigerator takes pictures of the newly added ingredients by the user.

[2011] Step 2:

[2012] Terminal: Performs initial analysis of images captured by the camera and extracts features such as the shape, color, and tags of ingredients.

[2013] Step 3:

[2014] Terminal: Sends image data along with initial analysis data to a server via the internet.

[2015] Analysis of food ingredient data

[2016] Step 4:

[2017] Server: The received image data is input into a deep learning module to analyze the detailed type, quantity, and condition of the ingredients.

[2018] Step 5:

[2019] Server: Reads the expiration date printed on the food packaging using OCR (optical character recognition) technology.

[2020] Step 6:

[2021] Server: Stores the analysis results in a food ingredient database. Stored data includes type, quantity, expiration date, and condition.

[2022] User health management and emotion recognition

[2023] Step 7:

[2024] User: Launches the smartphone app and inputs their current health condition and emotions. For example, they input that their physical condition is "feeling a bit like a cold" and their emotion is "stressed."

[2025] Step 8:

[2026] Server: Updates the received health status data in the profile database.

[2027] Step 9:

[2028] Server: Analyzes the user's emotions from the input data using the emotion engine. The analysis results are stored in the user's emotion database.

[2029] Recipe generation and distribution

[2030] Step 10:

[2031] Server: The generative AI generates optimal recipes based on the stored ingredient data, the user's health status, and emotional data. For example, if a user is feeling unwell and stressed, it will suggest a recipe for porridge and herbal tea that helps relieve stress.

[2032] Step 11:

[2033] Server: Sends the generated recipe to the user's device.

[2034] Step 12:

[2035] User: A smartphone app displays a suggested recipe, for example, "Today, try making porridge and herbal tea."

[2036] Automatic ordering of missing ingredients

[2037] Step 13:

[2038] Server: Based on the proposed recipe, check the amount of ingredients currently in the refrigerator and check for any missing ingredients.

[2039] Step 14:

[2040] Server: Creates an order list to make up for any missing ingredients and generates order data for the food delivery service API.

[2041] Step 15:

[2042] User: Checks the status of a grocery delivery on a smartphone app. For example, it might say, "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[2043] Specific examples

[2044] Dealing with poor health and emotional frustration

[2045] Step 1:

[2046] User: The user types "feeling a bit under the weather" and "feeling stressed" into a smartphone app.

[2047] Step 2:

[2048] Server: The server updates the health and emotional data of the user and generates recipes for porridge and herbal tea for stress relief that are suited to the user's physical condition.

[2049] Step 3:

[2050] Server: Sends the generated recipe to the user's device and notifies the user.

[2051] Step 4:

[2052] User: "Let's make porridge and herbal tea today" appears on their phone.

[2053] Utilizing ingredients that are close to their expiration date and making suggestions based on emotions

[2054] Step 1:

[2055] Device: The camera in the refrigerator takes a picture of the newly added milk and sends the image to the server.

[2056] Step 2:

[2057] Server: Analyzes image data and identifies the type of milk and its expiration date.

[2058] Step 3:

[2059] Server: Updates the food ingredient database and determines when the expiration date is approaching.

[2060] Step 4:

[2061] User: The user types "I'm tired" into a smartphone app.

[2062] Step 5:

[2063] Server: Generates recipes for milk-based dishes (e.g., cream stew) and soothing desserts (e.g., chocolate mousse) based on ingredient data and emotion data.

[2064] Step 6:

[2065] Server: Sends recipes to user devices.

[2066] Step 7:

[2067] User: "Today, let's make cream stew and chocolate mousse" appears on their phone.

[2068] Example 2

[2069] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2070] In modern households, it is difficult to manage food ingredients, and many ingredients pass their expiration date. Health management is also difficult because meal suggestions do not take into account the user's health and emotional state. As a result, food waste and unhealthy eating habits have become problems.

[2071] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2072] In this invention, the server includes means for acquiring images of ingredients using an image acquisition device built into the refrigerator, means for transmitting image data of the ingredients to the server, means for analyzing the image data in the server and recognizing the type, amount, condition, and expiration date of the ingredients, means for the user to input their health condition and emotions, means for generating recipes using a generative AI model based on the health condition, emotion data, and ingredient data and transmitting the recipes to the user terminal, and means for displaying the generated recipes. This makes it possible to reduce food waste and suggest meals that take into consideration the user's health management and emotions.

[2073] An "image capture device built into a refrigerator" is a combination of hardware and software that is installed inside the refrigerator and is used to capture images of ingredients and acquire the data.

[2074] A "server" is a computer system that analyzes the received image data, health condition data, and emotion data and performs the necessary processing.

[2075] The "means for transmitting image data" is a communication means for transmitting image data of ingredients captured by the image capturing device in the refrigerator to the server.

[2076] "Means for analyzing image data" refers to software and algorithms that analyze the image data received by the server using a deep learning module or the like to recognize the type, quantity, condition, and expiration date of ingredients.

[2077] The "means for inputting health status" refers to a user terminal equipped with an interface for the user to input their own health status.

[2078] The "means for inputting emotions" refers to a user terminal equipped with an interface for the user to input their own emotions.

[2079] "Means for generating recipes using a generative artificial intelligence model" refers to software and algorithms for generating optimal recipes using a generative AI model based on health data, emotional data, and ingredient data.

[2080] A "user terminal" is a communication terminal device such as a smartphone or tablet that allows a user to check the generated recipe.

[2081] The "means for displaying the generated recipe" refers to an interface and software for visually displaying the generated recipe on the user terminal.

[2082] "Means for generating and sending orders to food delivery services" refers to software and communication means for automatically generating and sending orders using the food delivery service's API when necessary ingredients are in short supply.

[2083] "Initial processing means" refers to software and algorithms that extract features from the image data captured by the camera and perform preliminary analysis on the data before sending it to the server.

[2084] This invention is a system for reducing food waste, maintaining the user's health, and suggesting recipes based on emotions. This system is realized using an image capture device built into a refrigerator, a server, a generative AI model, a user terminal, an emotion engine, and a food delivery service. The operation of the system is as follows.

[2085] System Configuration

[2086] Refrigerator image capture device: This is a device that is installed inside the refrigerator to capture images of added ingredients.

[2087] User device: This consists of a smartphone or tablet, and is used by the user to input their health status and emotions, and to receive and display the generated recipes. The user device contains the interface and software.

[2088] Server: This is a computer system that receives and analyzes image data. The server uses deep learning modules (e.g., TensorFlow or PyTorch), databases (e.g., MySQL or PostgreSQL), and emotion engines (e.g., Emotion API).

[2089] Generative AI model: This is an artificial intelligence model that generates optimal recipes based on health status, emotional data, and ingredient data. For example, GPT-4 is used as a generative AI model.

[2090] Emotion engine: This is a system that recognizes the user's emotions and provides that information to the server. Specifically, it uses the Emotion API.

[2091] Food delivery service: A service that automatically generates order data when necessary ingredients are in short supply. For example, it uses APIs such as Amazon Fresh and Instacart.

[2092] Operation overview

[2093] This system operates in the following manner.

[2094] 1. Register ingredients and send data

[2095] When a user places new ingredients in the refrigerator, an image capture device inside the refrigerator takes a picture. The image data is sent to a server, which analyzes it using a deep learning module. This identifies the type, quantity, condition, and expiration date of the ingredients. The analysis results are then stored in a database.

[2096] 2. User Health Management and Emotion Recognition

[2097] The user inputs their current health status and emotions using a smartphone app. The server analyzes this data using an emotion engine and stores the results in an emotion database.

[2098] 3. Recipe generation and distribution

[2099] The server uses a generative AI model based on the stored ingredient data, the user's health status data, and emotional data to generate the optimal recipe. For example, the prompt text could read, "The user is feeling a bit under the weather and under a lot of stress. The ingredients available are cabbage, tomato, and chicken. Please suggest a dish that will have a relaxing effect using these ingredients."

[2100] 4. Automatic ordering of missing ingredients

[2101] Based on the proposed recipe, the system checks the amount of ingredients currently in the refrigerator and checks for any missing ingredients. If any ingredients are missing, it creates an order list and automatically sends the order data to the food delivery service's API.

[2102] Specific examples

[2103] 1. Dealing with poor health and emotional frustration

[2104] A user enters "feeling a bit like a cold" and "feeling stressed" into a smartphone app and submits the message.

[2105] Based on the data received by the server, a generative AI model is used to generate recipes for porridge and stress-relieving herbal tea.

[2106] The generated recipe is sent to the user terminal and notified to the user.

[2107] The user sees a message on their smartphone saying, "Let's make porridge and herbal tea today."

[2108] 2. Utilizing ingredients that are close to their expiration date and making suggestions based on emotions

[2109] The image capture device in the refrigerator takes a picture of the newly added milk and sends the image to the server.

[2110] The server analyzes the image data and identifies the type of milk and its expiration date.

[2111] The server updates the food ingredient database and determines when the expiration date is approaching.

[2112] The server generates recipes for cream stew using milk and a soothing dessert, for example, based on the user's emotional data.

[2113] The generated recipe is sent to the user terminal.

[2114] The user sees on their smartphone a message saying, "Today, let's make cream stew and chocolate mousse."

[2115] The present invention makes it possible to reduce food waste, maintain the user's health, and suggest meals that suit the user's emotions.

[2116] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2117] Step 1:

[2118] Food Registration

[2119] Terminal: When a user adds a new ingredient to the refrigerator, an image capture device in the refrigerator takes an image of the ingredient. The captured image data is sent from the terminal to the server.

[2120] Input: Image data of the newly added ingredient.

[2121] Output: Image data sent to the server.

[2122] Step 2:

[2123] Initial data analysis

[2124] Server: The received image data is analyzed using a deep learning module (e.g., TensorFlow or PyTorch). This analysis identifies the type, quantity, condition, and expiration date of the ingredients. The analysis results are stored in a database.

[2125] Input: Image data sent to the server.

[2126] Output: Data on the type, quantity, condition and expiry date of the recognized ingredients.

[2127] Step 3:

[2128] Health and Emotion Input

[2129] User: The user uses the smartphone app to input their own health condition and emotions. They select a condition such as "feeling a bit like a cold" or "feeling stressed" from the options and press the send button.

[2130] Input: Health and emotional information entered by the user into the app.

[2131] Output: Health and emotion data sent to the server.

[2132] Step 4:

[2133] Emotional Data Analysis

[2134] Server: Analyzes the received health status data and emotion data using an emotion engine (e.g., Emotion API). The analysis results are stored in an emotion database.

[2135] Input: User-entered health and emotion data.

[2136] Output: Stored sentiment analysis results.

[2137] Step 5:

[2138] Recipe Generation

[2139] Server: Generates optimal recipes using a generative AI model (such as GPT-4) based on the stored ingredient data, health status data, and emotion data. The prompt text is entered as follows: "The user is feeling a bit under the weather and is under a lot of stress. The ingredients available are cabbage, tomato, and chicken. Please suggest a dish that will have a relaxing effect using these ingredients."

[2140] Input: Food ingredient data, health condition data, emotion data, prompt sentence.

[2141] Output: The generated recipe data.

[2142] Step 6:

[2143] Recipe distribution

[2144] Server: Sends the generated recipe to the user's device.

[2145] Input: The generated recipe data.

[2146] Output: The recipe sent to the user's device.

[2147] Step 7:

[2148] Recipe Display

[2149] User: The user checks the suggested recipes on the smartphone app. When the user opens the app, the suggested recipes are displayed.

[2150] Input: Recipe data sent from the server.

[2151] Output: The recipe displayed in the app.

[2152] Step 8:

[2153] Ingredients checking and automatic ordering

[2154] Server: Based on the proposed recipe, check the database for the amount of ingredients currently in the refrigerator. Check the required amount to see if any ingredients are missing.

[2155] Input: Current ingredient data and suggested recipe data.

[2156] Output: A list of missing ingredients.

[2157] Step 9:

[2158] Automatic Order Generation

[2159] Server: Based on the list of missing ingredients, order data is automatically generated and sent using the food delivery service's API.

[2160] Input: Missing ingredient list.

[2161] Output: Order data for food delivery service.

[2162] Step 10:

[2163] Delivery status notifications

[2164] Server: After the order is completed, check the delivery status and send the delivery information to the user's device.

[2165] Input: Delivery status information from a grocery delivery service.

[2166] Output: Delivery status notification sent to user device.

[2167] Step 11:

[2168] Delivery status display

[2169] User: The user checks the delivery status on their smartphone app, for example, "Cabbage and tomatoes are scheduled to be delivered tomorrow."

[2170] Input: Delivery status notification.

[2171] Output: Delivery status information displayed in the app.

[2172] This will make it possible to reduce food waste, maintain health, and suggest recipes that respond to emotions.

[2173] (Application example 2)

[2174] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2175] Managing meals and maintaining one's health is important for passengers to stay comfortable in self-driving vehicles during long-distance travel. However, it is difficult to store appropriate ingredients, supply new ingredients, and suggest meals that take into account the passenger's health and emotions while traveling. Therefore, there is a need to provide a system that reduces food waste and makes meal suggestions that take into account the passenger's health and emotions while traveling in self-driving vehicles during long-distance travel.

[2176] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2177] In this invention, the server includes a means for capturing images of ingredients using an imaging device built into the refrigerator, a means for transmitting the ingredient information data to the server, a means for analyzing the information data in the server and recognizing the type, quantity, condition, and expiration date of the ingredients, a means for the user to input their health and emotional state, a means for generating suggestions based on the health and ingredient data and the emotional state data and transmitting the suggestions to the user terminal, and a means for displaying the generated suggestions. This automates meal management in an autonomous vehicle and enables meal suggestions based on the passenger's health and emotional state. Furthermore, if ingredients run low, new ingredients can be automatically replenished using a delivery service, allowing for comfortable meals to be served even while on the move.

[2178] An "imaging device" is a device that captures an image of a subject.

[2179] A "server" is a computer system that sends, receives, and processes data over a network.

[2180] "Information data" refers to data including images of ingredients and the user's health and emotional state.

[2181] "Analysis" is the process of breaking down input data and classifying, organizing, and evaluating it.

[2182] "Type of food ingredient" is information indicating the name and classification of food.

[2183] "Quantity" is information indicating the quantity or volume of ingredients.

[2184] "Status" is information indicating the freshness and storage state of the ingredients.

[2185] The "use by" date is information indicating the period during which the food ingredient can be safely used.

[2186] A "user terminal" is a device used by a user, such as a smartphone or tablet.

[2187] "Health status" is information relating to the user's physical condition and health.

[2188] "Emotional state" is information about the user's emotional and mental state.

[2189] "Suggestions" are recipes and meal recommendations generated based on the analysis results.

[2190] A "display means" is a method or device that visually shows the generated suggestions to the user.

[2191] "Delivery service" refers to a service that delivers food ingredients ordered online to a specified address.

[2192] The present invention provides a system for optimizing the dietary management and health status of passengers during long-distance travel in an autonomous vehicle and for suggesting recipes that take into consideration their emotions. This system is specifically implemented as follows.

[2193] System Configuration

[2194] 1. Imaging device (camera)

[2195] The system uses an imaging device built into the refrigerator inside the autonomous vehicle, which periodically captures images of the food items in the refrigerator and acquires the image data.

[2196] 2. Server

[2197] The image data is sent to a server where it is analyzed and the server uses deep learning techniques (e.g., TensorFlow, PyTorch) to identify the type, quantity, condition, and expiration date of the ingredients.

[2198] 3. User Device

[2199] Passengers use a user device such as a smartphone or tablet to access an application that inputs their health and emotional status. The application was developed using React Native.

[2200] 4. Emotion Engine

[2201] The health and emotional state data entered by the user into the device is analyzed by an emotion engine running on a cloud server (e.g., Microsoft Azure Cognitive Services).

[2202] 5. Generation AI

[2203] The generative AI (e.g., OpenAI GPT-4) on the server generates appropriate recipes based on the stored ingredient data and the user's health and emotional state.

[2204] 6. Display means

[2205] The generated recipe is sent to the user's smartphone or tablet in real time and displayed.

[2206] 7. Delivery Services

[2207] A delivery service will be used to automatically order and replenish missing ingredients at stops where the autonomous vehicle stops.

[2208] Operation overview

[2209] Program processing

[2210] The server receives the image data sent from the imaging device and analyzes the data using deep learning. As a result of the analysis, the type, quantity, condition, and expiration date of the ingredients are confirmed and stored in a database. The user inputs their health and emotional state using their user terminal. This input data is analyzed by the emotion engine and stored in the user's emotion database. The generation AI generates an optimal recipe based on the stored ingredient data and the user's health and emotional state, and sends it to the user terminal. The recipe is displayed in real time on the user terminal as a display method. If necessary ingredients are in short supply, an order is automatically issued to a delivery service, and they are replenished at the stop.

[2211] Specific examples

[2212] 1. User Input

[2213] A user types into a smartphone app, "I'm tired and would like a simple meal."

[2214] 2. Server Analysis

[2215] The server updates the health and emotion data and generates a recipe for "chicken and vegetable soup" that suits the user's physical condition.

[2216] 3. Sending and viewing recipes

[2217] The generated recipe is sent to the user terminal and displayed.

[2218] Generative AI model and prompts

[2219] Here's an example prompt that uses a generative AI model to generate a recipe:

[2220] User's current health status: Fatigue

[2221] User's current emotion: Wanting to relax

[2222] Usable ingredients: cabbage, carrots, chicken

[2223] Suggest a recipe to make:

[2224] This will enable passengers to enjoy comfortable meals and maintain their health while inside self-driving vehicles.

[2225] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2226] Step 1:

[2227] Photographing ingredients using an imaging device

[2228] The image capture device periodically captures images of food items in the refrigerator. The input is an image of the food items in the refrigerator, and the output is the captured image data. This image data is necessary to analyze the type, quantity, condition, and expiration date of the food items.

[2229] Step 2:

[2230] Sending image data to the server

[2231] Image data captured by an imaging device is sent to a server. The input is the captured image data, and the output is the image data uploaded to the server. This transmission prepares the server for subsequent analysis.

[2232] Step 3:

[2233] Image data analysis

[2234] The server analyzes the transmitted image data using deep learning technology (e.g., TensorFlow, PyTorch). The input is the image data uploaded to the server, and the output is the analysis results showing the type, quantity, condition, and expiration date of the ingredients. This analysis allows specific information about the ingredients to be registered in a database.

[2235] Step 4:

[2236] Input of the user's health and emotional state

[2237] Users use a smartphone or tablet to input their health and emotional status into the application. The input is information about the user's health and emotional status, and the output is the transmission of the input information. This allows data to be collected on the server for analysis by the emotion engine.

[2238] Step 5:

[2239] Analysis by emotion engine

[2240] The server uses an emotion engine (e.g., Microsoft Azure Cognitive Services) to analyze the user's health and emotional state. The input is the user's health and emotional state information, and the output is the analyzed emotion data. This data is stored in the user's emotion database.

[2241] Step 6:

[2242] Recipe generation using generative AI

[2243] The server's generative AI (e.g., OpenAI GPT-4) generates optimal recipes based on the stored ingredient data and the user's health and emotional state. The input is the stored ingredient data and the user's health and emotional state information, and the output is the generated recipe. This process suggests appropriate meals based on the user's state.

[2244] Step 7:

[2245] Sending and displaying recipes to user devices

[2246] The generated recipe is sent from the server to the user's smartphone or tablet. The input is the generated recipe, and the output is the recipe display on the user's device. The user can then cook a dish based on this recipe.

[2247] Step 8:

[2248] Use of delivery services

[2249] If a required ingredient is missing, the server automatically orders the missing ingredient from the delivery service. The input is the current ingredient status and the generated recipe, and the output is the order submission and replenishment of ingredients. This allows necessary ingredients to be replenished in a timely manner even while on the move, enabling comfortable meals to be served.

[2250] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2251] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2252] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2253] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2254] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2255] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2256] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2257] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2258] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2259] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2260] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2261] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2262] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2263] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2264] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2265] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2266] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2267] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2268] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2269] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid c...

Claims

1. A means for taking an image of the food using a camera built into the refrigerator; means for transmitting image data of ingredients to a server; A means for analyzing image data in the server and recognizing the type, amount, condition and expiration date of ingredients; a means for a user to input a health status; A means for generating a recipe based on the health condition and ingredient data and transmitting the recipe to a user terminal; A display means for the generated recipe; A system including:

2. The system according to claim 1, further comprising means for automatically generating and transmitting an order to a food delivery service when a required ingredient is in short supply.

3. 10. The system of claim 1, further comprising means for performing initial processing to extract and analyze features of image data captured by the camera.

Images