System

Abstract

A system is provided.SOLUTION: This system includes a means for photographing the best-before date of the purchased food and storing it as data, a means for notifying the user of the food whose best-before date is approaching, a means for proposing the optimum preservation method of the food, a means for providing a recipe obtained by combining the foods whose best-before dates are approaching, and a means for proposing the recipe under the consideration of the weather and season of the day and the physical condition of the user.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's busy society, people often forget about food expiration dates, resulting in the use of expired food and the waste of food. This problem is particularly important for busy people, such as young people living alone and housewives who are in charge of household chores. Furthermore, if people do not properly understand how to store food, the quality and safety of the food can be compromised. Furthermore, there is a lack of ideas for effectively utilizing food that is approaching its expiration date, leading to increased food waste. There is a need to solve these problems, enable users to easily manage their food, and reduce waste. [Means for solving the problem]

[0005] The present invention provides a means for photographing and saving the expiration dates of purchased foods as data, and a means for notifying users of foods approaching their expiration dates. It also provides a system that includes a means for suggesting optimal food storage methods and a means for providing recipes that combine foods with upcoming expiration dates. This system also includes a means for suggesting recipes that take into account the day's weather, seasonality, and the user's physical condition. Specifically, the system automatically extracts expiration dates using image recognition technology and has a function for sending push notifications based on notification conditions set by the user. It also suggests optimal storage methods based on the type of food and provides recipes using generative AI for using foods with approaching expiration dates. This allows users to effectively manage food expiration dates, reduce waste, and use ingredients safely and efficiently.

[0006] "Purchased food" refers to ingredients or food that a user purchases from a market, store, etc. and intends to consume at home.

[0007] The "best before" date is the date by which food is believed to retain its flavor and quality, and indicates that the quality may gradually deteriorate after this date.

[0008] "Photographing" refers to the act of using a camera or other image capture device to record an image of a particular object, in this case the expiration date of a food product.

[0009] "Data" refers to information managed within the system, and in this case refers to information such as food expiration dates, types, and storage methods.

[0010] "Notification" refers to a message or alert that notifies the user of a particular event or situation, and in this invention, is a notification that the expiration date is approaching.

[0011] "Best storage method" refers to the recommended method for effectively preserving a particular food product for an extended period of time, and refers to the appropriate conditions for maintaining its quality and safety.

[0012] A "recipe" is a document that describes the steps and methods for cooking a dish using specific ingredients.

[0013] "Combining" means using two or more different ingredients together in one dish or menu.

[0014] "Climate" refers to the weather conditions in a particular region or period, and is used in the present invention as a condition for recipe suggestions.

[0015] "Season" refers to the time when a particular food ingredient is at its most delicious and nutritious.

[0016] "User's physical condition" refers to the user's health condition at a specific date and time, as well as any changes in health condition, and indicates the factors to be taken into consideration when proposing recipes.

[0017] "Image recognition technology" is a technology for extracting specific information from an image taken by a camera, and in the present invention is used to automatically extract the expiration date.

[0018] "Push notification" refers to a form of notification in which specific information is automatically sent from a server to a user's device and displayed.

[0019] "Generative AI" refers to systems and technologies that use artificial intelligence to generate new information, and in this invention is used to create recipes. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] The present invention is a system for effectively managing the expiration dates of food products and reducing food waste. Specific embodiments of the system are described below.

[0042] System Configuration

[0043] The system consists of a user device, a server, and a database. Users use devices such as smartphones and tablets to manage expiration dates. The server processes and stores data, and the database stores food information.

[0044] Food information registration

[0045] To register the expiration date of purchased food, a user follows these steps: the user takes a photo of the food, and the device automatically extracts the expiration date using image recognition technology. If necessary, the user can enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database.

[0046] Examples:

[0047] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[0048] Best before date notification

[0049] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, it sends a push notification, for example, three days before the expiration date. The device receives this notification and displays a warning to the user.

[0050] Examples:

[0051] The server checks the database and finds that the milk's expiration date is three days away. Because the notification period set by the user is three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[0052] Preservation method

[0053] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[0054] Examples:

[0055] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[0056] Recipe Suggestions

[0057] If there are multiple foods nearing their expiration date, the server uses generative AI technology to create recipes that combine these foods. It also suggests recipes taking into account information such as the day's weather, seasonality, and the user's physical condition. The generated recipe is sent to the device and displayed to the user.

[0058] Examples:

[0059] If a user registers "chicken" and "carrots" as foods that are close to expiry, the server will use generative AI to suggest a recipe for "chicken and carrot teriyaki rice bowl." In addition, on cold days, it will also suggest recipes such as "chicken and carrot ginger soup."

[0060] In this way, the system of the present invention automates the management of food expiration dates and supports users in using food effectively by suggesting optimal storage methods and recipes.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] The user takes a photo of the expiration date on the food they purchased. The user uses the camera on their smartphone to take a photo of the part of the food that has the expiration date written on it.

[0064] Step 2:

[0065] The device will use image recognition to automatically extract the expiration date, and the app on the device will use image processing technology to extract the necessary date information from the image.

[0066] Step 3:

[0067] The user enters additional information such as the type of food and storage method. The user can manually enter the food name, purchase date, storage method, etc. using the app screen.

[0068] Step 4:

[0069] The terminal sends the collected food information to the server, which then packages the extracted expiration date and any additional information entered by the user and sends it to the server.

[0070] Step 5:

[0071] The server saves the food information in a database. The server analyzes the received food data and stores it in the database.

[0072] Step 6:

[0073] The server periodically checks the database to identify foods that are approaching their expiration date. A server-side program scans the database at regular intervals and selects foods that are nearing their expiration date.

[0074] Step 7:

[0075] The server prepares and sends notifications for the relevant foods to the terminal. The server generates notification messages for the identified foods and sends them to the terminal a set number of days in advance.

[0076] Step 8:

[0077] The terminal receives the notification and displays a warning to the user. The terminal receives the notification from the server and displays a message on the user's screen warning that the expiration date is approaching.

[0078] Step 9:

[0079] The user interacts with their device to learn how to store food. The user selects the option within the app to learn how to store food.

[0080] Step 10:

[0081] The device sends a request to the server regarding how to store food. The device sends a request to the server to inquire about how to store food.

[0082] Step 11:

[0083] The server retrieves information about the storage method from the database. The server retrieves the data about the storage method in the database and searches for related information.

[0084] Step 12:

[0085] The server sends information about how to save the data to the device. The server compiles the search results and sends details about how to save the data to the device.

[0086] Step 13:

[0087] The device displays the storage method to the user. The device displays the received storage method information on the app screen so that the user can check it.

[0088] Step 14:

[0089] The server sends a request to the generation AI to generate a recipe using foods that are close to their expiration date. The server creates a list of foods that are close to their expiration date and sends a request to the generation AI to generate a recipe.

[0090] Step 15:

[0091] The AI ​​generates recipes based on requests, taking into account the target food, climate, season, and the user's physical condition to create the optimal recipe.

[0092] Step 16:

[0093] The server saves the generated recipe and sends it to the device. The server temporarily saves the recipe information received from the generation AI in a database and sends it to the device.

[0094] Step 17:

[0095] The device displays the generated recipe to the user. The device displays the received recipe in the app, allowing the user to cook a dish based on it.

[0096] Through these steps, the system of the present invention helps users effectively manage food expiration dates, reduce waste, and use ingredients safely and efficiently.

[0097] Example 1

[0098] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0099] Currently, consumers must manually record the expiration dates of purchased food, which requires a great deal of effort. Furthermore, many people discard food without realizing that its expiration date is approaching, resulting in significant waste and environmental impact. Furthermore, it is difficult to maintain food quality due to lack of proper storage methods. To solve these issues, a system is needed that allows consumers to efficiently manage food, reduce waste, and properly store and use food.

[0100] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0101] In this invention, the server includes a device that photographs the expiration dates of food products and stores them as data, a device that notifies users of food products that are approaching their expiration dates, a device that suggests optimal food storage methods, a device that creates recipes that combine foods that are approaching their expiration dates, and a device that suggests recipes taking into account the weather, season, and the user's physical condition on that day. This allows consumers to efficiently manage the expiration dates of purchased food products, reduce waste, and ensure appropriate use of food products while maintaining their quality.

[0102] A "device that photographs the expiration date of food products and stores it as data" is a device that has the function of photographing the expiration date of food products purchased by a user, extracting the date information using image recognition technology, and storing the extracted data.

[0103] A "device that notifies users of food products approaching their expiration date" is a device that periodically checks food information stored in a database, identifies foods that are approaching their expiration date, and notifies users.

[0104] The "device that suggests the optimum method of preserving food" is a device that has the function of searching for the optimum method of preserving food based on the type of food registered and suggesting it to the user.

[0105] The "device for generating recipes that combine foods with approaching expiration dates" is a device that combines multiple foods that are approaching their expiration dates, generates appropriate recipes using generative AI technology, and provides them to users.

[0106] "A device that suggests recipes taking into account the weather, season, and the user's physical condition on that day" refers to a device that has the function of using generative AI technology to suggest appropriate recipes based on the current weather, season, and the user's physical condition, and providing them to the user.

[0107] The present invention is a system that aims to effectively manage food expiration dates and reduce food waste. This system is composed of a user's terminal, a server, and a database, and describes specific embodiments that include registering food information, notifying users of expiration dates, suggesting storage methods, and suggesting recipes.

[0108] System Configuration

[0109] The system allows users to manage expiration dates using devices such as smartphones and tablets. The server processes and stores data, and the database stores food information.

[0110] Food information registration

[0111] To register the expiration date of purchased food, the user first takes a photo of the food with the device. The device then uses image recognition technology (e.g., Google Cloud Vision API or Amazon Rekognition) to automatically extract the expiration date from the image. The user can then enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database (e.g., MySQL or PostgreSQL).

[0112] Examples:

[0113] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[0114] Best before date notification

[0115] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. The check is performed periodically, such as once a day. Based on the user's settings, a push notification is sent, for example, three days before the expiration date. The device receives this notification and displays a warning to the user (for example, using Firebase Cloud Messaging to send a push notification).

[0116] Examples:

[0117] The server checks the database and finds that the milk's expiration date is three days away. Because the user has set the notification period to three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[0118] Preservation method

[0119] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the optimal storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[0120] Examples:

[0121] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[0122] Recipe Suggestions

[0123] If there are multiple foods nearing their expiration date, the server uses generative AI technology (e.g., OpenAI GPT-3) to create recipes that combine these foods. Furthermore, it takes into account information such as the weather, season, and the user's physical condition on the day to suggest recipes. The generated recipe is sent to the device and displayed to the user.

[0124] Examples:

[0125] If a user registers "chicken" and "carrots" as foods that are close to expiry, the server will use generative AI to suggest a recipe for "chicken and carrot teriyaki rice bowl." In addition, on cold days, it will also suggest recipes such as "chicken and carrot ginger soup."

[0126] Example prompt sentence:

[0127] Please suggest a recipe using "chicken and carrots." The current temperature is 10 degrees and the user is feeling cold.

[0128] As described above, the system of the present invention automates the management of food expiration dates and supports users in using food effectively by suggesting optimal storage methods and recipes.

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] System program processing flow

[0131] Step 1:

[0132] The user takes a photo of the food they have purchased using the camera on their smartphone or tablet.

[0133] Specific behavior:

[0134] The user activates the smartphone camera and takes a picture of the food package.

[0135] Input: Food to be photographed

[0136] Output: Food image data

[0137] Step 2:

[0138] The device processes the captured image and uses image recognition technology to extract the expiration date.

[0139] Specific behavior:

[0140] The device uses the Google Cloud Vision API or Amazon Rekognition to analyze the image and extract the expiration date.

[0141] Input: Food image data

[0142] Output: Extracted best-before date data

[0143] Step 3:

[0144] If necessary, the user can input additional information such as the type of food and storage method into the terminal.

[0145] Specific behavior:

[0146] The user enters the type of food (e.g., milk) and storage method (e.g., refrigerate) into an application form.

[0147] Input: Additional information entered by the user about the type of food and storage method

[0148] Output: Complete food data (expiration date, type, storage method)

[0149] Step 4:

[0150] The terminal transmits the extracted and entered information to the server.

[0151] Specific behavior:

[0152] The device sends JSON format data to the server via an HTTP request.

[0153] Input: Complete food data

[0154] Output: Data sent to server completed

[0155] Step 5:

[0156] The server stores the received data in a database.

[0157] Specific behavior:

[0158] The server connects to a MySQL database and stores data using SQL queries.

[0159] Input: Complete food data

[0160] Output: Food data stored in a database

[0161] Step 6:

[0162] The server periodically checks the database to identify foods that are nearing their expiration date.

[0163] Specific behavior:

[0164] The server sets up a Cron job to check the expiration date in the database every day at midnight.

[0165] Input: Food data in the database

[0166] Output: A list of foods that are close to expiry date

[0167] Step 7:

[0168] The server generates a push notification about the identified food item and sends it to the device.

[0169] Specific behavior:

[0170] The server uses the Firebase Cloud Messaging API to generate a push notification and send it to the device.

[0171] Input: List of foods that are close to expiry date

[0172] Output: Push notification sent to device

[0173] Step 8:

[0174] The terminal displays the received notification to the user.

[0175] Specific behavior:

[0176] The device receives a notification and displays a pop-up message saying, "The milk's expiration date is in three days."

[0177] Input: Push notification from the server

[0178] Output: Display a warning message to the user

[0179] Step 9:

[0180] When a user registers a food item, the device requests storage instructions from the server.

[0181] Specific behavior:

[0182] The user registers "tomato" and presses the button "Please tell me how to store it." The device then requests the server how to store that food.

[0183] Input: Food information registered by the user

[0184] Output: Request sent to server

[0185] Step 10:

[0186] The server searches the database for storage methods based on the type of food and sends them to the terminal.

[0187] Specific behavior:

[0188] The server searches the "food preservation method" table for the method for preserving "tomatoes" and sends it to the terminal.

[0189] Input: Save method request from user

[0190] Output: Sending save method data

[0191] Step 11:

[0192] The terminal displays the received saving method to the user.

[0193] Specific behavior:

[0194] The device will display "How to store tomatoes: Store at room temperature, refrigerate after cutting."

[0195] Input: Save method data from the server

[0196] Output: Show user how to save

[0197] Step 12:

[0198] The server organizes information about food items that are close to their expiration date and generates recipes using a generative AI model.

[0199] Specific behavior:

[0200] The server inputs the prompt "Foods with an approaching expiration date: chicken, carrots. Please suggest a recipe using these." into the generation AI and generates a recipe.

[0201] Input: Food information about expiry date approaching, prompt text

[0202] Output: The generated recipe

[0203] Step 13:

[0204] The server sends the generated recipe to the terminal.

[0205] Specific behavior:

[0206] The server sends the recipe for "Teriyaki Chicken and Carrot Rice Bowl" to the terminal.

[0207] Input: Generated recipe

[0208] Output: Send recipe to terminal

[0209] Step 14:

[0210] The terminal displays the received recipe to the user.

[0211] Specific behavior:

[0212] The device displays "Teriyaki Rice Bowl Recipe: Chicken, Carrots, Soy Sauce, Mirin, Sugar."

[0213] Input: Recipe information from the server

[0214] Output: Recipe displayed to user

[0215] (Application example 1)

[0216] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0217] In today's food delivery services, improving the efficiency of food inventory management and reducing food waste are major challenges. In particular, insufficient expiration date management has led to widespread food waste due to expired food, resulting in economic losses and increasing the burden on the environment. Furthermore, it is difficult to propose recipes and create menus that effectively utilize food that is approaching its expiration date, increasing the burden on employees. To solve these issues, a system that automatically manages expiration dates and suggests optimal storage methods and recipes is needed.

[0218] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0219] In this invention, the server includes means for photographing the expiration dates of purchased foods and saving them as data, means for notifying users of foods approaching their expiration dates, means for suggesting optimal food storage methods, means for generating and providing recipes combining foods approaching their expiration dates using a generative AI model, means for suggesting recipes taking into consideration the weather, seasons, and the user's physical condition on that day, and means for creating menus based on foods approaching their expiration dates and supporting efficient food delivery. This enables efficient food inventory management and reduces food waste.

[0220] "Means for photographing the expiration date of purchased food and saving it as data" refers to a system that photographs the expiration date of food with a camera and automatically recognizes and saves the data.

[0221] The "means for notifying about foods approaching their expiration date" is a function for automatically detecting foods approaching their expiration date and sending a notification to the user.

[0222] A "means for suggesting the optimal method of food preservation" is a system for providing appropriate preservation methods for specific foods.

[0223] "Means for generating and providing recipes that combine food items with close expiration dates using a generative AI model" refers to a system for combining food items with close expiration dates, creating recipes using a generative AI model, and providing the recipes.

[0224] "Means for suggesting recipes taking into consideration the weather, season, and the user's physical condition on that day" is a function for suggesting appropriate recipes taking into consideration the weather, season, and the user's physical condition on that day.

[0225] "A means for creating menus based on foods with approaching expiration dates and supporting efficient food delivery" is a system for creating menus based on foods with approaching expiration dates and operating food delivery efficiently.

[0226] MODE FOR CARRYING OUT THE INVENTION

[0227] The present invention provides a system for improving the efficiency of food inventory management and reducing food waste in food delivery services. Specific embodiments of the system are described below.

[0228] System configuration

[0229] The system consists of a user's device, a server, and a database. Users use devices such as smartphones to manage food expiration dates. The server processes and stores data, and the database stores food information.

[0230] Food information registration

[0231] Users take a photo of the expiration date of purchased food using their smartphone camera. The device uses OpenCV and pytesseract for image recognition, automatically extracts the expiration date, and saves it as food information. Users can also enter additional information such as the type of food and storage method. This data is sent to the server, which stores it in a database.

[0232] Best before date notification

[0233] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, a push notification is sent, for example, three days before the expiration date. The device receives this notification and displays a warning to the user. The notification is sent using plyer.

[0234] Preservation method

[0235] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the optimal storage method for that food item and sends it to the device. The device then displays this storage method to the user.

[0236] Recipe Suggestions

[0237] If there are multiple foods approaching their expiration date, the server uses a generative AI model to create recipes that combine these foods. It also suggests recipes taking into account information such as the day's weather, seasonality, and the user's physical condition. The generated recipes are sent to the device and displayed to the user. It also creates menus based on foods approaching their expiration date, supporting the operation of the delivery service.

[0238] Specific examples

[0239] A user purchases milk and takes a photo of the label with a smartphone, which says the expiration date "December 15, 2023." The device automatically extracts this date and sends it to the server as "milk" information along with the "keep refrigerated" information. The server then stores it in a database.

[0240] When the expiration date is approaching in three days, the server sends a notification to the terminal stating "The expiration date of the milk is approaching in three days" and displays a warning to the user.

[0241] If a user registers "chicken" and "carrots" as foods that are close to their expiration date, the server will use generative AI to suggest a recipe for "chicken and carrot teriyaki rice bowl." It will also suggest recipes such as "chicken and carrot ginger soup" for cold days.

[0242] Prompt Sentence Examples

[0243] "Based on the food information below, generate a list of foods nearing their end of life and display them sorted by expiry date. Set up subsequent notifications and suggest storage methods and recipes."

[0244] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0245] Step 1: Photograph and extract food information

[0246] A user takes a photo of the expiration date of a purchased food item with their smartphone. The device then analyzes the image using OpenCV and pytesseract and extracts the expiration date as text data. The input data is the image of the food, and the output data is the extracted expiration date. This process saves the user the trouble of manually entering the data.

[0247] Step 2: Save food information

[0248] The device sends the extracted expiration date and any additional information entered by the user, such as the type of food and storage method, to the server. The server receives this information and stores it in a database. The input data is information such as the expiration date, type of food, and storage method, and the output data is the updated database. This allows for centralized management of food expiration date information.

[0249] Step 3: Regularly check expiration dates

[0250] The server periodically (e.g., daily) checks the food information in the database and identifies foods that are approaching their expiration date. This process makes it possible to quickly detect foods that are approaching their expiration date. The input data is all food information in the database, and the output data is a list of foods that are approaching their expiration date.

[0251] Step 4: Best before notification

[0252] The server creates a list of foods that are close to their expiration date and sends a push notification to the device three days before the expiration date. The device receives this notification and displays a warning to the user. The input data is a list of foods that are close to their expiration date, and the output data is a notification message to the user. This allows the user to use the food without worrying about the expiration date.

[0253] Step 5: Recommending the best storage method

[0254] When a user registers a food item, the device sends a request for storage method to the server. The server searches the database for the optimal storage method and sends it to the device. The device then displays this storage method to the user. The input data is the type of food, and the output data is the optimal storage method. This allows the quality of the food to be maintained at its optimum.

[0255] Step 6: Recipe suggestions

[0256] If there are multiple foods nearing their expiration date, the server uses a generative AI model to create a recipe that combines these foods. The recipe is also customized taking into account the weather, seasons, and the user's physical condition. The generative AI uses specific prompts. The input data is a list of foods nearing their expiration date and information about the weather and physical condition, and the output data is a suggested recipe. This reduces food waste while supporting the user's health.

[0257] As a specific example, the prompt text used is, "Based on the food information below, generate a list of foods that are nearing the end of their lifespan, and display them sorted by date for those with the nearest expiration date. Set the system to notify you in the future, and suggest storage methods and recipes."

[0258] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0259] The present invention is a system that aims to effectively manage food expiration dates and reduce food waste. Furthermore, by combining it with an emotion engine that recognizes the user's emotions and adding a function that suggests recipes based on the user's emotions, the user experience is improved. Specific embodiments of the system are described below.

[0260] System Configuration

[0261] The system consists of a user device, a server, a database, and an emotion engine. Users use devices such as smartphones and tablets to manage expiration dates and recognize emotions. The server processes and stores data, and the database stores information about food and emotions. The emotion engine is a dedicated component for recognizing and analyzing user emotions.

[0262] Food information registration

[0263] To register the expiration date of purchased food, a user follows these steps: the user takes a photo of the food, and the device automatically extracts the expiration date using image recognition technology. If necessary, the user can enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database.

[0264] Examples:

[0265] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[0266] Best before date notification

[0267] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, it sends a push notification, for example, three days before the expiration date. The device receives this notification and displays a warning to the user.

[0268] Examples:

[0269] The server checks the database and finds that the milk's expiration date is three days away. Because the notification period set by the user is three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[0270] Preservation method

[0271] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[0272] Examples:

[0273] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[0274] Emotion Recognition and Recipe Suggestion

[0275] The app uses cameras and sensors installed on the device to recognize the user's emotions. While the user is using the app, the emotion engine analyzes the user's facial expressions and voice to generate emotion data. This data is sent to the server for analysis. Based on the analysis results, the server uses generative AI to suggest recipes that take into account the day's weather, seasonal ingredients, the user's physical condition, and emotions.

[0276] Examples:

[0277] While the user is using the app, the emotion engine recognizes the user's facial expressions and generates emotional data such as "tired." Based on this information, the server uses generative AI to suggest recipes such as "chicken and carrot soup with fatigue-relieving effects." These are displayed on the user's device, and the user can use the recipe to cook the dish.

[0278] In this way, the system of the present invention automates food expiration date management, improves user experience through emotion-based recipe suggestions, reduces food waste, and supports safe and efficient use of ingredients.

[0279] The processing flow will be explained below.

[0280] Step 1:

[0281] The user takes a photo of the expiration date on the food product they purchased. The user uses the camera on their smartphone to take a photo of the expiration date printed on the food product label.

[0282] Step 2:

[0283] The device will perform image recognition and automatically extract the expiration date, while the app on the device will use image processing technology to analyze and extract the expiration date from the image.

[0284] Step 3:

[0285] The user enters additional information such as the type of food and storage method. The user enters the name of the food, purchase date, and storage method through the app screen.

[0286] Step 4:

[0287] The terminal sends the collected food information to the server, which then packages the extracted expiration date and any additional information entered by the user and sends it to the server.

[0288] Step 5:

[0289] The server stores the food information in a database. The server analyzes the received food data and stores it in a database.

[0290] Step 6:

[0291] The server periodically checks the database to identify foods that are nearing their expiration date. A program on the server periodically scans the database to identify foods that are nearing their expiration date.

[0292] Step 7:

[0293] The server prepares notifications for the relevant foods and sends them to the terminal. The server generates notification messages for the identified foods and sends them to the terminal a set number of days in advance.

[0294] Step 8:

[0295] The terminal receives the notification and displays a warning to the user. The terminal receives the notification from the server and displays a warning message on the user's screen informing them that the expiration date is approaching.

[0296] Step 9:

[0297] The user operates the device to check how to store food. The user selects an option in the device's app to check how to store food.

[0298] Step 10:

[0299] The device sends a request to the server about how to store food.The device sends a request to the server about how to store food.

[0300] Step 11:

[0301] The server searches the database for information on the storage method. The server searches the database for information on the relevant storage method based on the request.

[0302] Step 12:

[0303] The server sends information on how to save the data to the device. The server compiles the search results and sends information on how to save the data to the device.

[0304] Step 13:

[0305] The device displays the storage method to the user. The device displays the received storage method information on the app screen so that the user can check it.

[0306] Step 14:

[0307] The device uses cameras and sensors to recognize the user's emotions, and the emotion engine analyzes the user's facial expressions and voice to generate emotion data.

[0308] Step 15:

[0309] The device sends the generated emotion data to the server. The device packages the user's emotion data and sends it to the server.

[0310] Step 16:

[0311] The server analyzes the user's emotional data. The server analyzes the received emotional data and identifies the user's emotional state.

[0312] Step 17:

[0313] The server takes into consideration emotional data, weather, seasonality, and the user's physical condition, and sends a request to the generation AI to generate a recipe. The server then sends a recipe request to the generation AI based on this information.

[0314] Step 18:

[0315] The generative AI generates the optimal recipe. Based on the information it receives from the server, the generative AI creates a recipe that takes into account the user's emotional state and other conditions.

[0316] Step 19:

[0317] The server saves the generated recipe and sends it to the device. The server temporarily saves the recipe information received from the generation AI in a database and sends it to the device.

[0318] Step 20:

[0319] The device displays the generated recipe to the user. The device displays the received recipe within the app, allowing the user to use the recipe as a reference when cooking.

[0320] Through these steps, the system of the present invention automates food expiration date management, improves user experience through emotion-based recipe suggestions, reduces food waste, and supports safe and efficient ingredient use.

[0321] Example 2

[0322] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0323] In modern society, the increase in food waste has become a major problem. This problem is particularly caused by poor expiration date management within the home. Furthermore, there is a lack of meal suggestions that take into account the user's emotions and physical condition, and there is a need for methods to use ingredients appropriately and without waste. Conventional systems have difficulty meeting these requirements, so an effective solution is needed to simultaneously manage food expiration dates and improve the user experience.

[0324] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0325] In this invention, the server includes a means for photographing the expiration dates of purchased items and storing them as data, a means for notifying the user of items approaching their expiration dates, a means for suggesting optimal storage methods for the items, and a means for analyzing the user's emotions, thereby reducing food waste and improving the user experience through emotion-based meal suggestions.

[0326] "Goods" refers to food or consumable items purchased or acquired by the user.

[0327] "Best before date" refers to the period during which an item can be consumed while maintaining its quality.

[0328] "Image recognition technology" refers to a form of computer vision that extracts and identifies specific information from image data.

[0329] "Data" refers to electromagnetically recorded information, including information such as expiration date, type of product, and storage method.

[0330] "Notification" refers to an alert or message that is displayed on a user's device.

[0331] "Preservation method" refers to the specific procedures and conditions for preserving an item in optimal condition.

[0332] "Cooking method" refers to the steps and methods for creating a dish using an item.

[0333] "Climate" refers to the weather conditions and temperature of the day.

[0334] "Season" refers to the time when something is at its most delicious and nutritious.

[0335] "User" refers to the consumer or end user who uses the system.

[0336] "Emotion" refers to the psychological state detected from the user's facial expressions, voice, etc.

[0337] "Emotion engine" is a general term for the algorithms and hardware that analyze the user's emotions and generate the results as data.

[0338] "Server" refers to a computer system that processes, stores, and analyzes data.

[0339] "Database" refers to a digital storage system for systematically storing and managing information such as expiration dates, storage methods, and emotional data.

[0340] "Generative AI model" refers to a machine learning model that generates useful results (such as recipes) based on input prompts.

[0341] A "prompt sentence" refers to an instruction sentence used as input to a generative AI model.

[0342] The present invention is a system that allows users to manage food expiration dates and receive cooking method suggestions based on their emotions. This system is composed of a terminal, a server, a database, and an emotion engine. Specific embodiments of the system are described below.

[0343] System Configuration

[0344] The system consists of a user device (smartphone or tablet), a server, a database, and an emotion engine. The server uses a powerful computer system to process, store, and analyze data. The database is a digital storage system for systematically storing and managing information such as expiration dates, storage methods, and emotion data. The emotion engine consists of hardware and software components for analyzing user emotions and generating the results as data.

[0345] Food information registration

[0346] To register the expiration date of purchased food, a user takes a photo of the food label using the device's camera. The device then uses image recognition technology to extract the expiration date as text data from the captured image. If necessary, the user can manually enter the type of food and storage method. The device then formats this data and sends an HTTP request to the server. The server then analyzes the received data and stores it in a database in the required format.

[0347] Best before date notification

[0348] The server periodically checks the database to identify foods that are approaching their expiration date. Based on user settings, the server creates and sends a notification three days before the expiration date, for example. The device receives this notification and displays a warning message to the user, such as "The milk's expiration date is three days away."

[0349] Preservation method

[0350] When a user registers a food item, the device requests the server for the optimal storage method for that food item. The server searches the database for the optimal storage method for that food item and sends it to the device as an HTTP response. The device then presents the received storage method to the user, displaying specific storage instructions such as "Store tomatoes at room temperature, or refrigerate if cut."

[0351] Emotion Recognition and Recipe Suggestion

[0352] While the user is using the app, emotion data is collected using the device's on-board camera and sensors. The device then analyzes the collected emotion data using an emotion engine and sends the results to the server. The server generates prompts based on the emotion data and other factors (weather, season, physical condition, etc.) and inputs them into the generative AI model. The server receives the resulting recipes generated by the generative AI model and sends them to the device. The device then presents specific recipes to the user, such as "chicken and carrot soup with fatigue-relieving effects."

[0353] Examples of prompt statements

[0354] Example prompts to input to a generative AI model:

[0355] "What recipes would you recommend to users who are feeling tired? For example, a simple soup recipe using foods that have a fatigue-relieving effect."

[0356] In this way, the system of the present invention automates food expiration date management, improves user experience through emotion-based cooking suggestions, reduces food waste, and supports safe and efficient use of ingredients.

[0357] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0358] Step 1:

[0359] The user takes a photo of the food they purchased on their device.

[0360] Input: A user takes a photo of a food label with their smartphone camera.

[0361] How it works: The user launches the camera app, frames a food label with an expiration date, and presses the capture button.

[0362] Output: The image data of the photographed food label is saved on the device.

[0363] Step 2:

[0364] The device uses image recognition technology to extract the expiration date.

[0365] Input: Image data acquired in step 1.

[0366] How it works: The device uses OCR (optical character recognition) technology to analyze the text data in the image and extract the expiration date as text data.

[0367] Output: Best before date data is obtained in text format.

[0368] Step 3:

[0369] The user enters any additional information needed (such as the type of food and storage method)

[0370] Input: Best before date data obtained in step 2 and manual user input information.

[0371] What it does: A user manually enters the type of food (e.g., milk, bread) and storage method (e.g., refrigerate, freeze) using an app input form.

[0372] Output: The expiration date data and the type of food and storage method data entered by the user are obtained.

[0373] Step 4:

[0374] The device sends the data to the server

[0375] Input: Best before date data obtained in step 3, food type, and storage method data.

[0376] How it works: The device formats this data and sends an HTTP request to the server.

[0377] Output: The data is sent to the server.

[0378] Step 5:

[0379] The server stores the received information in a database

[0380] Input: The data sent from the terminal in step 4.

[0381] How it works: The server parses the received data and stores it in the required format in the database.

[0382] Output: The expiration date, food type, and storage method data are saved in the database.

[0383] Step 6:

[0384] The server periodically checks the database

[0385] Input: Food information in the database.

[0386] How it works: The server runs a periodic job that checks all food items in the database.

[0387] Output: A list of foods that are nearing their expiration date is generated.

[0388] Step 7:

[0389] The server identifies food that is nearing its expiration date

[0390] Input: The list of foods obtained in step 6.

[0391] How it works: The server compares the expiration dates of the foods with the current date to identify foods that require notification.

[0392] Output: A list of foods that require notification is generated.

[0393] Step 8:

[0394] The server creates and sends the push notification

[0395] Input: The list of foods requiring notifications obtained in step 7.

[0396] How it works: The server creates a notification based on the user's settings, for example, three days before the expiration date, and sends a push notification to the user's device.

[0397] Output: A push notification is sent to the user's device.

[0398] Step 9:

[0399] The device receives the notification and displays it to the user.

[0400] Input: The push notification sent in step 8.

[0401] What it does: The device receives a notification from the server and displays a warning message such as "The milk's expiration date is coming up in three days."

[0402] Output: A notification message is displayed on the device screen.

[0403] Step 10:

[0404] Send a storage request when a user registers a food item

[0405] Input: The user enters the name of the food.

[0406] How it works: The user enters the name of a food item into the app's input form and presses the "Get storage instructions" button. The device then sends a request to the server.

[0407] Output: A save method request is sent to the server.

[0408] Step 11:

[0409] The server looks up the save method in the database

[0410] Input: The request submitted in step 10.

[0411] How it works: The server receives a request and searches its database for the best way to store the food.

[0412] Output: The search results for the saving method are obtained.

[0413] Step 12:

[0414] The server sends the save instructions to the device.

[0415] Input: Search results for saving methods obtained in step 11.

[0416] How it works: The server formats the search results and sends them to the device as an HTTP response.

[0417] Output: Information on how to save is sent to the device.

[0418] Step 13:

[0419] The device will prompt the user for instructions on how to save.

[0420] Input: The save method information submitted in step 12.

[0421] Operation: The device presents the received storage instructions to the user, displaying specific storage instructions such as "Store tomatoes at room temperature, or refrigerate if cut."

[0422] Output: Information about the saving method will be displayed on the device screen.

[0423] Step 14:

[0424] The device collects emotion data while the user is using the app.

[0425] Input: User's facial and voice data.

[0426] How it works: The user opens the app and uses the camera and sensors to capture facial expressions and voice in real time. The device then captures this emotional data.

[0427] Output: Emotion data is saved on the device.

[0428] Step 15:

[0429] The device analyzes the emotional data it acquires and sends it to the server.

[0430] Input: Emotion data obtained in step 14.

[0431] Operation: The device analyzes the facial expression data using the emotion engine, extracts the emotional state (e.g., "tired"), and sends it to the server.

[0432] Output: The analyzed emotion data is sent to the server.

[0433] Step 16:

[0434] The server generates a prompt and inputs it into the generative AI model.

[0435] Input: Emotion data sent in step 15 and other factors (weather, season, physical condition information).

[0436] How it works: The server uses this information to generate a prompt (e.g., "What recipes would you recommend for a tired user?") and inputs it into the generative AI model.

[0437] Output: The generated prompt sentence is input to the generative AI model.

[0438] Step 17:

[0439] The server sends the generated recipe to the device.

[0440] Input: The recipe obtained from the generative AI model in step 16.

[0441] How it works: The server receives the recipe generated by the AI ​​model and sends it to the device.

[0442] Output: Recipe data is sent to the device.

[0443] Step 18:

[0444] The device displays the recipe to the user

[0445] Input: Recipe data submitted in step 17.

[0446] How it works: The device displays the recipe it receives from the server on the screen, and suggests specific recipes such as "chicken and carrot soup with fatigue-relieving effects."

[0447] Output: The recipe is displayed on the device screen and provided to the user.

[0448] In this way, by linking each processing step with each other, the system of the present invention can effectively manage food expiration dates, suggest storage methods, and suggest cooking methods based on emotions.

[0449] (Application example 2)

[0450] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0451] In recent years, with the increasing importance of managing food expiration dates, there is a demand for suggestions of appropriate storage methods and recipes using food that is close to its expiration date. However, because individual factors such as emotions and physical condition are not taken into consideration, the user experience is often unsatisfactory. Furthermore, expiration date management is sometimes not thoroughly implemented in physical stores, leading to a serious problem of food waste. To solve these issues, a system that reflects user emotions and can also manage expiration dates in physical stores is needed.

[0452] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for photographing the expiration dates of purchased foods and saving them as data, means for notifying users of foods approaching their expiration dates, means for suggesting optimal food storage methods, means for providing recipes that combine foods with upcoming expiration dates, means for suggesting recipes taking into account the weather, seasons, and the user's physical condition on that day, means for recognizing the user's emotions and suggesting recipes based on the emotion data, and means for managing food in a physical store and suggesting recipes based on the emotions. This makes it possible to suggest recipes that correspond to the emotions of each individual user, reducing food waste and enabling efficient food management in physical stores.

[0453] The "best before date" of a food product is the date by which the food product will continue to maintain its appropriate quality.

[0454] "Storing as data" means electronically recording information such as expiration dates and making them accessible later.

[0455] "Notifying" means sending a warning or alert to the user when a specific condition is met.

[0456] "Best storage method" refers to the recommended storage environment and conditions to maintain the quality of food.

[0457] A "recipe" is a list of steps and ingredients for making a dish.

[0458] "Climate" refers to weather conditions such as temperature, humidity, wind speed, and precipitation over a certain period of time.

[0459] "Season" refers to the time when a particular food is most nutritious and delicious.

[0460] "User's physical condition" refers to the user's health condition and current physical condition.

[0461] "Emotion data" refers to emotional information obtained by analyzing the user's facial expressions, voice, behavior, etc.

[0462] A "brick and mortar store" is a physical retail establishment where consumers can visit and purchase products in person.

[0463] "Emotion recognition" means judging the user's emotional state by analyzing the user's facial expressions, etc.

[0464] "Emotion-based recipe suggestions" refers to taking into account the user's current emotional state and providing cooking ideas that are appropriate for that emotion.

[0465] "Managing food" means centrally monitoring and controlling food inventory, expiration dates, storage methods, etc.

[0466] "Image recognition technology" is a technology that analyzes image data and automatically recognizes specific information (such as characters or objects).

[0467] A "server" is a computer system that stores, processes, and manages data.

[0468] "Suggesting storage methods" means showing the user the best storage conditions for a particular food item.

[0469] "Providing recipe combinations" means teaching users how to combine multiple foods into one dish.

[0470] "User-specified" means that the user inputs his / her wishes and conditions into the system and makes a selection.

[0471] "Push notification" is a mechanism that notifies users of specific information in a timely manner.

[0472] This invention aims to effectively manage the expiration dates of food purchased by users and reduce food waste. Furthermore, by combining it with an emotion engine that recognizes user emotions, it can suggest recipes based on emotions, improving the user experience.

[0473] System Configuration

[0474] The system consists of a user device, a server, a database, and an emotion engine. Users use devices such as smartphones and tablets to manage expiration dates and recognize emotions. The server processes and stores data, and the database stores information about food and emotions. The emotion engine is a dedicated component for recognizing and analyzing user emotions.

[0475] Food information registration

[0476] To register the expiration date of purchased food, a user follows these steps: the user takes a photo of the food, and the device automatically extracts the expiration date using image recognition technology. If necessary, the user can enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database.

[0477] Specific examples

[0478] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[0479] Best before date notification

[0480] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, it sends a push notification, for example, three days before the expiration date. The device receives this notification and displays a warning to the user.

[0481] Specific examples

[0482] The server checks the database and finds that the milk's expiration date is three days away. Because the notification period set by the user is three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[0483] Preservation method

[0484] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[0485] Specific examples

[0486] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[0487] Emotion Recognition and Recipe Suggestion

[0488] The app uses cameras and sensors installed on the device to recognize the user's emotions. While the user is using the app, the emotion engine analyzes the user's facial expressions and voice to generate emotion data. This data is sent to the server for analysis. Based on the analysis results, the server uses generative AI to suggest recipes that take into account the day's weather, seasonal ingredients, the user's physical condition, and emotions.

[0489] Specific examples

[0490] While the user is using the app, the emotion engine recognizes the user's facial expressions and generates emotional data such as "tired." Based on this information, the server uses generative AI to suggest recipes such as "chicken and carrot soup with fatigue-relieving effects." These are displayed on the user's device, and the user can use the recipe to cook the dish.

[0491] Examples of prompts to input to a generative AI model:

[0492] Generate a recipe for someone who is feeling tired and has the following items: carrots, chicken broth, onions, garlic.

[0493] Application in physical stores

[0494] This system can also be applied in physical stores. Employees and customers can use their smartphones to manage food in the store and suggest recipes based on their emotions. This will make it easier to manage expiration dates in the store and is expected to reduce food waste.

[0495] In this way, the system of the present invention automates food expiration date management, improves user experience through emotion-based recipe suggestions, reduces food waste, and supports safe and efficient use of ingredients.

[0496] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0497] Step 1:

[0498] The user takes a photo of the food they have purchased. Using the device, they take a photo of the label containing the expiration date with a camera, and the image data is used as input.

[0499] Step 2:

[0500] The device automatically extracts the expiration date using image recognition technology. It uses an image recognition algorithm (e.g., OpenCV) to analyze the characters in the image and identify the expiration date. The identified expiration date is the output.

[0501] Step 3:

[0502] If necessary, the user inputs additional information, such as the type of food or storage method. The information is supplemented through the device's user interface with text input or selection menus. The input additional information is output.

[0503] Step 4:

[0504] The device sends this data (best before date, food type, storage method) to the server. The data is transferred to the server through the network. The raw data sent to the server is the output.

[0505] Step 5:

[0506] The server stores the received data in a database. The data is structured and stored using a database management system (e.g., SQLite). The food information stored in the database is the output.

[0507] Step 6:

[0508] The server periodically checks the food information in the database to identify foods that are approaching their expiration date. It then runs a query to extract foods with an expiration date within three days. The extracted list of foods is the output.

[0509] Step 7:

[0510] The server sends push notifications for the identified foods based on the user settings. If the notification condition (e.g., three days before) is met, the server generates the notification content and sends it to the user's device. The notification message displayed on the device is the output.

[0511] Step 8:

[0512] The terminal receives the notification and displays a warning to the user. The notification message is displayed in a popup format to get the user's immediate attention. The warning displayed to the user is the output.

[0513] Step 9:

[0514] When a user registers a food item, the device requests the server for the optimal storage method for that food item. The storage method request is sent to the server. The sent request data is the output.

[0515] Step 10:

[0516] The server searches the database for a storage method based on the type of food and sends it to the terminal. It searches for a record that stores the appropriate storage method and transfers the storage method data. The storage method sent to the terminal is the output.

[0517] Step 11:

[0518] The terminal displays this saving method to the user. The appropriate saving conditions are displayed as text on the screen and provided to the user. The saving method displayed to the user is the output.

[0519] Step 12:

[0520] While the user is using the app, the emotion engine analyzes the user's facial expressions and voice to generate emotion data. Camera and microphone data is used as input to run emotion analysis algorithms (e.g., OpenCV, dlib) to identify the user's emotional state. The identified emotion data is the output.

[0521] Step 13:

[0522] Emotion data is sent to the server, where it is analyzed. The generated emotion data is sent to the server over the network and processed by the analysis component. The analyzed emotion information is the output.

[0523] Step 14:

[0524] Based on the analysis results, the server uses generative AI to suggest recipes that take into account the day's weather, seasonal ingredients, the user's physical condition, and emotions. A prompt is entered into the generative AI model, and the recipe generation results are received.

[0525] Example prompt sentence:

[0526] Generate a recipe for someone who is feeling tired and has the following items: carrots, chicken broth, onions, garlic.

[0527] The generated recipe data is the output.

[0528] Step 15:

[0529] The server sends the generated recipe to the terminal, which displays it to the user. The recipe information is transferred via the network and displayed on the terminal screen. The recipe displayed to the user is the output.

[0530] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0531] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0532] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0533] [Second embodiment]

[0534] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0535] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0536] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0537] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0538] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0539] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0540] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0541] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0542] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0543] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0544] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0545] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0546] The present invention is a system for effectively managing the expiration dates of food products and reducing food waste. Specific embodiments of the system are described below.

[0547] System Configuration

[0548] The system consists of a user device, a server, and a database. Users use devices such as smartphones and tablets to manage expiration dates. The server processes and stores data, and the database stores food information.

[0549] Food information registration

[0550] To register the expiration date of purchased food, a user follows these steps: the user takes a photo of the food, and the device automatically extracts the expiration date using image recognition technology. If necessary, the user can enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database.

[0551] Examples:

[0552] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[0553] Best before date notification

[0554] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, it sends a push notification, for example, three days before the expiration date. The device receives this notification and displays a warning to the user.

[0555] Examples:

[0556] The server checks the database and finds that the milk's expiration date is three days away. Because the notification period set by the user is three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[0557] Preservation method

[0558] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[0559] Examples:

[0560] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[0561] Recipe Suggestions

[0562] If there are multiple foods nearing their expiration date, the server uses generative AI technology to create recipes that combine these foods. It also suggests recipes taking into account information such as the day's weather, seasonality, and the user's physical condition. The generated recipe is sent to the device and displayed to the user.

[0563] Examples:

[0564] If a user registers "chicken" and "carrots" as foods that are close to expiry, the server will use generative AI to suggest a recipe for "chicken and carrot teriyaki rice bowl." In addition, on cold days, it will also suggest recipes such as "chicken and carrot ginger soup."

[0565] In this way, the system of the present invention automates the management of food expiration dates and supports users in using food effectively by suggesting optimal storage methods and recipes.

[0566] The processing flow will be explained below.

[0567] Step 1:

[0568] The user takes a photo of the expiration date on the food they purchased. The user uses the camera on their smartphone to take a photo of the part of the food that has the expiration date written on it.

[0569] Step 2:

[0570] The device will use image recognition to automatically extract the expiration date, and the app on the device will use image processing technology to extract the necessary date information from the image.

[0571] Step 3:

[0572] The user enters additional information such as the type of food and storage method. The user can manually enter the food name, purchase date, storage method, etc. using the app screen.

[0573] Step 4:

[0574] The terminal sends the collected food information to the server, which then packages the extracted expiration date and any additional information entered by the user and sends it to the server.

[0575] Step 5:

[0576] The server saves the food information in a database. The server analyzes the received food data and stores it in the database.

[0577] Step 6:

[0578] The server periodically checks the database to identify foods that are approaching their expiration date. A server-side program scans the database at regular intervals and selects foods that are nearing their expiration date.

[0579] Step 7:

[0580] The server prepares and sends notifications for the relevant foods to the terminal. The server generates notification messages for the identified foods and sends them to the terminal a set number of days in advance.

[0581] Step 8:

[0582] The terminal receives the notification and displays a warning to the user. The terminal receives the notification from the server and displays a message on the user's screen warning that the expiration date is approaching.

[0583] Step 9:

[0584] The user interacts with their device to learn how to store food. The user selects the option within the app to learn how to store food.

[0585] Step 10:

[0586] The device sends a request to the server regarding how to store food. The device sends a request to the server to inquire about how to store food.

[0587] Step 11:

[0588] The server retrieves information about the storage method from the database. The server retrieves the data about the storage method in the database and searches for related information.

[0589] Step 12:

[0590] The server sends information about how to save the data to the device. The server compiles the search results and sends details about how to save the data to the device.

[0591] Step 13:

[0592] The device displays the storage method to the user. The device displays the received storage method information on the app screen so that the user can check it.

[0593] Step 14:

[0594] The server sends a request to the generation AI to generate a recipe using foods that are close to their expiration date. The server creates a list of foods that are close to their expiration date and sends a request to the generation AI to generate a recipe.

[0595] Step 15:

[0596] The AI ​​generates recipes based on requests, taking into account the target food, climate, season, and the user's physical condition to create the optimal recipe.

[0597] Step 16:

[0598] The server saves the generated recipe and sends it to the device. The server temporarily saves the recipe information received from the generation AI in a database and sends it to the device.

[0599] Step 17:

[0600] The device displays the generated recipe to the user. The device displays the received recipe in the app, allowing the user to cook a dish based on it.

[0601] Through these steps, the system of the present invention helps users effectively manage food expiration dates, reduce waste, and use ingredients safely and efficiently.

[0602] Example 1

[0603] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0604] Currently, consumers must manually record the expiration dates of purchased food, which requires a great deal of effort. Furthermore, many people discard food without realizing that its expiration date is approaching, resulting in significant waste and environmental impact. Furthermore, it is difficult to maintain food quality due to lack of proper storage methods. To solve these issues, a system is needed that allows consumers to efficiently manage food, reduce waste, and properly store and use food.

[0605] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0606] In this invention, the server includes a device that photographs the expiration dates of food products and stores them as data, a device that notifies users of food products that are approaching their expiration dates, a device that suggests optimal food storage methods, a device that creates recipes that combine foods that are approaching their expiration dates, and a device that suggests recipes taking into account the weather, season, and the user's physical condition on that day. This allows consumers to efficiently manage the expiration dates of purchased food products, reduce waste, and ensure appropriate use of food products while maintaining their quality.

[0607] A "device that photographs the expiration date of food products and stores it as data" is a device that has the function of photographing the expiration date of food products purchased by a user, extracting the date information using image recognition technology, and storing the extracted data.

[0608] A "device that notifies users of food products approaching their expiration date" is a device that periodically checks food information stored in a database, identifies foods that are approaching their expiration date, and notifies users.

[0609] The "device that suggests the optimum method of preserving food" is a device that has the function of searching for the optimum method of preserving food based on the type of food registered and suggesting it to the user.

[0610] The "device for generating recipes that combine foods with approaching expiration dates" is a device that combines multiple foods that are approaching their expiration dates, generates appropriate recipes using generative AI technology, and provides them to users.

[0611] "A device that suggests recipes taking into account the weather, season, and the user's physical condition on that day" refers to a device that has the function of using generative AI technology to suggest appropriate recipes based on the current weather, season, and the user's physical condition, and providing them to the user.

[0612] The present invention is a system that aims to effectively manage food expiration dates and reduce food waste. This system is composed of a user's terminal, a server, and a database, and describes specific embodiments that include registering food information, notifying users of expiration dates, suggesting storage methods, and suggesting recipes.

[0613] System Configuration

[0614] The system allows users to manage expiration dates using devices such as smartphones and tablets. The server processes and stores data, and the database stores food information.

[0615] Food information registration

[0616] To register the expiration date of purchased food, the user first takes a photo of the food with the device. The device then uses image recognition technology (e.g., Google Cloud Vision API or Amazon Rekognition) to automatically extract the expiration date from the image. The user can then enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database (e.g., MySQL or PostgreSQL).

[0617] Examples:

[0618] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[0619] Best before date notification

[0620] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. The check is performed periodically, such as once a day. Based on the user's settings, a push notification is sent, for example, three days before the expiration date. The device receives this notification and displays a warning to the user (for example, using Firebase Cloud Messaging to send a push notification).

[0621] Examples:

[0622] The server checks the database and finds that the milk's expiration date is three days away. Because the user has set the notification period to three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[0623] Preservation method

[0624] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the optimal storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[0625] Examples:

[0626] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[0627] Recipe Suggestions

[0628] If there are multiple foods nearing their expiration date, the server uses generative AI technology (e.g., OpenAI GPT-3) to create recipes that combine these foods. Furthermore, it takes into account information such as the weather, season, and the user's physical condition on the day to suggest recipes. The generated recipe is sent to the device and displayed to the user.

[0629] Examples:

[0630] If a user registers "chicken" and "carrots" as foods that are close to expiry, the server will use generative AI to suggest a recipe for "chicken and carrot teriyaki rice bowl." In addition, on cold days, it will also suggest recipes such as "chicken and carrot ginger soup."

[0631] Example prompt sentence:

[0632] Please suggest a recipe using "chicken and carrots." The current temperature is 10 degrees and the user is feeling cold.

[0633] As described above, the system of the present invention automates the management of food expiration dates and supports users in using food effectively by suggesting optimal storage methods and recipes.

[0634] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0635] System program processing flow

[0636] Step 1:

[0637] The user takes a photo of the food they have purchased using the camera on their smartphone or tablet.

[0638] Specific behavior:

[0639] The user activates the smartphone camera and takes a picture of the food package.

[0640] Input: Food to be photographed

[0641] Output: Food image data

[0642] Step 2:

[0643] The device processes the captured image and uses image recognition technology to extract the expiration date.

[0644] Specific behavior:

[0645] The device uses the Google Cloud Vision API or Amazon Rekognition to analyze the image and extract the expiration date.

[0646] Input: Food image data

[0647] Output: Extracted best-before date data

[0648] Step 3:

[0649] If necessary, the user can input additional information such as the type of food and storage method into the terminal.

[0650] Specific behavior:

[0651] The user enters the type of food (e.g., milk) and storage method (e.g., refrigerate) into an application form.

[0652] Input: Additional information entered by the user about the type of food and storage method

[0653] Output: Complete food data (expiration date, type, storage method)

[0654] Step 4:

[0655] The terminal transmits the extracted and entered information to the server.

[0656] Specific behavior:

[0657] The device sends JSON format data to the server via an HTTP request.

[0658] Input: Complete food data

[0659] Output: Data sent to server completed

[0660] Step 5:

[0661] The server stores the received data in a database.

[0662] Specific behavior:

[0663] The server connects to a MySQL database and stores data using SQL queries.

[0664] Input: Complete food data

[0665] Output: Food data stored in a database

[0666] Step 6:

[0667] The server periodically checks the database to identify foods that are nearing their expiration date.

[0668] Specific behavior:

[0669] The server sets up a Cron job to check the expiration date in the database every day at midnight.

[0670] Input: Food data in the database

[0671] Output: A list of foods that are close to expiry date

[0672] Step 7:

[0673] The server generates a push notification about the identified food item and sends it to the device.

[0674] Specific behavior:

[0675] The server uses the Firebase Cloud Messaging API to generate a push notification and send it to the device.

[0676] Input: List of foods that are close to expiry date

[0677] Output: Push notification sent to device

[0678] Step 8:

[0679] The terminal displays the received notification to the user.

[0680] Specific behavior:

[0681] The device receives a notification and displays a pop-up message saying, "The milk's expiration date is in three days."

[0682] Input: Push notification from the server

[0683] Output: Display a warning message to the user

[0684] Step 9:

[0685] When a user registers a food item, the device requests storage instructions from the server.

[0686] Specific behavior:

[0687] The user registers "tomato" and presses the button "Please tell me how to store it." The device then requests the server how to store that food.

[0688] Input: Food information registered by the user

[0689] Output: Request sent to server

[0690] Step 10:

[0691] The server searches the database for storage methods based on the type of food and sends them to the terminal.

[0692] Specific behavior:

[0693] The server searches the "food preservation method" table for the method for preserving "tomatoes" and sends it to the terminal.

[0694] Input: Save method request from user

[0695] Output: Sending save method data

[0696] Step 11:

[0697] The terminal displays the received saving method to the user.

[0698] Specific behavior:

[0699] The device will display "How to store tomatoes: Store at room temperature, refrigerate after cutting."

[0700] Input: Save method data from the server

[0701] Output: Show user how to save

[0702] Step 12:

[0703] The server organizes information about food items that are close to their expiration date and generates recipes using a generative AI model.

[0704] Specific behavior:

[0705] The server inputs the prompt "Foods with an approaching expiration date: chicken, carrots. Please suggest a recipe using these." into the generation AI and generates a recipe.

[0706] Input: Food information about expiry date approaching, prompt text

[0707] Output: The generated recipe

[0708] Step 13:

[0709] The server sends the generated recipe to the terminal.

[0710] Specific behavior:

[0711] The server sends the recipe for "Teriyaki Chicken and Carrot Rice Bowl" to the terminal.

[0712] Input: Generated recipe

[0713] Output: Send recipe to terminal

[0714] Step 14:

[0715] The terminal displays the received recipe to the user.

[0716] Specific behavior:

[0717] The device displays "Teriyaki Rice Bowl Recipe: Chicken, Carrots, Soy Sauce, Mirin, Sugar."

[0718] Input: Recipe information from the server

[0719] Output: Recipe displayed to user

[0720] (Application example 1)

[0721] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0722] In today's food delivery services, improving the efficiency of food inventory management and reducing food waste are major challenges. In particular, insufficient expiration date management has led to widespread food waste due to expired food, resulting in economic losses and increasing the burden on the environment. Furthermore, it is difficult to propose recipes and create menus that effectively utilize food that is approaching its expiration date, increasing the burden on employees. To solve these issues, a system that automatically manages expiration dates and suggests optimal storage methods and recipes is needed.

[0723] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0724] In this invention, the server includes means for photographing the expiration dates of purchased foods and saving them as data, means for notifying users of foods approaching their expiration dates, means for suggesting optimal food storage methods, means for generating and providing recipes combining foods approaching their expiration dates using a generative AI model, means for suggesting recipes taking into consideration the weather, seasons, and the user's physical condition on that day, and means for creating menus based on foods approaching their expiration dates and supporting efficient food delivery. This enables efficient food inventory management and reduces food waste.

[0725] "Means for photographing the expiration date of purchased food and saving it as data" refers to a system that photographs the expiration date of food with a camera and automatically recognizes and saves the data.

[0726] The "means for notifying about foods approaching their expiration date" is a function for automatically detecting foods approaching their expiration date and sending a notification to the user.

[0727] A "means for suggesting the optimal method of food preservation" is a system for providing appropriate preservation methods for specific foods.

[0728] "Means for generating and providing recipes that combine food items with close expiration dates using a generative AI model" refers to a system for combining food items with close expiration dates, creating recipes using a generative AI model, and providing the recipes.

[0729] "Means for suggesting recipes taking into consideration the weather, season, and the user's physical condition on that day" is a function for suggesting appropriate recipes taking into consideration the weather, season, and the user's physical condition on that day.

[0730] "A means for creating menus based on foods with approaching expiration dates and supporting efficient food delivery" is a system for creating menus based on foods with approaching expiration dates and operating food delivery efficiently.

[0731] MODE FOR CARRYING OUT THE INVENTION

[0732] The present invention provides a system for improving the efficiency of food inventory management and reducing food waste in food delivery services. Specific embodiments of the system are described below.

[0733] System configuration

[0734] The system consists of a user's device, a server, and a database. Users use devices such as smartphones to manage food expiration dates. The server processes and stores data, and the database stores food information.

[0735] Food information registration

[0736] Users take a photo of the expiration date of purchased food using their smartphone camera. The device uses OpenCV and pytesseract for image recognition, automatically extracts the expiration date, and saves it as food information. Users can also enter additional information such as the type of food and storage method. This data is sent to the server, which stores it in a database.

[0737] Best before date notification

[0738] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, a push notification is sent, for example, three days before the expiration date. The device receives this notification and displays a warning to the user. The notification is sent using plyer.

[0739] Preservation method

[0740] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the optimal storage method for that food item and sends it to the device. The device then displays this storage method to the user.

[0741] Recipe Suggestions

[0742] If there are multiple foods approaching their expiration date, the server uses a generative AI model to create recipes that combine these foods. It also suggests recipes taking into account information such as the day's weather, seasonality, and the user's physical condition. The generated recipes are sent to the device and displayed to the user. It also creates menus based on foods approaching their expiration date, supporting the operation of the delivery service.

[0743] Specific examples

[0744] A user purchases milk and takes a photo of the label with a smartphone, which says the expiration date "December 15, 2023." The device automatically extracts this date and sends it to the server as "milk" information along with the "keep refrigerated" information. The server then stores it in a database.

[0745] When the expiration date is approaching in three days, the server sends a notification to the terminal stating "The expiration date of the milk is approaching in three days" and displays a warning to the user.

[0746] If a user registers "chicken" and "carrots" as foods that are close to their expiration date, the server will use generative AI to suggest a recipe for "chicken and carrot teriyaki rice bowl." It will also suggest recipes such as "chicken and carrot ginger soup" for cold days.

[0747] Prompt Sentence Examples

[0748] "Based on the food information below, generate a list of foods nearing their end of life and display them sorted by expiry date. Set up subsequent notifications and suggest storage methods and recipes."

[0749] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0750] Step 1: Photograph and extract food information

[0751] A user takes a photo of the expiration date of a purchased food item with their smartphone. The device then analyzes the image using OpenCV and pytesseract and extracts the expiration date as text data. The input data is the image of the food, and the output data is the extracted expiration date. This process saves the user the trouble of manually entering the data.

[0752] Step 2: Save food information

[0753] The device sends the extracted expiration date and any additional information entered by the user, such as the type of food and storage method, to the server. The server receives this information and stores it in a database. The input data is information such as the expiration date, type of food, and storage method, and the output data is the updated database. This allows for centralized management of food expiration date information.

[0754] Step 3: Regularly check expiration dates

[0755] The server periodically (e.g., daily) checks the food information in the database and identifies foods that are approaching their expiration date. This process makes it possible to quickly detect foods that are approaching their expiration date. The input data is all food information in the database, and the output data is a list of foods that are approaching their expiration date.

[0756] Step 4: Best before notification

[0757] The server creates a list of foods that are close to their expiration date and sends a push notification to the device three days before the expiration date. The device receives this notification and displays a warning to the user. The input data is a list of foods that are close to their expiration date, and the output data is a notification message to the user. This allows the user to use the food without worrying about the expiration date.

[0758] Step 5: Recommending the best storage method

[0759] When a user registers a food item, the device sends a request for storage method to the server. The server searches the database for the optimal storage method and sends it to the device. The device then displays this storage method to the user. The input data is the type of food, and the output data is the optimal storage method. This allows the quality of the food to be maintained at its optimum.

[0760] Step 6: Recipe suggestions

[0761] If there are multiple foods nearing their expiration date, the server uses a generative AI model to create a recipe that combines these foods. The recipe is also customized taking into account the weather, seasons, and the user's physical condition. The generative AI uses specific prompts. The input data is a list of foods nearing their expiration date and information about the weather and physical condition, and the output data is a suggested recipe. This reduces food waste while supporting the user's health.

[0762] As a specific example, the prompt text used is, "Based on the food information below, generate a list of foods that are nearing the end of their lifespan, and display them sorted by date for those with the nearest expiration date. Set the system to notify you in the future, and suggest storage methods and recipes."

[0763] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0764] The present invention is a system that aims to effectively manage food expiration dates and reduce food waste. Furthermore, by combining it with an emotion engine that recognizes the user's emotions and adding a function that suggests recipes based on the user's emotions, the user experience is improved. Specific embodiments of the system are described below.

[0765] System Configuration

[0766] The system consists of a user device, a server, a database, and an emotion engine. Users use devices such as smartphones and tablets to manage expiration dates and recognize emotions. The server processes and stores data, and the database stores information about food and emotions. The emotion engine is a dedicated component for recognizing and analyzing user emotions.

[0767] Food information registration

[0768] To register the expiration date of purchased food, a user follows these steps: the user takes a photo of the food, and the device automatically extracts the expiration date using image recognition technology. If necessary, the user can enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database.

[0769] Examples:

[0770] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[0771] Best before date notification

[0772] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, it sends a push notification, for example, three days before the expiration date. The device receives this notification and displays a warning to the user.

[0773] Examples:

[0774] The server checks the database and finds that the milk's expiration date is three days away. Because the notification period set by the user is three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[0775] Preservation method

[0776] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[0777] Examples:

[0778] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[0779] Emotion Recognition and Recipe Suggestion

[0780] The app uses cameras and sensors installed on the device to recognize the user's emotions. While the user is using the app, the emotion engine analyzes the user's facial expressions and voice to generate emotion data. This data is sent to the server for analysis. Based on the analysis results, the server uses generative AI to suggest recipes that take into account the day's weather, seasonal ingredients, the user's physical condition, and emotions.

[0781] Examples:

[0782] While the user is using the app, the emotion engine recognizes the user's facial expressions and generates emotional data such as "tired." Based on this information, the server uses generative AI to suggest recipes such as "chicken and carrot soup with fatigue-relieving effects." These are displayed on the user's device, and the user can use the recipe to cook the dish.

[0783] In this way, the system of the present invention automates food expiration date management, improves user experience through emotion-based recipe suggestions, reduces food waste, and supports safe and efficient use of ingredients.

[0784] The processing flow will be explained below.

[0785] Step 1:

[0786] The user takes a photo of the expiration date on the food product they purchased. The user uses the camera on their smartphone to take a photo of the expiration date printed on the food product label.

[0787] Step 2:

[0788] The device will perform image recognition and automatically extract the expiration date, while the app on the device will use image processing technology to analyze and extract the expiration date from the image.

[0789] Step 3:

[0790] The user enters additional information such as the type of food and storage method. The user enters the name of the food, purchase date, and storage method through the app screen.

[0791] Step 4:

[0792] The terminal sends the collected food information to the server, which then packages the extracted expiration date and any additional information entered by the user and sends it to the server.

[0793] Step 5:

[0794] The server stores the food information in a database. The server analyzes the received food data and stores it in a database.

[0795] Step 6:

[0796] The server periodically checks the database to identify foods that are nearing their expiration date. A program on the server periodically scans the database to identify foods that are nearing their expiration date.

[0797] Step 7:

[0798] The server prepares notifications for the relevant foods and sends them to the terminal. The server generates notification messages for the identified foods and sends them to the terminal a set number of days in advance.

[0799] Step 8:

[0800] The terminal receives the notification and displays a warning to the user. The terminal receives the notification from the server and displays a warning message on the user's screen informing them that the expiration date is approaching.

[0801] Step 9:

[0802] The user operates the device to check how to store food. The user selects an option in the device's app to check how to store food.

[0803] Step 10:

[0804] The device sends a request to the server about how to store food.The device sends a request to the server about how to store food.

[0805] Step 11:

[0806] The server searches the database for information on the storage method. The server searches the database for information on the relevant storage method based on the request.

[0807] Step 12:

[0808] The server sends information on how to save the data to the device. The server compiles the search results and sends information on how to save the data to the device.

[0809] Step 13:

[0810] The device displays the storage method to the user. The device displays the received storage method information on the app screen so that the user can check it.

[0811] Step 14:

[0812] The device uses cameras and sensors to recognize the user's emotions, and the emotion engine analyzes the user's facial expressions and voice to generate emotion data.

[0813] Step 15:

[0814] The device sends the generated emotion data to the server. The device packages the user's emotion data and sends it to the server.

[0815] Step 16:

[0816] The server analyzes the user's emotional data. The server analyzes the received emotional data and identifies the user's emotional state.

[0817] Step 17:

[0818] The server takes into consideration emotional data, weather, seasonality, and the user's physical condition, and sends a request to the generation AI to generate a recipe. The server then sends a recipe request to the generation AI based on this information.

[0819] Step 18:

[0820] The generative AI generates the optimal recipe. Based on the information it receives from the server, the generative AI creates a recipe that takes into account the user's emotional state and other conditions.

[0821] Step 19:

[0822] The server saves the generated recipe and sends it to the device. The server temporarily saves the recipe information received from the generation AI in a database and sends it to the device.

[0823] Step 20:

[0824] The device displays the generated recipe to the user. The device displays the received recipe within the app, allowing the user to use the recipe as a reference when cooking.

[0825] Through these steps, the system of the present invention automates food expiration date management, improves user experience through emotion-based recipe suggestions, reduces food waste, and supports safe and efficient ingredient use.

[0826] Example 2

[0827] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0828] In modern society, the increase in food waste has become a major problem. This problem is particularly caused by poor expiration date management within the home. Furthermore, there is a lack of meal suggestions that take into account the user's emotions and physical condition, and there is a need for methods to use ingredients appropriately and without waste. Conventional systems have difficulty meeting these requirements, so an effective solution is needed to simultaneously manage food expiration dates and improve the user experience.

[0829] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0830] In this invention, the server includes a means for photographing the expiration dates of purchased items and storing them as data, a means for notifying the user of items approaching their expiration dates, a means for suggesting optimal storage methods for the items, and a means for analyzing the user's emotions, thereby reducing food waste and improving the user experience through emotion-based meal suggestions.

[0831] "Goods" refers to food or consumable items purchased or acquired by the user.

[0832] "Best before date" refers to the period during which an item can be consumed while maintaining its quality.

[0833] "Image recognition technology" refers to a form of computer vision that extracts and identifies specific information from image data.

[0834] "Data" refers to electromagnetically recorded information, including information such as expiration date, type of product, and storage method.

[0835] "Notification" refers to an alert or message that is displayed on a user's device.

[0836] "Preservation method" refers to the specific procedures and conditions for preserving an item in optimal condition.

[0837] "Cooking method" refers to the steps and methods for creating a dish using an item.

[0838] "Climate" refers to the weather conditions and temperature of the day.

[0839] "Season" refers to the time when something is at its most delicious and nutritious.

[0840] "User" refers to the consumer or end user who uses the system.

[0841] "Emotion" refers to the psychological state detected from the user's facial expressions, voice, etc.

[0842] "Emotion engine" is a general term for the algorithms and hardware that analyze the user's emotions and generate the results as data.

[0843] "Server" refers to a computer system that processes, stores, and analyzes data.

[0844] "Database" refers to a digital storage system for systematically storing and managing information such as expiration dates, storage methods, and emotional data.

[0845] "Generative AI model" refers to a machine learning model that generates useful results (such as recipes) based on input prompts.

[0846] A "prompt sentence" refers to an instruction sentence used as input to a generative AI model.

[0847] The present invention is a system that allows users to manage food expiration dates and receive cooking method suggestions based on their emotions. This system is composed of a terminal, a server, a database, and an emotion engine. Specific embodiments of the system are described below.

[0848] System Configuration

[0849] The system consists of a user device (smartphone or tablet), a server, a database, and an emotion engine. The server uses a powerful computer system to process, store, and analyze data. The database is a digital storage system for systematically storing and managing information such as expiration dates, storage methods, and emotion data. The emotion engine consists of hardware and software components for analyzing user emotions and generating the results as data.

[0850] Food information registration

[0851] To register the expiration date of purchased food, a user takes a photo of the food label using the device's camera. The device then uses image recognition technology to extract the expiration date as text data from the captured image. If necessary, the user can manually enter the type of food and storage method. The device then formats this data and sends an HTTP request to the server. The server then analyzes the received data and stores it in a database in the required format.

[0852] Best before date notification

[0853] The server periodically checks the database to identify foods that are approaching their expiration date. Based on user settings, the server creates and sends a notification three days before the expiration date, for example. The device receives this notification and displays a warning message to the user, such as "The milk's expiration date is three days away."

[0854] Preservation method

[0855] When a user registers a food item, the device requests the server for the optimal storage method for that food item. The server searches the database for the optimal storage method for that food item and sends it to the device as an HTTP response. The device then presents the received storage method to the user, displaying specific storage instructions such as "Store tomatoes at room temperature, or refrigerate if cut."

[0856] Emotion Recognition and Recipe Suggestion

[0857] While the user is using the app, emotion data is collected using the device's on-board camera and sensors. The device then analyzes the collected emotion data using an emotion engine and sends the results to the server. The server generates prompts based on the emotion data and other factors (weather, season, physical condition, etc.) and inputs them into the generative AI model. The server receives the resulting recipes generated by the generative AI model and sends them to the device. The device then presents specific recipes to the user, such as "chicken and carrot soup with fatigue-relieving effects."

[0858] Examples of prompt statements

[0859] Example prompts to input to a generative AI model:

[0860] "What recipes would you recommend to users who are feeling tired? For example, a simple soup recipe using foods that have a fatigue-relieving effect."

[0861] In this way, the system of the present invention automates food expiration date management, improves user experience through emotion-based cooking suggestions, reduces food waste, and supports safe and efficient use of ingredients.

[0862] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0863] Step 1:

[0864] The user takes a photo of the food they purchased on their device.

[0865] Input: A user takes a photo of a food label with their smartphone camera.

[0866] How it works: The user launches the camera app, frames a food label with an expiration date, and presses the capture button.

[0867] Output: The image data of the photographed food label is saved on the device.

[0868] Step 2:

[0869] The device uses image recognition technology to extract the expiration date.

[0870] Input: Image data acquired in step 1.

[0871] How it works: The device uses OCR (optical character recognition) technology to analyze the text data in the image and extract the expiration date as text data.

[0872] Output: Best before date data is obtained in text format.

[0873] Step 3:

[0874] The user enters any additional information needed (such as the type of food and storage method)

[0875] Input: Best before date data obtained in step 2 and manual user input information.

[0876] What it does: A user manually enters the type of food (e.g., milk, bread) and storage method (e.g., refrigerate, freeze) using an app input form.

[0877] Output: The expiration date data and the type of food and storage method data entered by the user are obtained.

[0878] Step 4:

[0879] The device sends the data to the server

[0880] Input: Best before date data obtained in step 3, food type, and storage method data.

[0881] How it works: The device formats this data and sends an HTTP request to the server.

[0882] Output: The data is sent to the server.

[0883] Step 5:

[0884] The server stores the received information in a database

[0885] Input: The data sent from the terminal in step 4.

[0886] How it works: The server parses the received data and stores it in the required format in the database.

[0887] Output: The expiration date, food type, and storage method data are saved in the database.

[0888] Step 6:

[0889] The server periodically checks the database

[0890] Input: Food information in the database.

[0891] How it works: The server runs a periodic job that checks all food items in the database.

[0892] Output: A list of foods that are nearing their expiration date is generated.

[0893] Step 7:

[0894] The server identifies food that is nearing its expiration date

[0895] Input: The list of foods obtained in step 6.

[0896] How it works: The server compares the expiration dates of the foods with the current date to identify foods that require notification.

[0897] Output: A list of foods that require notification is generated.

[0898] Step 8:

[0899] The server creates and sends the push notification

[0900] Input: The list of foods requiring notifications obtained in step 7.

[0901] How it works: The server creates a notification based on the user's settings, for example, three days before the expiration date, and sends a push notification to the user's device.

[0902] Output: A push notification is sent to the user's device.

[0903] Step 9:

[0904] The device receives the notification and displays it to the user.

[0905] Input: The push notification sent in step 8.

[0906] What it does: The device receives a notification from the server and displays a warning message such as "The milk's expiration date is coming up in three days."

[0907] Output: A notification message is displayed on the device screen.

[0908] Step 10:

[0909] Send a storage request when a user registers a food item

[0910] Input: The user enters the name of the food.

[0911] How it works: The user enters the name of a food item into the app's input form and presses the "Get storage instructions" button. The device then sends a request to the server.

[0912] Output: A save method request is sent to the server.

[0913] Step 11:

[0914] The server looks up the save method in the database

[0915] Input: The request submitted in step 10.

[0916] How it works: The server receives a request and searches its database for the best way to store the food.

[0917] Output: The search results for the saving method are obtained.

[0918] Step 12:

[0919] The server sends the save instructions to the device.

[0920] Input: Search results for saving methods obtained in step 11.

[0921] How it works: The server formats the search results and sends them to the device as an HTTP response.

[0922] Output: Information on how to save is sent to the device.

[0923] Step 13:

[0924] The device will prompt the user for instructions on how to save.

[0925] Input: The save method information submitted in step 12.

[0926] Operation: The device presents the received storage instructions to the user, displaying specific storage instructions such as "Store tomatoes at room temperature, or refrigerate if cut."

[0927] Output: Information about the saving method will be displayed on the device screen.

[0928] Step 14:

[0929] The device collects emotion data while the user is using the app.

[0930] Input: User's facial and voice data.

[0931] How it works: The user opens the app and uses the camera and sensors to capture facial expressions and voice in real time. The device then captures this emotional data.

[0932] Output: Emotion data is saved on the device.

[0933] Step 15:

[0934] The device analyzes the emotional data it acquires and sends it to the server.

[0935] Input: Emotion data obtained in step 14.

[0936] Operation: The device analyzes the facial expression data using the emotion engine, extracts the emotional state (e.g., "tired"), and sends it to the server.

[0937] Output: The analyzed emotion data is sent to the server.

[0938] Step 16:

[0939] The server generates a prompt and inputs it into the generative AI model.

[0940] Input: Emotion data sent in step 15 and other factors (weather, season, physical condition information).

[0941] How it works: The server uses this information to generate a prompt (e.g., "What recipes would you recommend for a tired user?") and inputs it into the generative AI model.

[0942] Output: The generated prompt sentence is input to the generative AI model.

[0943] Step 17:

[0944] The server sends the generated recipe to the device.

[0945] Input: The recipe obtained from the generative AI model in step 16.

[0946] How it works: The server receives the recipe generated by the AI ​​model and sends it to the device.

[0947] Output: Recipe data is sent to the device.

[0948] Step 18:

[0949] The device displays the recipe to the user

[0950] Input: Recipe data submitted in step 17.

[0951] How it works: The device displays the recipe it receives from the server on the screen, and suggests specific recipes such as "chicken and carrot soup with fatigue-relieving effects."

[0952] Output: The recipe is displayed on the device screen and provided to the user.

[0953] In this way, by linking each processing step with each other, the system of the present invention can effectively manage food expiration dates, suggest storage methods, and suggest cooking methods based on emotions.

[0954] (Application example 2)

[0955] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0956] In recent years, with the increasing importance of managing food expiration dates, there is a demand for suggestions of appropriate storage methods and recipes using food that is close to its expiration date. However, because individual factors such as emotions and physical condition are not taken into consideration, the user experience is often unsatisfactory. Furthermore, expiration date management is sometimes not thoroughly implemented in physical stores, leading to a serious problem of food waste. To solve these issues, a system that reflects user emotions and can also manage expiration dates in physical stores is needed.

[0957] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for photographing the expiration dates of purchased foods and saving them as data, means for notifying users of foods approaching their expiration dates, means for suggesting optimal food storage methods, means for providing recipes that combine foods with upcoming expiration dates, means for suggesting recipes taking into account the weather, seasons, and the user's physical condition on that day, means for recognizing the user's emotions and suggesting recipes based on the emotion data, and means for managing food in a physical store and suggesting recipes based on the emotions. This makes it possible to suggest recipes that correspond to the emotions of each individual user, reducing food waste and enabling efficient food management in physical stores.

[0958] The "best before date" of a food product is the date by which the food product will continue to maintain its appropriate quality.

[0959] "Storing as data" means electronically recording information such as expiration dates and making them accessible later.

[0960] "Notifying" means sending a warning or alert to the user when a specific condition is met.

[0961] "Best storage method" refers to the recommended storage environment and conditions to maintain the quality of food.

[0962] A "recipe" is a list of steps and ingredients for making a dish.

[0963] "Climate" refers to weather conditions such as temperature, humidity, wind speed, and precipitation over a certain period of time.

[0964] "Season" refers to the time when a particular food is most nutritious and delicious.

[0965] "User's physical condition" refers to the user's health condition and current physical condition.

[0966] "Emotion data" refers to emotional information obtained by analyzing the user's facial expressions, voice, behavior, etc.

[0967] A "brick and mortar store" is a physical retail establishment where consumers can visit and purchase products in person.

[0968] "Emotion recognition" means judging the user's emotional state by analyzing the user's facial expressions, etc.

[0969] "Emotion-based recipe suggestions" refers to taking into account the user's current emotional state and providing cooking ideas that are appropriate for that emotion.

[0970] "Managing food" means centrally monitoring and controlling food inventory, expiration dates, storage methods, etc.

[0971] "Image recognition technology" is a technology that analyzes image data and automatically recognizes specific information (such as characters or objects).

[0972] A "server" is a computer system that stores, processes, and manages data.

[0973] "Suggesting storage methods" means showing the user the best storage conditions for a particular food item.

[0974] "Providing recipe combinations" means teaching users how to combine multiple foods into one dish.

[0975] "User-specified" means that the user inputs his / her wishes and conditions into the system and makes a selection.

[0976] "Push notification" is a mechanism that notifies users of specific information in a timely manner.

[0977] This invention aims to effectively manage the expiration dates of food purchased by users and reduce food waste. Furthermore, by combining it with an emotion engine that recognizes user emotions, it can suggest recipes based on emotions, improving the user experience.

[0978] System Configuration

[0979] The system consists of a user device, a server, a database, and an emotion engine. Users use devices such as smartphones and tablets to manage expiration dates and recognize emotions. The server processes and stores data, and the database stores information about food and emotions. The emotion engine is a dedicated component for recognizing and analyzing user emotions.

[0980] Food information registration

[0981] To register the expiration date of purchased food, a user follows these steps: the user takes a photo of the food, and the device automatically extracts the expiration date using image recognition technology. If necessary, the user can enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database.

[0982] Specific examples

[0983] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[0984] Best before date notification

[0985] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, it sends a push notification, for example, three days before the expiration date. The device receives this notification and displays a warning to the user.

[0986] Specific examples

[0987] The server checks the database and finds that the milk's expiration date is three days away. Because the notification period set by the user is three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[0988] Preservation method

[0989] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[0990] Specific examples

[0991] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[0992] Emotion Recognition and Recipe Suggestion

[0993] The app uses cameras and sensors installed on the device to recognize the user's emotions. While the user is using the app, the emotion engine analyzes the user's facial expressions and voice to generate emotion data. This data is sent to the server for analysis. Based on the analysis results, the server uses generative AI to suggest recipes that take into account the day's weather, seasonal ingredients, the user's physical condition, and emotions.

[0994] Specific examples

[0995] While the user is using the app, the emotion engine recognizes the user's facial expressions and generates emotional data such as "tired." Based on this information, the server uses generative AI to suggest recipes such as "chicken and carrot soup with fatigue-relieving effects." These are displayed on the user's device, and the user can use the recipe to cook the dish.

[0996] Examples of prompts to input to a generative AI model:

[0997] Generate a recipe for someone who is feeling tired and has the following items: carrots, chicken broth, onions, garlic.

[0998] Application in physical stores

[0999] This system can also be applied in physical stores. Employees and customers can use their smartphones to manage food in the store and suggest recipes based on their emotions. This will make it easier to manage expiration dates in the store and is expected to reduce food waste.

[1000] In this way, the system of the present invention automates food expiration date management, improves user experience through emotion-based recipe suggestions, reduces food waste, and supports safe and efficient use of ingredients.

[1001] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1002] Step 1:

[1003] The user takes a photo of the food they have purchased. Using the device, they take a photo of the label containing the expiration date with a camera, and the image data is used as input.

[1004] Step 2:

[1005] The device automatically extracts the expiration date using image recognition technology. It uses an image recognition algorithm (e.g., OpenCV) to analyze the characters in the image and identify the expiration date. The identified expiration date is the output.

[1006] Step 3:

[1007] If necessary, the user inputs additional information, such as the type of food or storage method. The information is supplemented through the device's user interface with text input or selection menus. The input additional information is output.

[1008] Step 4:

[1009] The device sends this data (best before date, food type, storage method) to the server. The data is transferred to the server through the network. The raw data sent to the server is the output.

[1010] Step 5:

[1011] The server stores the received data in a database. The data is structured and stored using a database management system (e.g., SQLite). The food information stored in the database is the output.

[1012] Step 6:

[1013] The server periodically checks the food information in the database to identify foods that are approaching their expiration date. It then runs a query to extract foods with an expiration date within three days. The extracted list of foods is the output.

[1014] Step 7:

[1015] The server sends push notifications for the identified foods based on the user settings. If the notification condition (e.g., three days before) is met, the server generates the notification content and sends it to the user's device. The notification message displayed on the device is the output.

[1016] Step 8:

[1017] The terminal receives the notification and displays a warning to the user. The notification message is displayed in a popup format to get the user's immediate attention. The warning displayed to the user is the output.

[1018] Step 9:

[1019] When a user registers a food item, the device requests the server for the optimal storage method for that food item. The storage method request is sent to the server. The sent request data is the output.

[1020] Step 10:

[1021] The server searches the database for a storage method based on the type of food and sends it to the terminal. It searches for a record that stores the appropriate storage method and transfers the storage method data. The storage method sent to the terminal is the output.

[1022] Step 11:

[1023] The terminal displays this saving method to the user. The appropriate saving conditions are displayed as text on the screen and provided to the user. The saving method displayed to the user is the output.

[1024] Step 12:

[1025] While the user is using the app, the emotion engine analyzes the user's facial expressions and voice to generate emotion data. Camera and microphone data is used as input to run emotion analysis algorithms (e.g., OpenCV, dlib) to identify the user's emotional state. The identified emotion data is the output.

[1026] Step 13:

[1027] Emotion data is sent to the server, where it is analyzed. The generated emotion data is sent to the server over the network and processed by the analysis component. The analyzed emotion information is the output.

[1028] Step 14:

[1029] Based on the analysis results, the server uses generative AI to suggest recipes that take into account the day's weather, seasonal ingredients, the user's physical condition, and emotions. A prompt is entered into the generative AI model, and the recipe generation results are received.

[1030] Example prompt sentence:

[1031] Generate a recipe for someone who is feeling tired and has the following items: carrots, chicken broth, onions, garlic.

[1032] The generated recipe data is the output.

[1033] Step 15:

[1034] The server sends the generated recipe to the terminal, which displays it to the user. The recipe information is transferred via the network and displayed on the terminal screen. The recipe displayed to the user is the output.

[1035] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1036] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1037] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1038] [Third embodiment]

[1039] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1040] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1041] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1042] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1043] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1044] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1045] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1046] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1047] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1048] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1049] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1050] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1051] The present invention is a system for effectively managing the expiration dates of food products and reducing food waste. Specific embodiments of the system are described below.

[1052] System Configuration

[1053] The system consists of a user device, a server, and a database. Users use devices such as smartphones and tablets to manage expiration dates. The server processes and stores data, and the database stores food information.

[1054] Food information registration

[1055] To register the expiration date of purchased food, a user follows these steps: the user takes a photo of the food, and the device automatically extracts the expiration date using image recognition technology. If necessary, the user can enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database.

[1056] Examples:

[1057] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[1058] Best before date notification

[1059] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, it sends a push notification, for example, three days before the expiration date. The device receives this notification and displays a warning to the user.

[1060] Examples:

[1061] The server checks the database and finds that the milk's expiration date is three days away. Because the notification period set by the user is three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[1062] Preservation method

[1063] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[1064] Examples:

[1065] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[1066] Recipe Suggestions

[1067] If there are multiple foods nearing their expiration date, the server uses generative AI technology to create recipes that combine these foods. It also suggests recipes taking into account information such as the day's weather, seasonality, and the user's physical condition. The generated recipe is sent to the device and displayed to the user.

[1068] Examples:

[1069] If a user registers "chicken" and "carrots" as foods that are close to expiry, the server will use generative AI to suggest a recipe for "chicken and carrot teriyaki rice bowl." In addition, on cold days, it will also suggest recipes such as "chicken and carrot ginger soup."

[1070] In this way, the system of the present invention automates the management of food expiration dates and supports users in using food effectively by suggesting optimal storage methods and recipes.

[1071] The processing flow will be explained below.

[1072] Step 1:

[1073] The user takes a photo of the expiration date on the food they purchased. The user uses the camera on their smartphone to take a photo of the part of the food that has the expiration date written on it.

[1074] Step 2:

[1075] The device will use image recognition to automatically extract the expiration date, and the app on the device will use image processing technology to extract the necessary date information from the image.

[1076] Step 3:

[1077] The user enters additional information such as the type of food and storage method. The user can manually enter the food name, purchase date, storage method, etc. using the app screen.

[1078] Step 4:

[1079] The terminal sends the collected food information to the server, which then packages the extracted expiration date and any additional information entered by the user and sends it to the server.

[1080] Step 5:

[1081] The server saves the food information in a database. The server analyzes the received food data and stores it in the database.

[1082] Step 6:

[1083] The server periodically checks the database to identify foods that are approaching their expiration date. A server-side program scans the database at regular intervals and selects foods that are nearing their expiration date.

[1084] Step 7:

[1085] The server prepares and sends notifications for the relevant foods to the terminal. The server generates notification messages for the identified foods and sends them to the terminal a set number of days in advance.

[1086] Step 8:

[1087] The terminal receives the notification and displays a warning to the user. The terminal receives the notification from the server and displays a message on the user's screen warning that the expiration date is approaching.

[1088] Step 9:

[1089] The user interacts with their device to learn how to store food. The user selects the option within the app to learn how to store food.

[1090] Step 10:

[1091] The device sends a request to the server regarding how to store food. The device sends a request to the server to inquire about how to store food.

[1092] Step 11:

[1093] The server retrieves information about the storage method from the database. The server retrieves the data about the storage method in the database and searches for related information.

[1094] Step 12:

[1095] The server sends information about how to save the data to the device. The server compiles the search results and sends details about how to save the data to the device.

[1096] Step 13:

[1097] The device displays the storage method to the user. The device displays the received storage method information on the app screen so that the user can check it.

[1098] Step 14:

[1099] The server sends a request to the generation AI to generate a recipe using foods that are close to their expiration date. The server creates a list of foods that are close to their expiration date and sends a request to the generation AI to generate a recipe.

[1100] Step 15:

[1101] The AI ​​generates recipes based on requests, taking into account the target food, climate, season, and the user's physical condition to create the optimal recipe.

[1102] Step 16:

[1103] The server saves the generated recipe and sends it to the device. The server temporarily saves the recipe information received from the generation AI in a database and sends it to the device.

[1104] Step 17:

[1105] The device displays the generated recipe to the user. The device displays the received recipe in the app, allowing the user to cook a dish based on it.

[1106] Through these steps, the system of the present invention helps users effectively manage food expiration dates, reduce waste, and use ingredients safely and efficiently.

[1107] Example 1

[1108] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1109] Currently, consumers must manually record the expiration dates of purchased food, which requires a great deal of effort. Furthermore, many people discard food without realizing that its expiration date is approaching, resulting in significant waste and environmental impact. Furthermore, it is difficult to maintain food quality due to lack of proper storage methods. To solve these issues, a system is needed that allows consumers to efficiently manage food, reduce waste, and properly store and use food.

[1110] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1111] In this invention, the server includes a device that photographs the expiration dates of food products and stores them as data, a device that notifies users of food products that are approaching their expiration dates, a device that suggests optimal food storage methods, a device that creates recipes that combine foods that are approaching their expiration dates, and a device that suggests recipes taking into account the weather, season, and the user's physical condition on that day. This allows consumers to efficiently manage the expiration dates of purchased food products, reduce waste, and ensure appropriate use of food products while maintaining their quality.

[1112] A "device that photographs the expiration date of food products and stores it as data" is a device that has the function of photographing the expiration date of food products purchased by a user, extracting the date information using image recognition technology, and storing the extracted data.

[1113] A "device that notifies users of food products approaching their expiration date" is a device that periodically checks food information stored in a database, identifies foods that are approaching their expiration date, and notifies users.

[1114] The "device that suggests the optimum method of preserving food" is a device that has the function of searching for the optimum method of preserving food based on the type of food registered and suggesting it to the user.

[1115] The "device for generating recipes that combine foods with approaching expiration dates" is a device that combines multiple foods that are approaching their expiration dates, generates appropriate recipes using generative AI technology, and provides them to users.

[1116] "A device that suggests recipes taking into account the weather, season, and the user's physical condition on that day" refers to a device that has the function of using generative AI technology to suggest appropriate recipes based on the current weather, season, and the user's physical condition, and providing them to the user.

[1117] The present invention is a system that aims to effectively manage food expiration dates and reduce food waste. This system is composed of a user's terminal, a server, and a database, and describes specific embodiments that include registering food information, notifying users of expiration dates, suggesting storage methods, and suggesting recipes.

[1118] System Configuration

[1119] The system allows users to manage expiration dates using devices such as smartphones and tablets. The server processes and stores data, and the database stores food information.

[1120] Food information registration

[1121] To register the expiration date of purchased food, the user first takes a photo of the food with the device. The device then uses image recognition technology (e.g., Google Cloud Vision API or Amazon Rekognition) to automatically extract the expiration date from the image. The user can then enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database (e.g., MySQL or PostgreSQL).

[1122] Examples:

[1123] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[1124] Best before date notification

[1125] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. The check is performed periodically, such as once a day. Based on the user's settings, a push notification is sent, for example, three days before the expiration date. The device receives this notification and displays a warning to the user (for example, using Firebase Cloud Messaging to send a push notification).

[1126] Examples:

[1127] The server checks the database and finds that the milk's expiration date is three days away. Because the user has set the notification period to three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[1128] Preservation method

[1129] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the optimal storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[1130] Examples:

[1131] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[1132] Recipe Suggestions

[1133] If there are multiple foods nearing their expiration date, the server uses generative AI technology (e.g., OpenAI GPT-3) to create recipes that combine these foods. Furthermore, it takes into account information such as the weather, season, and the user's physical condition on the day to suggest recipes. The generated recipe is sent to the device and displayed to the user.

[1134] Examples:

[1135] If a user registers "chicken" and "carrots" as foods that are close to expiry, the server will use generative AI to suggest a recipe for "chicken and carrot teriyaki rice bowl." In addition, on cold days, it will also suggest recipes such as "chicken and carrot ginger soup."

[1136] Example prompt sentence:

[1137] Please suggest a recipe using "chicken and carrots." The current temperature is 10 degrees and the user is feeling cold.

[1138] As described above, the system of the present invention automates the management of food expiration dates and supports users in using food effectively by suggesting optimal storage methods and recipes.

[1139] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1140] System program processing flow

[1141] Step 1:

[1142] The user takes a photo of the food they have purchased using the camera on their smartphone or tablet.

[1143] Specific behavior:

[1144] The user activates the smartphone camera and takes a picture of the food package.

[1145] Input: Food to be photographed

[1146] Output: Food image data

[1147] Step 2:

[1148] The device processes the captured image and uses image recognition technology to extract the expiration date.

[1149] Specific behavior:

[1150] The device uses the Google Cloud Vision API or Amazon Rekognition to analyze the image and extract the expiration date.

[1151] Input: Food image data

[1152] Output: Extracted best-before date data

[1153] Step 3:

[1154] If necessary, the user can input additional information such as the type of food and storage method into the terminal.

[1155] Specific behavior:

[1156] The user enters the type of food (e.g., milk) and storage method (e.g., refrigerate) into an application form.

[1157] Input: Additional information entered by the user about the type of food and storage method

[1158] Output: Complete food data (expiration date, type, storage method)

[1159] Step 4:

[1160] The terminal transmits the extracted and entered information to the server.

[1161] Specific behavior:

[1162] The device sends JSON format data to the server via an HTTP request.

[1163] Input: Complete food data

[1164] Output: Data sent to server completed

[1165] Step 5:

[1166] The server stores the received data in a database.

[1167] Specific behavior:

[1168] The server connects to a MySQL database and stores data using SQL queries.

[1169] Input: Complete food data

[1170] Output: Food data stored in a database

[1171] Step 6:

[1172] The server periodically checks the database to identify foods that are nearing their expiration date.

[1173] Specific behavior:

[1174] The server sets up a Cron job to check the expiration date in the database every day at midnight.

[1175] Input: Food data in the database

[1176] Output: A list of foods that are close to expiry date

[1177] Step 7:

[1178] The server generates a push notification about the identified food item and sends it to the device.

[1179] Specific behavior:

[1180] The server uses the Firebase Cloud Messaging API to generate a push notification and send it to the device.

[1181] Input: List of foods that are close to expiry date

[1182] Output: Push notification sent to device

[1183] Step 8:

[1184] The terminal displays the received notification to the user.

[1185] Specific behavior:

[1186] The device receives a notification and displays a pop-up message saying, "The milk's expiration date is in three days."

[1187] Input: Push notification from the server

[1188] Output: Display a warning message to the user

[1189] Step 9:

[1190] When a user registers a food item, the device requests storage instructions from the server.

[1191] Specific behavior:

[1192] The user registers "tomato" and presses the button "Please tell me how to store it." The device then requests the server how to store that food.

[1193] Input: Food information registered by the user

[1194] Output: Request sent to server

[1195] Step 10:

[1196] The server searches the database for storage methods based on the type of food and sends them to the terminal.

[1197] Specific behavior:

[1198] The server searches the "food preservation method" table for the method for preserving "tomatoes" and sends it to the terminal.

[1199] Input: Save method request from user

[1200] Output: Sending save method data

[1201] Step 11:

[1202] The terminal displays the received saving method to the user.

[1203] Specific behavior:

[1204] The device will display "How to store tomatoes: Store at room temperature, refrigerate after cutting."

[1205] Input: Save method data from the server

[1206] Output: Show user how to save

[1207] Step 12:

[1208] The server organizes information about food items that are close to their expiration date and generates recipes using a generative AI model.

[1209] Specific behavior:

[1210] The server inputs the prompt "Foods with an approaching expiration date: chicken, carrots. Please suggest a recipe using these." into the generation AI and generates a recipe.

[1211] Input: Food information about expiry date approaching, prompt text

[1212] Output: The generated recipe

[1213] Step 13:

[1214] The server sends the generated recipe to the terminal.

[1215] Specific behavior:

[1216] The server sends the recipe for "Teriyaki Chicken and Carrot Rice Bowl" to the terminal.

[1217] Input: Generated recipe

[1218] Output: Send recipe to terminal

[1219] Step 14:

[1220] The terminal displays the received recipe to the user.

[1221] Specific behavior:

[1222] The device displays "Teriyaki Rice Bowl Recipe: Chicken, Carrots, Soy Sauce, Mirin, Sugar."

[1223] Input: Recipe information from the server

[1224] Output: Recipe displayed to user

[1225] (Application example 1)

[1226] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1227] In today's food delivery services, improving the efficiency of food inventory management and reducing food waste are major challenges. In particular, insufficient expiration date management has led to widespread food waste due to expired food, resulting in economic losses and increasing the burden on the environment. Furthermore, it is difficult to propose recipes and create menus that effectively utilize food that is approaching its expiration date, increasing the burden on employees. To solve these issues, a system that automatically manages expiration dates and suggests optimal storage methods and recipes is needed.

[1228] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1229] In this invention, the server includes means for photographing the expiration dates of purchased foods and saving them as data, means for notifying users of foods approaching their expiration dates, means for suggesting optimal food storage methods, means for generating and providing recipes combining foods approaching their expiration dates using a generative AI model, means for suggesting recipes taking into consideration the weather, seasons, and the user's physical condition on that day, and means for creating menus based on foods approaching their expiration dates and supporting efficient food delivery. This enables efficient food inventory management and reduces food waste.

[1230] "Means for photographing the expiration date of purchased food and saving it as data" refers to a system that photographs the expiration date of food with a camera and automatically recognizes and saves the data.

[1231] The "means for notifying about foods approaching their expiration date" is a function for automatically detecting foods approaching their expiration date and sending a notification to the user.

[1232] A "means for suggesting the optimal method of food preservation" is a system for providing appropriate preservation methods for specific foods.

[1233] "Means for generating and providing recipes that combine food items with close expiration dates using a generative AI model" refers to a system for combining food items with close expiration dates, creating recipes using a generative AI model, and providing the recipes.

[1234] "Means for suggesting recipes taking into consideration the weather, season, and the user's physical condition on that day" is a function for suggesting appropriate recipes taking into consideration the weather, season, and the user's physical condition on that day.

[1235] "A means for creating menus based on foods with approaching expiration dates and supporting efficient food delivery" is a system for creating menus based on foods with approaching expiration dates and operating food delivery efficiently.

[1236] MODE FOR CARRYING OUT THE INVENTION

[1237] The present invention provides a system for improving the efficiency of food inventory management and reducing food waste in food delivery services. Specific embodiments of the system are described below.

[1238] System configuration

[1239] The system consists of a user's device, a server, and a database. Users use devices such as smartphones to manage food expiration dates. The server processes and stores data, and the database stores food information.

[1240] Food information registration

[1241] Users take a photo of the expiration date of purchased food using their smartphone camera. The device uses OpenCV and pytesseract for image recognition, automatically extracts the expiration date, and saves it as food information. Users can also enter additional information such as the type of food and storage method. This data is sent to the server, which stores it in a database.

[1242] Best before date notification

[1243] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, a push notification is sent, for example, three days before the expiration date. The device receives this notification and displays a warning to the user. The notification is sent using plyer.

[1244] Preservation method

[1245] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the optimal storage method for that food item and sends it to the device. The device then displays this storage method to the user.

[1246] Recipe Suggestions

[1247] If there are multiple foods approaching their expiration date, the server uses a generative AI model to create recipes that combine these foods. It also suggests recipes taking into account information such as the day's weather, seasonality, and the user's physical condition. The generated recipes are sent to the device and displayed to the user. It also creates menus based on foods approaching their expiration date, supporting the operation of the delivery service.

[1248] Specific examples

[1249] A user purchases milk and takes a photo of the label with a smartphone, which says the expiration date "December 15, 2023." The device automatically extracts this date and sends it to the server as "milk" information along with the "keep refrigerated" information. The server then stores it in a database.

[1250] When the expiration date is approaching in three days, the server sends a notification to the terminal stating "The expiration date of the milk is approaching in three days" and displays a warning to the user.

[1251] If a user registers "chicken" and "carrots" as foods that are close to their expiration date, the server will use generative AI to suggest a recipe for "chicken and carrot teriyaki rice bowl." It will also suggest recipes such as "chicken and carrot ginger soup" for cold days.

[1252] Prompt Sentence Examples

[1253] "Based on the food information below, generate a list of foods nearing their end of life and display them sorted by expiry date. Set up subsequent notifications and suggest storage methods and recipes."

[1254] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1255] Step 1: Photograph and extract food information

[1256] A user takes a photo of the expiration date of a purchased food item with their smartphone. The device then analyzes the image using OpenCV and pytesseract and extracts the expiration date as text data. The input data is the image of the food, and the output data is the extracted expiration date. This process saves the user the trouble of manually entering the data.

[1257] Step 2: Save food information

[1258] The device sends the extracted expiration date and any additional information entered by the user, such as the type of food and storage method, to the server. The server receives this information and stores it in a database. The input data is information such as the expiration date, type of food, and storage method, and the output data is the updated database. This allows for centralized management of food expiration date information.

[1259] Step 3: Regularly check expiration dates

[1260] The server periodically (e.g., daily) checks the food information in the database and identifies foods that are approaching their expiration date. This process makes it possible to quickly detect foods that are approaching their expiration date. The input data is all food information in the database, and the output data is a list of foods that are approaching their expiration date.

[1261] Step 4: Best before notification

[1262] The server creates a list of foods that are close to their expiration date and sends a push notification to the device three days before the expiration date. The device receives this notification and displays a warning to the user. The input data is a list of foods that are close to their expiration date, and the output data is a notification message to the user. This allows the user to use the food without worrying about the expiration date.

[1263] Step 5: Recommending the best storage method

[1264] When a user registers a food item, the device sends a request for storage method to the server. The server searches the database for the optimal storage method and sends it to the device. The device then displays this storage method to the user. The input data is the type of food, and the output data is the optimal storage method. This allows the quality of the food to be maintained at its optimum.

[1265] Step 6: Recipe suggestions

[1266] If there are multiple foods nearing their expiration date, the server uses a generative AI model to create a recipe that combines these foods. The recipe is also customized taking into account the weather, seasons, and the user's physical condition. The generative AI uses specific prompts. The input data is a list of foods nearing their expiration date and information about the weather and physical condition, and the output data is a suggested recipe. This reduces food waste while supporting the user's health.

[1267] As a specific example, the prompt text used is, "Based on the food information below, generate a list of foods that are nearing the end of their lifespan, and display them sorted by date for those with the nearest expiration date. Set the system to notify you in the future, and suggest storage methods and recipes."

[1268] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1269] The present invention is a system that aims to effectively manage food expiration dates and reduce food waste. Furthermore, by combining it with an emotion engine that recognizes the user's emotions and adding a function that suggests recipes based on the user's emotions, the user experience is improved. Specific embodiments of the system are described below.

[1270] System Configuration

[1271] The system consists of a user device, a server, a database, and an emotion engine. Users use devices such as smartphones and tablets to manage expiration dates and recognize emotions. The server processes and stores data, and the database stores information about food and emotions. The emotion engine is a dedicated component for recognizing and analyzing user emotions.

[1272] Food information registration

[1273] To register the expiration date of purchased food, a user follows these steps: the user takes a photo of the food, and the device automatically extracts the expiration date using image recognition technology. If necessary, the user can enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database.

[1274] Examples:

[1275] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[1276] Best before date notification

[1277] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, it sends a push notification, for example, three days before the expiration date. The device receives this notification and displays a warning to the user.

[1278] Examples:

[1279] The server checks the database and finds that the milk's expiration date is three days away. Because the notification period set by the user is three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[1280] Preservation method

[1281] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[1282] Examples:

[1283] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[1284] Emotion Recognition and Recipe Suggestion

[1285] The app uses cameras and sensors installed on the device to recognize the user's emotions. While the user is using the app, the emotion engine analyzes the user's facial expressions and voice to generate emotion data. This data is sent to the server for analysis. Based on the analysis results, the server uses generative AI to suggest recipes that take into account the day's weather, seasonal ingredients, the user's physical condition, and emotions.

[1286] Examples:

[1287] While the user is using the app, the emotion engine recognizes the user's facial expressions and generates emotional data such as "tired." Based on this information, the server uses generative AI to suggest recipes such as "chicken and carrot soup with fatigue-relieving effects." These are displayed on the user's device, and the user can use the recipe to cook the dish.

[1288] In this way, the system of the present invention automates food expiration date management, improves user experience through emotion-based recipe suggestions, reduces food waste, and supports safe and efficient use of ingredients.

[1289] The processing flow will be explained below.

[1290] Step 1:

[1291] The user takes a photo of the expiration date on the food product they purchased. The user uses the camera on their smartphone to take a photo of the expiration date printed on the food product label.

[1292] Step 2:

[1293] The device will perform image recognition and automatically extract the expiration date, while the app on the device will use image processing technology to analyze and extract the expiration date from the image.

[1294] Step 3:

[1295] The user enters additional information such as the type of food and storage method. The user enters the name of the food, purchase date, and storage method through the app screen.

[1296] Step 4:

[1297] The terminal sends the collected food information to the server, which then packages the extracted expiration date and any additional information entered by the user and sends it to the server.

[1298] Step 5:

[1299] The server stores the food information in a database. The server analyzes the received food data and stores it in a database.

[1300] Step 6:

[1301] The server periodically checks the database to identify foods that are nearing their expiration date. A program on the server periodically scans the database to identify foods that are nearing their expiration date.

[1302] Step 7:

[1303] The server prepares notifications for the relevant foods and sends them to the terminal. The server generates notification messages for the identified foods and sends them to the terminal a set number of days in advance.

[1304] Step 8:

[1305] The terminal receives the notification and displays a warning to the user. The terminal receives the notification from the server and displays a warning message on the user's screen informing them that the expiration date is approaching.

[1306] Step 9:

[1307] The user operates the device to check how to store food. The user selects an option in the device's app to check how to store food.

[1308] Step 10:

[1309] The device sends a request to the server about how to store food.The device sends a request to the server about how to store food.

[1310] Step 11:

[1311] The server searches the database for information on the storage method. The server searches the database for information on the relevant storage method based on the request.

[1312] Step 12:

[1313] The server sends information on how to save the data to the device. The server compiles the search results and sends information on how to save the data to the device.

[1314] Step 13:

[1315] The device displays the storage method to the user. The device displays the received storage method information on the app screen so that the user can check it.

[1316] Step 14:

[1317] The device uses cameras and sensors to recognize the user's emotions, and the emotion engine analyzes the user's facial expressions and voice to generate emotion data.

[1318] Step 15:

[1319] The device sends the generated emotion data to the server. The device packages the user's emotion data and sends it to the server.

[1320] Step 16:

[1321] The server analyzes the user's emotional data. The server analyzes the received emotional data and identifies the user's emotional state.

[1322] Step 17:

[1323] The server takes into consideration emotional data, weather, seasonality, and the user's physical condition, and sends a request to the generation AI to generate a recipe. The server then sends a recipe request to the generation AI based on this information.

[1324] Step 18:

[1325] The generative AI generates the optimal recipe. Based on the information it receives from the server, the generative AI creates a recipe that takes into account the user's emotional state and other conditions.

[1326] Step 19:

[1327] The server saves the generated recipe and sends it to the device. The server temporarily saves the recipe information received from the generation AI in a database and sends it to the device.

[1328] Step 20:

[1329] The device displays the generated recipe to the user. The device displays the received recipe within the app, allowing the user to use the recipe as a reference when cooking.

[1330] Through these steps, the system of the present invention automates food expiration date management, improves user experience through emotion-based recipe suggestions, reduces food waste, and supports safe and efficient ingredient use.

[1331] Example 2

[1332] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1333] In modern society, the increase in food waste has become a major problem. This problem is particularly caused by poor expiration date management within the home. Furthermore, there is a lack of meal suggestions that take into account the user's emotions and physical condition, and there is a need for methods to use ingredients appropriately and without waste. Conventional systems have difficulty meeting these requirements, so an effective solution is needed to simultaneously manage food expiration dates and improve the user experience.

[1334] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1335] In this invention, the server includes a means for photographing the expiration dates of purchased items and storing them as data, a means for notifying the user of items approaching their expiration dates, a means for suggesting optimal storage methods for the items, and a means for analyzing the user's emotions, thereby reducing food waste and improving the user experience through emotion-based meal suggestions.

[1336] "Goods" refers to food or consumable items purchased or acquired by the user.

[1337] "Best before date" refers to the period during which an item can be consumed while maintaining its quality.

[1338] "Image recognition technology" refers to a form of computer vision that extracts and identifies specific information from image data.

[1339] "Data" refers to electromagnetically recorded information, including information such as expiration date, type of product, and storage method.

[1340] "Notification" refers to an alert or message that is displayed on a user's device.

[1341] "Preservation method" refers to the specific procedures and conditions for preserving an item in optimal condition.

[1342] "Cooking method" refers to the steps and methods for creating a dish using an item.

[1343] "Climate" refers to the weather conditions and temperature of the day.

[1344] "Season" refers to the time when something is at its most delicious and nutritious.

[1345] "User" refers to the consumer or end user who uses the system.

[1346] "Emotion" refers to the psychological state detected from the user's facial expressions, voice, etc.

[1347] "Emotion engine" is a general term for the algorithms and hardware that analyze the user's emotions and generate the results as data.

[1348] "Server" refers to a computer system that processes, stores, and analyzes data.

[1349] "Database" refers to a digital storage system for systematically storing and managing information such as expiration dates, storage methods, and emotional data.

[1350] "Generative AI model" refers to a machine learning model that generates useful results (such as recipes) based on input prompts.

[1351] A "prompt sentence" refers to an instruction sentence used as input to a generative AI model.

[1352] The present invention is a system that allows users to manage food expiration dates and receive cooking method suggestions based on their emotions. This system is composed of a terminal, a server, a database, and an emotion engine. Specific embodiments of the system are described below.

[1353] System Configuration

[1354] The system consists of a user device (smartphone or tablet), a server, a database, and an emotion engine. The server uses a powerful computer system to process, store, and analyze data. The database is a digital storage system for systematically storing and managing information such as expiration dates, storage methods, and emotion data. The emotion engine consists of hardware and software components for analyzing user emotions and generating the results as data.

[1355] Food information registration

[1356] To register the expiration date of purchased food, a user takes a photo of the food label using the device's camera. The device then uses image recognition technology to extract the expiration date as text data from the captured image. If necessary, the user can manually enter the type of food and storage method. The device then formats this data and sends an HTTP request to the server. The server then analyzes the received data and stores it in a database in the required format.

[1357] Best before date notification

[1358] The server periodically checks the database to identify foods that are approaching their expiration date. Based on user settings, the server creates and sends a notification three days before the expiration date, for example. The device receives this notification and displays a warning message to the user, such as "The milk's expiration date is three days away."

[1359] Preservation method

[1360] When a user registers a food item, the device requests the server for the optimal storage method for that food item. The server searches the database for the optimal storage method for that food item and sends it to the device as an HTTP response. The device then presents the received storage method to the user, displaying specific storage instructions such as "Store tomatoes at room temperature, or refrigerate if cut."

[1361] Emotion Recognition and Recipe Suggestion

[1362] While the user is using the app, emotion data is collected using the device's on-board camera and sensors. The device then analyzes the collected emotion data using an emotion engine and sends the results to the server. The server generates prompts based on the emotion data and other factors (weather, season, physical condition, etc.) and inputs them into the generative AI model. The server receives the resulting recipes generated by the generative AI model and sends them to the device. The device then presents specific recipes to the user, such as "chicken and carrot soup with fatigue-relieving effects."

[1363] Examples of prompt statements

[1364] Example prompts to input to a generative AI model:

[1365] "What recipes would you recommend to users who are feeling tired? For example, a simple soup recipe using foods that have a fatigue-relieving effect."

[1366] In this way, the system of the present invention automates food expiration date management, improves user experience through emotion-based cooking suggestions, reduces food waste, and supports safe and efficient use of ingredients.

[1367] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1368] Step 1:

[1369] The user takes a photo of the food they purchased on their device.

[1370] Input: A user takes a photo of a food label with their smartphone camera.

[1371] How it works: The user launches the camera app, frames a food label with an expiration date, and presses the capture button.

[1372] Output: The image data of the photographed food label is saved on the device.

[1373] Step 2:

[1374] The device uses image recognition technology to extract the expiration date.

[1375] Input: Image data acquired in step 1.

[1376] How it works: The device uses OCR (optical character recognition) technology to analyze the text data in the image and extract the expiration date as text data.

[1377] Output: Best before date data is obtained in text format.

[1378] Step 3:

[1379] The user enters any additional information needed (such as the type of food and storage method)

[1380] Input: Best before date data obtained in step 2 and manual user input information.

[1381] What it does: A user manually enters the type of food (e.g., milk, bread) and storage method (e.g., refrigerate, freeze) using an app input form.

[1382] Output: The expiration date data and the type of food and storage method data entered by the user are obtained.

[1383] Step 4:

[1384] The device sends the data to the server

[1385] Input: Best before date data obtained in step 3, food type, and storage method data.

[1386] How it works: The device formats this data and sends an HTTP request to the server.

[1387] Output: The data is sent to the server.

[1388] Step 5:

[1389] The server stores the received information in a database

[1390] Input: The data sent from the terminal in step 4.

[1391] How it works: The server parses the received data and stores it in the required format in the database.

[1392] Output: The expiration date, food type, and storage method data are saved in the database.

[1393] Step 6:

[1394] The server periodically checks the database

[1395] Input: Food information in the database.

[1396] How it works: The server runs a periodic job that checks all food items in the database.

[1397] Output: A list of foods that are nearing their expiration date is generated.

[1398] Step 7:

[1399] The server identifies food that is nearing its expiration date

[1400] Input: The list of foods obtained in step 6.

[1401] How it works: The server compares the expiration dates of the foods with the current date to identify foods that require notification.

[1402] Output: A list of foods that require notification is generated.

[1403] Step 8:

[1404] The server creates and sends the push notification

[1405] Input: The list of foods requiring notifications obtained in step 7.

[1406] How it works: The server creates a notification based on the user's settings, for example, three days before the expiration date, and sends a push notification to the user's device.

[1407] Output: A push notification is sent to the user's device.

[1408] Step 9:

[1409] The device receives the notification and displays it to the user.

[1410] Input: The push notification sent in step 8.

[1411] What it does: The device receives a notification from the server and displays a warning message such as "The milk's expiration date is coming up in three days."

[1412] Output: A notification message is displayed on the device screen.

[1413] Step 10:

[1414] Send a storage request when a user registers a food item

[1415] Input: The user enters the name of the food.

[1416] How it works: The user enters the name of a food item into the app's input form and presses the "Get storage instructions" button. The device then sends a request to the server.

[1417] Output: A save method request is sent to the server.

[1418] Step 11:

[1419] The server looks up the save method in the database

[1420] Input: The request submitted in step 10.

[1421] How it works: The server receives a request and searches its database for the best way to store the food.

[1422] Output: The search results for the saving method are obtained.

[1423] Step 12:

[1424] The server sends the save instructions to the device.

[1425] Input: Search results for saving methods obtained in step 11.

[1426] How it works: The server formats the search results and sends them to the device as an HTTP response.

[1427] Output: Information on how to save is sent to the device.

[1428] Step 13:

[1429] The device will prompt the user for instructions on how to save.

[1430] Input: The save method information submitted in step 12.

[1431] Operation: The device presents the received storage instructions to the user, displaying specific storage instructions such as "Store tomatoes at room temperature, or refrigerate if cut."

[1432] Output: Information about the saving method will be displayed on the device screen.

[1433] Step 14:

[1434] The device collects emotion data while the user is using the app.

[1435] Input: User's facial and voice data.

[1436] How it works: The user opens the app and uses the camera and sensors to capture facial expressions and voice in real time. The device then captures this emotional data.

[1437] Output: Emotion data is saved on the device.

[1438] Step 15:

[1439] The device analyzes the emotional data it acquires and sends it to the server.

[1440] Input: Emotion data obtained in step 14.

[1441] Operation: The device analyzes the facial expression data using the emotion engine, extracts the emotional state (e.g., "tired"), and sends it to the server.

[1442] Output: The analyzed emotion data is sent to the server.

[1443] Step 16:

[1444] The server generates a prompt and inputs it into the generative AI model.

[1445] Input: Emotion data sent in step 15 and other factors (weather, season, physical condition information).

[1446] How it works: The server uses this information to generate a prompt (e.g., "What recipes would you recommend for a tired user?") and inputs it into the generative AI model.

[1447] Output: The generated prompt sentence is input to the generative AI model.

[1448] Step 17:

[1449] The server sends the generated recipe to the device.

[1450] Input: The recipe obtained from the generative AI model in step 16.

[1451] How it works: The server receives the recipe generated by the AI ​​model and sends it to the device.

[1452] Output: Recipe data is sent to the device.

[1453] Step 18:

[1454] The device displays the recipe to the user

[1455] Input: Recipe data submitted in step 17.

[1456] How it works: The device displays the recipe it receives from the server on the screen, and suggests specific recipes such as "chicken and carrot soup with fatigue-relieving effects."

[1457] Output: The recipe is displayed on the device screen and provided to the user.

[1458] In this way, by linking each processing step with each other, the system of the present invention can effectively manage food expiration dates, suggest storage methods, and suggest cooking methods based on emotions.

[1459] (Application example 2)

[1460] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1461] In recent years, with the increasing importance of managing food expiration dates, there is a demand for suggestions of appropriate storage methods and recipes using food that is close to its expiration date. However, because individual factors such as emotions and physical condition are not taken into consideration, the user experience is often unsatisfactory. Furthermore, expiration date management is sometimes not thoroughly implemented in physical stores, leading to a serious problem of food waste. To solve these issues, a system that reflects user emotions and can also manage expiration dates in physical stores is needed.

[1462] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for photographing the expiration dates of purchased foods and saving them as data, means for notifying users of foods approaching their expiration dates, means for suggesting optimal food storage methods, means for providing recipes that combine foods with upcoming expiration dates, means for suggesting recipes taking into account the weather, seasons, and the user's physical condition on that day, means for recognizing the user's emotions and suggesting recipes based on the emotion data, and means for managing food in a physical store and suggesting recipes based on the emotions. This makes it possible to suggest recipes that correspond to the emotions of each individual user, reducing food waste and enabling efficient food management in physical stores.

[1463] The "best before date" of a food product is the date by which the food product will continue to maintain its appropriate quality.

[1464] "Storing as data" means electronically recording information such as expiration dates and making them accessible later.

[1465] "Notifying" means sending a warning or alert to the user when a specific condition is met.

[1466] "Best storage method" refers to the recommended storage environment and conditions to maintain the quality of food.

[1467] A "recipe" is a list of steps and ingredients for making a dish.

[1468] "Climate" refers to weather conditions such as temperature, humidity, wind speed, and precipitation over a certain period of time.

[1469] "Season" refers to the time when a particular food is most nutritious and delicious.

[1470] "User's physical condition" refers to the user's health condition and current physical condition.

[1471] "Emotion data" refers to emotional information obtained by analyzing the user's facial expressions, voice, behavior, etc.

[1472] A "brick and mortar store" is a physical retail establishment where consumers can visit and purchase products in person.

[1473] "Emotion recognition" means judging the user's emotional state by analyzing the user's facial expressions, etc.

[1474] "Emotion-based recipe suggestions" refers to taking into account the user's current emotional state and providing cooking ideas that are appropriate for that emotion.

[1475] "Managing food" means centrally monitoring and controlling food inventory, expiration dates, storage methods, etc.

[1476] "Image recognition technology" is a technology that analyzes image data and automatically recognizes specific information (such as characters or objects).

[1477] A "server" is a computer system that stores, processes, and manages data.

[1478] "Suggesting storage methods" means showing the user the best storage conditions for a particular food item.

[1479] "Providing recipe combinations" means teaching users how to combine multiple foods into one dish.

[1480] "User-specified" means that the user inputs his / her wishes and conditions into the system and makes a selection.

[1481] "Push notification" is a mechanism that notifies users of specific information in a timely manner.

[1482] This invention aims to effectively manage the expiration dates of food purchased by users and reduce food waste. Furthermore, by combining it with an emotion engine that recognizes user emotions, it can suggest recipes based on emotions, improving the user experience.

[1483] System Configuration

[1484] The system consists of a user device, a server, a database, and an emotion engine. Users use devices such as smartphones and tablets to manage expiration dates and recognize emotions. The server processes and stores data, and the database stores information about food and emotions. The emotion engine is a dedicated component for recognizing and analyzing user emotions.

[1485] Food information registration

[1486] To register the expiration date of purchased food, a user follows these steps: the user takes a photo of the food, and the device automatically extracts the expiration date using image recognition technology. If necessary, the user can enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database.

[1487] Specific examples

[1488] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[1489] Best before date notification

[1490] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, it sends a push notification, for example, three days before the expiration date. The device receives this notification and displays a warning to the user.

[1491] Specific examples

[1492] The server checks the database and finds that the milk's expiration date is three days away. Because the notification period set by the user is three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[1493] Preservation method

[1494] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[1495] Specific examples

[1496] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[1497] Emotion Recognition and Recipe Suggestion

[1498] The app uses cameras and sensors installed on the device to recognize the user's emotions. While the user is using the app, the emotion engine analyzes the user's facial expressions and voice to generate emotion data. This data is sent to the server for analysis. Based on the analysis results, the server uses generative AI to suggest recipes that take into account the day's weather, seasonal ingredients, the user's physical condition, and emotions.

[1499] Specific examples

[1500] While the user is using the app, the emotion engine recognizes the user's facial expressions and generates emotional data such as "tired." Based on this information, the server uses generative AI to suggest recipes such as "chicken and carrot soup with fatigue-relieving effects." These are displayed on the user's device, and the user can use the recipe to cook the dish.

[1501] Examples of prompts to input to a generative AI model:

[1502] Generate a recipe for someone who is feeling tired and has the following items: carrots, chicken broth, onions, garlic.

[1503] Application in physical stores

[1504] This system can also be applied in physical stores. Employees and customers can use their smartphones to manage food in the store and suggest recipes based on their emotions. This will make it easier to manage expiration dates in the store and is expected to reduce food waste.

[1505] In this way, the system of the present invention automates food expiration date management, improves user experience through emotion-based recipe suggestions, reduces food waste, and supports safe and efficient use of ingredients.

[1506] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1507] Step 1:

[1508] The user takes a photo of the food they have purchased. Using the device, they take a photo of the label containing the expiration date with a camera, and the image data is used as input.

[1509] Step 2:

[1510] The device automatically extracts the expiration date using image recognition technology. It uses an image recognition algorithm (e.g., OpenCV) to analyze the characters in the image and identify the expiration date. The identified expiration date is the output.

[1511] Step 3:

[1512] If necessary, the user inputs additional information, such as the type of food or storage method. The information is supplemented through the device's user interface with text input or selection menus. The input additional information is output.

[1513] Step 4:

[1514] The device sends this data (best before date, food type, storage method) to the server. The data is transferred to the server through the network. The raw data sent to the server is the output.

[1515] Step 5:

[1516] The server stores the received data in a database. The data is structured and stored using a database management system (e.g., SQLite). The food information stored in the database is the output.

[1517] Step 6:

[1518] The server periodically checks the food information in the database to identify foods that are approaching their expiration date. It then runs a query to extract foods with an expiration date within three days. The extracted list of foods is the output.

[1519] Step 7:

[1520] The server sends push notifications for the identified foods based on the user settings. If the notification condition (e.g., three days before) is met, the server generates the notification content and sends it to the user's device. The notification message displayed on the device is the output.

[1521] Step 8:

[1522] The terminal receives the notification and displays a warning to the user. The notification message is displayed in a popup format to get the user's immediate attention. The warning displayed to the user is the output.

[1523] Step 9:

[1524] When a user registers a food item, the device requests the server for the optimal storage method for that food item. The storage method request is sent to the server. The sent request data is the output.

[1525] Step 10:

[1526] The server searches the database for a storage method based on the type of food and sends it to the terminal. It searches for a record that stores the appropriate storage method and transfers the storage method data. The storage method sent to the terminal is the output.

[1527] Step 11:

[1528] The terminal displays this saving method to the user. The appropriate saving conditions are displayed as text on the screen and provided to the user. The saving method displayed to the user is the output.

[1529] Step 12:

[1530] While the user is using the app, the emotion engine analyzes the user's facial expressions and voice to generate emotion data. Camera and microphone data is used as input to run emotion analysis algorithms (e.g., OpenCV, dlib) to identify the user's emotional state. The identified emotion data is the output.

[1531] Step 13:

[1532] Emotion data is sent to the server, where it is analyzed. The generated emotion data is sent to the server over the network and processed by the analysis component. The analyzed emotion information is the output.

[1533] Step 14:

[1534] Based on the analysis results, the server uses generative AI to suggest recipes that take into account the day's weather, seasonal ingredients, the user's physical condition, and emotions. A prompt is entered into the generative AI model, and the recipe generation results are received.

[1535] Example prompt sentence:

[1536] Generate a recipe for someone who is feeling tired and has the following items: carrots, chicken broth, onions, garlic.

[1537] The generated recipe data is the output.

[1538] Step 15:

[1539] The server sends the generated recipe to the terminal, which displays it to the user. The recipe information is transferred via the network and displayed on the terminal screen. The recipe displayed to the user is the output.

[1540] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1541] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1542] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1543] [Fourth embodiment]

[1544] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1545] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1546] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1547] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1548] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1549] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1550] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1551] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1552] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1553] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1554] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1555] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1556] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1557] The present invention is a system for effectively managing the expiration dates of food products and reducing food waste. Specific embodiments of the system are described below.

[1558] System Configuration

[1559] The system consists of a user device, a server, and a database. Users use devices such as smartphones and tablets to manage expiration dates. The server processes and stores data, and the database stores food information.

[1560] Food information registration

[1561] To register the expiration date of purchased food, a user follows these steps: the user takes a photo of the food, and the device automatically extracts the expiration date using image recognition technology. If necessary, the user can enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database.

[1562] Examples:

[1563] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[1564] Best before date notification

[1565] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, it sends a push notification, for example, three days before the expiration date. The device receives this notification and displays a warning to the user.

[1566] Examples:

[1567] The server checks the database and finds that the milk's expiration date is three days away. Because the notification period set by the user is three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[1568] Preservation method

[1569] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[1570] Examples:

[1571] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[1572] Recipe Suggestions

[1573] If there are multiple foods nearing their expiration date, the server uses generative AI technology to create recipes that combine these foods. It also suggests recipes taking into account information such as the day's weather, seasonality, and the user's physical condition. The generated recipe is sent to the device and displayed to the user.

[1574] Examples:

[1575] If a user registers "chicken" and "carrots" as foods that are close to expiry, the server will use generative AI to suggest a recipe for "chicken and carrot teriyaki rice bowl." In addition, on cold days, it will also suggest recipes such as "chicken and carrot ginger soup."

[1576] In this way, the system of the present invention automates the management of food expiration dates and supports users in using food effectively by suggesting optimal storage methods and recipes.

[1577] The processing flow will be explained below.

[1578] Step 1:

[1579] The user takes a photo of the expiration date on the food they purchased. The user uses the camera on their smartphone to take a photo of the part of the food that has the expiration date written on it.

[1580] Step 2:

[1581] The device will use image recognition to automatically extract the expiration date, and the app on the device will use image processing technology to extract the necessary date information from the image.

[1582] Step 3:

[1583] The user enters additional information such as the type of food and storage method. The user can manually enter the food name, purchase date, storage method, etc. using the app screen.

[1584] Step 4:

[1585] The terminal sends the collected food information to the server, which then packages the extracted expiration date and any additional information entered by the user and sends it to the server.

[1586] Step 5:

[1587] The server saves the food information in a database. The server analyzes the received food data and stores it in the database.

[1588] Step 6:

[1589] The server periodically checks the database to identify foods that are approaching their expiration date. A server-side program scans the database at regular intervals and selects foods that are nearing their expiration date.

[1590] Step 7:

[1591] The server prepares and sends notifications for the relevant foods to the terminal. The server generates notification messages for the identified foods and sends them to the terminal a set number of days in advance.

[1592] Step 8:

[1593] The terminal receives the notification and displays a warning to the user. The terminal receives the notification from the server and displays a message on the user's screen warning that the expiration date is approaching.

[1594] Step 9:

[1595] The user interacts with their device to learn how to store food. The user selects the option within the app to learn how to store food.

[1596] Step 10:

[1597] The device sends a request to the server regarding how to store food. The device sends a request to the server to inquire about how to store food.

[1598] Step 11:

[1599] The server retrieves information about the storage method from the database. The server retrieves the data about the storage method in the database and searches for related information.

[1600] Step 12:

[1601] The server sends information about how to save the data to the device. The server compiles the search results and sends details about how to save the data to the device.

[1602] Step 13:

[1603] The device displays the storage method to the user. The device displays the received storage method information on the app screen so that the user can check it.

[1604] Step 14:

[1605] The server sends a request to the generation AI to generate a recipe using foods that are close to their expiration date. The server creates a list of foods that are close to their expiration date and sends a request to the generation AI to generate a recipe.

[1606] Step 15:

[1607] The AI ​​generates recipes based on requests, taking into account the target food, climate, season, and the user's physical condition to create the optimal recipe.

[1608] Step 16:

[1609] The server saves the generated recipe and sends it to the device. The server temporarily saves the recipe information received from the generation AI in a database and sends it to the device.

[1610] Step 17:

[1611] The device displays the generated recipe to the user. The device displays the received recipe in the app, allowing the user to cook a dish based on it.

[1612] Through these steps, the system of the present invention helps users effectively manage food expiration dates, reduce waste, and use ingredients safely and efficiently.

[1613] Example 1

[1614] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1615] Currently, consumers must manually record the expiration dates of purchased food, which requires a great deal of effort. Furthermore, many people discard food without realizing that its expiration date is approaching, resulting in significant waste and environmental impact. Furthermore, it is difficult to maintain food quality due to lack of proper storage methods. To solve these issues, a system is needed that allows consumers to efficiently manage food, reduce waste, and properly store and use food.

[1616] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1617] In this invention, the server includes a device that photographs the expiration dates of food products and stores them as data, a device that notifies users of food products that are approaching their expiration dates, a device that suggests optimal food storage methods, a device that creates recipes that combine foods that are approaching their expiration dates, and a device that suggests recipes taking into account the weather, season, and the user's physical condition on that day. This allows consumers to efficiently manage the expiration dates of purchased food products, reduce waste, and ensure appropriate use of food products while maintaining their quality.

[1618] A "device that photographs the expiration date of food products and stores it as data" is a device that has the function of photographing the expiration date of food products purchased by a user, extracting the date information using image recognition technology, and storing the extracted data.

[1619] A "device that notifies users of food products approaching their expiration date" is a device that periodically checks food information stored in a database, identifies foods that are approaching their expiration date, and notifies users.

[1620] The "device that suggests the optimum method of preserving food" is a device that has the function of searching for the optimum method of preserving food based on the type of food registered and suggesting it to the user.

[1621] The "device for generating recipes that combine foods with approaching expiration dates" is a device that combines multiple foods that are approaching their expiration dates, generates appropriate recipes using generative AI technology, and provides them to users.

[1622] "A device that suggests recipes taking into account the weather, season, and the user's physical condition on that day" refers to a device that has the function of using generative AI technology to suggest appropriate recipes based on the current weather, season, and the user's physical condition, and providing them to the user.

[1623] The present invention is a system that aims to effectively manage food expiration dates and reduce food waste. This system is composed of a user's terminal, a server, and a database, and describes specific embodiments that include registering food information, notifying users of expiration dates, suggesting storage methods, and suggesting recipes.

[1624] System Configuration

[1625] The system allows users to manage expiration dates using devices such as smartphones and tablets. The server processes and stores data, and the database stores food information.

[1626] Food information registration

[1627] To register the expiration date of purchased food, the user first takes a photo of the food with the device. The device then uses image recognition technology (e.g., Google Cloud Vision API or Amazon Rekognition) to automatically extract the expiration date from the image. The user can then enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database (e.g., MySQL or PostgreSQL).

[1628] Examples:

[1629] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[1630] Best before date notification

[1631] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. The check is performed periodically, such as once a day. Based on the user's settings, a push notification is sent, for example, three days before the expiration date. The device receives this notification and displays a warning to the user (for example, using Firebase Cloud Messaging to send a push notification).

[1632] Examples:

[1633] The server checks the database and finds that the milk's expiration date is three days away. Because the user has set the notification period to three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[1634] Preservation method

[1635] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the optimal storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[1636] Examples:

[1637] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[1638] Recipe Suggestions

[1639] If there are multiple foods nearing their expiration date, the server uses generative AI technology (e.g., OpenAI GPT-3) to create recipes that combine these foods. Furthermore, it takes into account information such as the weather, season, and the user's physical condition on the day to suggest recipes. The generated recipe is sent to the device and displayed to the user.

[1640] Examples:

[1641] If a user registers "chicken" and "carrots" as foods that are close to expiry, the server will use generative AI to suggest a recipe for "chicken and carrot teriyaki rice bowl." In addition, on cold days, it will also suggest recipes such as "chicken and carrot ginger soup."

[1642] Example prompt sentence:

[1643] Please suggest a recipe using "chicken and carrots." The current temperature is 10 degrees and the user is feeling cold.

[1644] As described above, the system of the present invention automates the management of food expiration dates and supports users in using food effectively by suggesting optimal storage methods and recipes.

[1645] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1646] System program processing flow

[1647] Step 1:

[1648] The user takes a photo of the food they have purchased using the camera on their smartphone or tablet.

[1649] Specific behavior:

[1650] The user activates the smartphone camera and takes a picture of the food package.

[1651] Input: Food to be photographed

[1652] Output: Food image data

[1653] Step 2:

[1654] The device processes the captured image and uses image recognition technology to extract the expiration date.

[1655] Specific behavior:

[1656] The device uses the Google Cloud Vision API or Amazon Rekognition to analyze the image and extract the expiration date.

[1657] Input: Food image data

[1658] Output: Extracted best-before date data

[1659] Step 3:

[1660] If necessary, the user can input additional information such as the type of food and storage method into the terminal.

[1661] Specific behavior:

[1662] The user enters the type of food (e.g., milk) and storage method (e.g., refrigerate) into an application form.

[1663] Input: Additional information entered by the user about the type of food and storage method

[1664] Output: Complete food data (expiration date, type, storage method)

[1665] Step 4:

[1666] The terminal transmits the extracted and entered information to the server.

[1667] Specific behavior:

[1668] The device sends JSON format data to the server via an HTTP request.

[1669] Input: Complete food data

[1670] Output: Data sent to server completed

[1671] Step 5:

[1672] The server stores the received data in a database.

[1673] Specific behavior:

[1674] The server connects to a MySQL database and stores data using SQL queries.

[1675] Input: Complete food data

[1676] Output: Food data stored in a database

[1677] Step 6:

[1678] The server periodically checks the database to identify foods that are nearing their expiration date.

[1679] Specific behavior:

[1680] The server sets up a Cron job to check the expiration date in the database every day at midnight.

[1681] Input: Food data in the database

[1682] Output: A list of foods that are close to expiry date

[1683] Step 7:

[1684] The server generates a push notification about the identified food item and sends it to the device.

[1685] Specific behavior:

[1686] The server uses the Firebase Cloud Messaging API to generate a push notification and send it to the device.

[1687] Input: List of foods that are close to expiry date

[1688] Output: Push notification sent to device

[1689] Step 8:

[1690] The terminal displays the received notification to the user.

[1691] Specific behavior:

[1692] The device receives a notification and displays a pop-up message saying, "The milk's expiration date is in three days."

[1693] Input: Push notification from the server

[1694] Output: Display a warning message to the user

[1695] Step 9:

[1696] When a user registers a food item, the device requests storage instructions from the server.

[1697] Specific behavior:

[1698] The user registers "tomato" and presses the button "Please tell me how to store it." The device then requests the server how to store that food.

[1699] Input: Food information registered by the user

[1700] Output: Request sent to server

[1701] Step 10:

[1702] The server searches the database for storage methods based on the type of food and sends them to the terminal.

[1703] Specific behavior:

[1704] The server searches the "food preservation method" table for the method for preserving "tomatoes" and sends it to the terminal.

[1705] Input: Save method request from user

[1706] Output: Sending save method data

[1707] Step 11:

[1708] The terminal displays the received saving method to the user.

[1709] Specific behavior:

[1710] The device will display "How to store tomatoes: Store at room temperature, refrigerate after cutting."

[1711] Input: Save method data from the server

[1712] Output: Show user how to save

[1713] Step 12:

[1714] The server organizes information about food items that are close to their expiration date and generates recipes using a generative AI model.

[1715] Specific behavior:

[1716] The server inputs the prompt "Foods with an approaching expiration date: chicken, carrots. Please suggest a recipe using these." into the generation AI and generates a recipe.

[1717] Input: Food information about expiry date approaching, prompt text

[1718] Output: The generated recipe

[1719] Step 13:

[1720] The server sends the generated recipe to the terminal.

[1721] Specific behavior:

[1722] The server sends the recipe for "Teriyaki Chicken and Carrot Rice Bowl" to the terminal.

[1723] Input: Generated recipe

[1724] Output: Send recipe to terminal

[1725] Step 14:

[1726] The terminal displays the received recipe to the user.

[1727] Specific behavior:

[1728] The device displays "Teriyaki Rice Bowl Recipe: Chicken, Carrots, Soy Sauce, Mirin, Sugar."

[1729] Input: Recipe information from the server

[1730] Output: Recipe displayed to user

[1731] (Application example 1)

[1732] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1733] In today's food delivery services, improving the efficiency of food inventory management and reducing food waste are major challenges. In particular, insufficient expiration date management has led to widespread food waste due to expired food, resulting in economic losses and increasing the burden on the environment. Furthermore, it is difficult to propose recipes and create menus that effectively utilize food that is approaching its expiration date, increasing the burden on employees. To solve these issues, a system that automatically manages expiration dates and suggests optimal storage methods and recipes is needed.

[1734] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1735] In this invention, the server includes means for photographing the expiration dates of purchased foods and saving them as data, means for notifying users of foods approaching their expiration dates, means for suggesting optimal food storage methods, means for generating and providing recipes combining foods approaching their expiration dates using a generative AI model, means for suggesting recipes taking into consideration the weather, seasons, and the user's physical condition on that day, and means for creating menus based on foods approaching their expiration dates and supporting efficient food delivery. This enables efficient food inventory management and reduces food waste.

[1736] "Means for photographing the expiration date of purchased food and saving it as data" refers to a system that photographs the expiration date of food with a camera and automatically recognizes and saves the data.

[1737] The "means for notifying about foods approaching their expiration date" is a function for automatically detecting foods approaching their expiration date and sending a notification to the user.

[1738] A "means for suggesting the optimal method of food preservation" is a system for providing appropriate preservation methods for specific foods.

[1739] "Means for generating and providing recipes that combine food items with close expiration dates using a generative AI model" refers to a system for combining food items with close expiration dates, creating recipes using a generative AI model, and providing the recipes.

[1740] "Means for suggesting recipes taking into consideration the weather, season, and the user's physical condition on that day" is a function for suggesting appropriate recipes taking into consideration the weather, season, and the user's physical condition on that day.

[1741] "A means for creating menus based on foods with approaching expiration dates and supporting efficient food delivery" is a system for creating menus based on foods with approaching expiration dates and operating food delivery efficiently.

[1742] MODE FOR CARRYING OUT THE INVENTION

[1743] The present invention provides a system for improving the efficiency of food inventory management and reducing food waste in food delivery services. Specific embodiments of the system are described below.

[1744] System configuration

[1745] The system consists of a user's device, a server, and a database. Users use devices such as smartphones to manage food expiration dates. The server processes and stores data, and the database stores food information.

[1746] Food information registration

[1747] Users take a photo of the expiration date of purchased food using their smartphone camera. The device uses OpenCV and pytesseract for image recognition, automatically extracts the expiration date, and saves it as food information. Users can also enter additional information such as the type of food and storage method. This data is sent to the server, which stores it in a database.

[1748] Best before date notification

[1749] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, a push notification is sent, for example, three days before the expiration date. The device receives this notification and displays a warning to the user. The notification is sent using plyer.

[1750] Preservation method

[1751] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the optimal storage method for that food item and sends it to the device. The device then displays this storage method to the user.

[1752] Recipe Suggestions

[1753] If there are multiple foods approaching their expiration date, the server uses a generative AI model to create recipes that combine these foods. It also suggests recipes taking into account information such as the day's weather, seasonality, and the user's physical condition. The generated recipes are sent to the device and displayed to the user. It also creates menus based on foods approaching their expiration date, supporting the operation of the delivery service.

[1754] Specific examples

[1755] A user purchases milk and takes a photo of the label with a smartphone, which says the expiration date "December 15, 2023." The device automatically extracts this date and sends it to the server as "milk" information along with the "keep refrigerated" information. The server then stores it in a database.

[1756] When the expiration date is approaching in three days, the server sends a notification to the terminal stating "The expiration date of the milk is approaching in three days" and displays a warning to the user.

[1757] If a user registers "chicken" and "carrots" as foods that are close to their expiration date, the server will use generative AI to suggest a recipe for "chicken and carrot teriyaki rice bowl." It will also suggest recipes such as "chicken and carrot ginger soup" for cold days.

[1758] Prompt Sentence Examples

[1759] "Based on the food information below, generate a list of foods nearing their end of life and display them sorted by expiry date. Set up subsequent notifications and suggest storage methods and recipes."

[1760] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1761] Step 1: Photograph and extract food information

[1762] A user takes a photo of the expiration date of a purchased food item with their smartphone. The device then analyzes the image using OpenCV and pytesseract and extracts the expiration date as text data. The input data is the image of the food, and the output data is the extracted expiration date. This process saves the user the trouble of manually entering the data.

[1763] Step 2: Save food information

[1764] The device sends the extracted expiration date and any additional information entered by the user, such as the type of food and storage method, to the server. The server receives this information and stores it in a database. The input data is information such as the expiration date, type of food, and storage method, and the output data is the updated database. This allows for centralized management of food expiration date information.

[1765] Step 3: Regularly check expiration dates

[1766] The server periodically (e.g., daily) checks the food information in the database and identifies foods that are approaching their expiration date. This process makes it possible to quickly detect foods that are approaching their expiration date. The input data is all food information in the database, and the output data is a list of foods that are approaching their expiration date.

[1767] Step 4: Best before notification

[1768] The server creates a list of foods that are close to their expiration date and sends a push notification to the device three days before the expiration date. The device receives this notification and displays a warning to the user. The input data is a list of foods that are close to their expiration date, and the output data is a notification message to the user. This allows the user to use the food without worrying about the expiration date.

[1769] Step 5: Recommending the best storage method

[1770] When a user registers a food item, the device sends a request for storage method to the server. The server searches the database for the optimal storage method and sends it to the device. The device then displays this storage method to the user. The input data is the type of food, and the output data is the optimal storage method. This allows the quality of the food to be maintained at its optimum.

[1771] Step 6: Recipe suggestions

[1772] If there are multiple foods nearing their expiration date, the server uses a generative AI model to create a recipe that combines these foods. The recipe is also customized taking into account the weather, seasons, and the user's physical condition. The generative AI uses specific prompts. The input data is a list of foods nearing their expiration date and information about the weather and physical condition, and the output data is a suggested recipe. This reduces food waste while supporting the user's health.

[1773] As a specific example, the prompt text used is, "Based on the food information below, generate a list of foods that are nearing the end of their lifespan, and display them sorted by date for those with the nearest expiration date. Set the system to notify you in the future, and suggest storage methods and recipes."

[1774] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1775] The present invention is a system that aims to effectively manage food expiration dates and reduce food waste. Furthermore, by combining it with an emotion engine that recognizes the user's emotions and adding a function that suggests recipes based on the user's emotions, the user experience is improved. Specific embodiments of the system are described below.

[1776] System Configuration

[1777] The system consists of a user device, a server, a database, and an emotion engine. Users use devices such as smartphones and tablets to manage expiration dates and recognize emotions. The server processes and stores data, and the database stores information about food and emotions. The emotion engine is a dedicated component for recognizing and analyzing user emotions.

[1778] Food information registration

[1779] To register the expiration date of purchased food, a user follows these steps: the user takes a photo of the food, and the device automatically extracts the expiration date using image recognition technology. If necessary, the user can enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database.

[1780] Examples:

[1781] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[1782] Best before date notification

[1783] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, it sends a push notification, for example, three days before the expiration date. The device receives this notification and displays a warning to the user.

[1784] Examples:

[1785] The server checks the database and finds that the milk's expiration date is three days away. Because the notification period set by the user is three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[1786] Preservation method

[1787] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[1788] Examples:

[1789] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[1790] Emotion Recognition and Recipe Suggestion

[1791] The app uses cameras and sensors installed on the device to recognize the user's emotions. While the user is using the app, the emotion engine analyzes the user's facial expressions and voice to generate emotion data. This data is sent to the server for analysis. Based on the analysis results, the server uses generative AI to suggest recipes that take into account the day's weather, seasonal ingredients, the user's physical condition, and emotions.

[1792] Examples:

[1793] While the user is using the app, the emotion engine recognizes the user's facial expressions and generates emotional data such as "tired." Based on this information, the server uses generative AI to suggest recipes such as "chicken and carrot soup with fatigue-relieving effects." These are displayed on the user's device, and the user can use the recipe to cook the dish.

[1794] In this way, the system of the present invention automates food expiration date management, improves user experience through emotion-based recipe suggestions, reduces food waste, and supports safe and efficient use of ingredients.

[1795] The processing flow will be explained below.

[1796] Step 1:

[1797] The user takes a photo of the expiration date on the food product they purchased. The user uses the camera on their smartphone to take a photo of the expiration date printed on the food product label.

[1798] Step 2:

[1799] The device will perform image recognition and automatically extract the expiration date, while the app on the device will use image processing technology to analyze and extract the expiration date from the image.

[1800] Step 3:

[1801] The user enters additional information such as the type of food and storage method. The user enters the name of the food, purchase date, and storage method through the app screen.

[1802] Step 4:

[1803] The terminal sends the collected food information to the server, which then packages the extracted expiration date and any additional information entered by the user and sends it to the server.

[1804] Step 5:

[1805] The server stores the food information in a database. The server analyzes the received food data and stores it in a database.

[1806] Step 6:

[1807] The server periodically checks the database to identify foods that are nearing their expiration date. A program on the server periodically scans the database to identify foods that are nearing their expiration date.

[1808] Step 7:

[1809] The server prepares notifications for the relevant foods and sends them to the terminal. The server generates notification messages for the identified foods and sends them to the terminal a set number of days in advance.

[1810] Step 8:

[1811] The terminal receives the notification and displays a warning to the user. The terminal receives the notification from the server and displays a warning message on the user's screen informing them that the expiration date is approaching.

[1812] Step 9:

[1813] The user operates the device to check how to store food. The user selects an option in the device's app to check how to store food.

[1814] Step 10:

[1815] The device sends a request to the server about how to store food.The device sends a request to the server about how to store food.

[1816] Step 11:

[1817] The server searches the database for information on the storage method. The server searches the database for information on the relevant storage method based on the request.

[1818] Step 12:

[1819] The server sends information on how to save the data to the device. The server compiles the search results and sends information on how to save the data to the device.

[1820] Step 13:

[1821] The device displays the storage method to the user. The device displays the received storage method information on the app screen so that the user can check it.

[1822] Step 14:

[1823] The device uses cameras and sensors to recognize the user's emotions, and the emotion engine analyzes the user's facial expressions and voice to generate emotion data.

[1824] Step 15:

[1825] The device sends the generated emotion data to the server. The device packages the user's emotion data and sends it to the server.

[1826] Step 16:

[1827] The server analyzes the user's emotional data. The server analyzes the received emotional data and identifies the user's emotional state.

[1828] Step 17:

[1829] The server takes into consideration emotional data, weather, seasonality, and the user's physical condition, and sends a request to the generation AI to generate a recipe. The server then sends a recipe request to the generation AI based on this information.

[1830] Step 18:

[1831] The generative AI generates the optimal recipe. Based on the information it receives from the server, the generative AI creates a recipe that takes into account the user's emotional state and other conditions.

[1832] Step 19:

[1833] The server saves the generated recipe and sends it to the device. The server temporarily saves the recipe information received from the generation AI in a database and sends it to the device.

[1834] Step 20:

[1835] The device displays the generated recipe to the user. The device displays the received recipe within the app, allowing the user to use the recipe as a reference when cooking.

[1836] Through these steps, the system of the present invention automates food expiration date management, improves user experience through emotion-based recipe suggestions, reduces food waste, and supports safe and efficient ingredient use.

[1837] Example 2

[1838] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1839] In modern society, the increase in food waste has become a major problem. This problem is particularly caused by poor expiration date management within the home. Furthermore, there is a lack of meal suggestions that take into account the user's emotions and physical condition, and there is a need for methods to use ingredients appropriately and without waste. Conventional systems have difficulty meeting these requirements, so an effective solution is needed to simultaneously manage food expiration dates and improve the user experience.

[1840] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1841] In this invention, the server includes a means for photographing the expiration dates of purchased items and storing them as data, a means for notifying the user of items approaching their expiration dates, a means for suggesting optimal storage methods for the items, and a means for analyzing the user's emotions, thereby reducing food waste and improving the user experience through emotion-based meal suggestions.

[1842] "Goods" refers to food or consumable items purchased or acquired by the user.

[1843] "Best before date" refers to the period during which an item can be consumed while maintaining its quality.

[1844] "Image recognition technology" refers to a form of computer vision that extracts and identifies specific information from image data.

[1845] "Data" refers to electromagnetically recorded information, including information such as expiration date, type of product, and storage method.

[1846] "Notification" refers to an alert or message that is displayed on a user's device.

[1847] "Preservation method" refers to the specific procedures and conditions for preserving an item in optimal condition.

[1848] "Cooking method" refers to the steps and methods for creating a dish using an item.

[1849] "Climate" refers to the weather conditions and temperature of the day.

[1850] "Season" refers to the time when something is at its most delicious and nutritious.

[1851] "User" refers to the consumer or end user who uses the system.

[1852] "Emotion" refers to the psychological state detected from the user's facial expressions, voice, etc.

[1853] "Emotion engine" is a general term for the algorithms and hardware that analyze the user's emotions and generate the results as data.

[1854] "Server" refers to a computer system that processes, stores, and analyzes data.

[1855] "Database" refers to a digital storage system for systematically storing and managing information such as expiration dates, storage methods, and emotional data.

[1856] "Generative AI model" refers to a machine learning model that generates useful results (such as recipes) based on input prompts.

[1857] A "prompt sentence" refers to an instruction sentence used as input to a generative AI model.

[1858] The present invention is a system that allows users to manage food expiration dates and receive cooking method suggestions based on their emotions. This system is composed of a terminal, a server, a database, and an emotion engine. Specific embodiments of the system are described below.

[1859] System Configuration

[1860] The system consists of a user device (smartphone or tablet), a server, a database, and an emotion engine. The server uses a powerful computer system to process, store, and analyze data. The database is a digital storage system for systematically storing and managing information such as expiration dates, storage methods, and emotion data. The emotion engine consists of hardware and software components for analyzing user emotions and generating the results as data.

[1861] Food information registration

[1862] To register the expiration date of purchased food, a user takes a photo of the food label using the device's camera. The device then uses image recognition technology to extract the expiration date as text data from the captured image. If necessary, the user can manually enter the type of food and storage method. The device then formats this data and sends an HTTP request to the server. The server then analyzes the received data and stores it in a database in the required format.

[1863] Best before date notification

[1864] The server periodically checks the database to identify foods that are approaching their expiration date. Based on user settings, the server creates and sends a notification three days before the expiration date, for example. The device receives this notification and displays a warning message to the user, such as "The milk's expiration date is three days away."

[1865] Preservation method

[1866] When a user registers a food item, the device requests the server for the optimal storage method for that food item. The server searches the database for the optimal storage method for that food item and sends it to the device as an HTTP response. The device then presents the received storage method to the user, displaying specific storage instructions such as "Store tomatoes at room temperature, or refrigerate if cut."

[1867] Emotion Recognition and Recipe Suggestion

[1868] While the user is using the app, emotion data is collected using the device's on-board camera and sensors. The device then analyzes the collected emotion data using an emotion engine and sends the results to the server. The server generates prompts based on the emotion data and other factors (weather, season, physical condition, etc.) and inputs them into the generative AI model. The server receives the resulting recipes generated by the generative AI model and sends them to the device. The device then presents specific recipes to the user, such as "chicken and carrot soup with fatigue-relieving effects."

[1869] Examples of prompt statements

[1870] Example prompts to input to a generative AI model:

[1871] "What recipes would you recommend to users who are feeling tired? For example, a simple soup recipe using foods that have a fatigue-relieving effect."

[1872] In this way, the system of the present invention automates food expiration date management, improves user experience through emotion-based cooking suggestions, reduces food waste, and supports safe and efficient use of ingredients.

[1873] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1874] Step 1:

[1875] The user takes a photo of the food they purchased on their device.

[1876] Input: A user takes a photo of a food label with their smartphone camera.

[1877] How it works: The user launches the camera app, frames a food label with an expiration date, and presses the capture button.

[1878] Output: The image data of the photographed food label is saved on the device.

[1879] Step 2:

[1880] The device uses image recognition technology to extract the expiration date.

[1881] Input: Image data acquired in step 1.

[1882] How it works: The device uses OCR (optical character recognition) technology to analyze the text data in the image and extract the expiration date as text data.

[1883] Output: Best before date data is obtained in text format.

[1884] Step 3:

[1885] The user enters any additional information needed (such as the type of food and storage method)

[1886] Input: Best before date data obtained in step 2 and manual user input information.

[1887] What it does: A user manually enters the type of food (e.g., milk, bread) and storage method (e.g., refrigerate, freeze) using an app input form.

[1888] Output: The expiration date data and the type of food and storage method data entered by the user are obtained.

[1889] Step 4:

[1890] The device sends the data to the server

[1891] Input: Best before date data obtained in step 3, food type, and storage method data.

[1892] How it works: The device formats this data and sends an HTTP request to the server.

[1893] Output: The data is sent to the server.

[1894] Step 5:

[1895] The server stores the received information in a database

[1896] Input: The data sent from the terminal in step 4.

[1897] How it works: The server parses the received data and stores it in the required format in the database.

[1898] Output: The expiration date, food type, and storage method data are saved in the database.

[1899] Step 6:

[1900] The server periodically checks the database

[1901] Input: Food information in the database.

[1902] How it works: The server runs a periodic job that checks all food items in the database.

[1903] Output: A list of foods that are nearing their expiration date is generated.

[1904] Step 7:

[1905] The server identifies food that is nearing its expiration date

[1906] Input: The list of foods obtained in step 6.

[1907] How it works: The server compares the expiration dates of the foods with the current date to identify foods that require notification.

[1908] Output: A list of foods that require notification is generated.

[1909] Step 8:

[1910] The server creates and sends the push notification

[1911] Input: The list of foods requiring notifications obtained in step 7.

[1912] How it works: The server creates a notification based on the user's settings, for example, three days before the expiration date, and sends a push notification to the user's device.

[1913] Output: A push notification is sent to the user's device.

[1914] Step 9:

[1915] The device receives the notification and displays it to the user.

[1916] Input: The push notification sent in step 8.

[1917] What it does: The device receives a notification from the server and displays a warning message such as "The milk's expiration date is coming up in three days."

[1918] Output: A notification message is displayed on the device screen.

[1919] Step 10:

[1920] Send a storage request when a user registers a food item

[1921] Input: The user enters the name of the food.

[1922] How it works: The user enters the name of a food item into the app's input form and presses the "Get storage instructions" button. The device then sends a request to the server.

[1923] Output: A save method request is sent to the server.

[1924] Step 11:

[1925] The server looks up the save method in the database

[1926] Input: The request submitted in step 10.

[1927] How it works: The server receives a request and searches its database for the best way to store the food.

[1928] Output: The search results for the saving method are obtained.

[1929] Step 12:

[1930] The server sends the save instructions to the device.

[1931] Input: Search results for saving methods obtained in step 11.

[1932] How it works: The server formats the search results and sends them to the device as an HTTP response.

[1933] Output: Information on how to save is sent to the device.

[1934] Step 13:

[1935] The device will prompt the user for instructions on how to save.

[1936] Input: The save method information submitted in step 12.

[1937] Operation: The device presents the received storage instructions to the user, displaying specific storage instructions such as "Store tomatoes at room temperature, or refrigerate if cut."

[1938] Output: Information about the saving method will be displayed on the device screen.

[1939] Step 14:

[1940] The device collects emotion data while the user is using the app.

[1941] Input: User's facial and voice data.

[1942] How it works: The user opens the app and uses the camera and sensors to capture facial expressions and voice in real time. The device then captures this emotional data.

[1943] Output: Emotion data is saved on the device.

[1944] Step 15:

[1945] The device analyzes the emotional data it acquires and sends it to the server.

[1946] Input: Emotion data obtained in step 14.

[1947] Operation: The device analyzes the facial expression data using the emotion engine, extracts the emotional state (e.g., "tired"), and sends it to the server.

[1948] Output: The analyzed emotion data is sent to the server.

[1949] Step 16:

[1950] The server generates a prompt and inputs it into the generative AI model.

[1951] Input: Emotion data sent in step 15 and other factors (weather, season, physical condition information).

[1952] How it works: The server uses this information to generate a prompt (e.g., "What recipes would you recommend for a tired user?") and inputs it into the generative AI model.

[1953] Output: The generated prompt sentence is input to the generative AI model.

[1954] Step 17:

[1955] The server sends the generated recipe to the device.

[1956] Input: The recipe obtained from the generative AI model in step 16.

[1957] How it works: The server receives the recipe generated by the AI ​​model and sends it to the device.

[1958] Output: Recipe data is sent to the device.

[1959] Step 18:

[1960] The device displays the recipe to the user

[1961] Input: Recipe data submitted in step 17.

[1962] How it works: The device displays the recipe it receives from the server on the screen, and suggests specific recipes such as "chicken and carrot soup with fatigue-relieving effects."

[1963] Output: The recipe is displayed on the device screen and provided to the user.

[1964] In this way, by linking each processing step with each other, the system of the present invention can effectively manage food expiration dates, suggest storage methods, and suggest cooking methods based on emotions.

[1965] (Application example 2)

[1966] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1967] In recent years, with the increasing importance of managing food expiration dates, there is a demand for suggestions of appropriate storage methods and recipes using food that is close to its expiration date. However, because individual factors such as emotions and physical condition are not taken into consideration, the user experience is often unsatisfactory. Furthermore, expiration date management is sometimes not thoroughly implemented in physical stores, leading to a serious problem of food waste. To solve these issues, a system that reflects user emotions and can also manage expiration dates in physical stores is needed.

[1968] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for photographing the expiration dates of purchased foods and saving them as data, means for notifying users of foods approaching their expiration dates, means for suggesting optimal food storage methods, means for providing recipes that combine foods with upcoming expiration dates, means for suggesting recipes taking into account the weather, seasons, and the user's physical condition on that day, means for recognizing the user's emotions and suggesting recipes based on the emotion data, and means for managing food in a physical store and suggesting recipes based on the emotions. This makes it possible to suggest recipes that correspond to the emotions of each individual user, reducing food waste and enabling efficient food management in physical stores.

[1969] The "best before date" of a food product is the date by which the food product will continue to maintain its appropriate quality.

[1970] "Storing as data" means electronically recording information such as expiration dates and making them accessible later.

[1971] "Notifying" means sending a warning or alert to the user when a specific condition is met.

[1972] "Best storage method" refers to the recommended storage environment and conditions to maintain the quality of food.

[1973] A "recipe" is a list of steps and ingredients for making a dish.

[1974] "Climate" refers to weather conditions such as temperature, humidity, wind speed, and precipitation over a certain period of time.

[1975] "Season" refers to the time when a particular food is most nutritious and delicious.

[1976] "User's physical condition" refers to the user's health condition and current physical condition.

[1977] "Emotion data" refers to emotional information obtained by analyzing the user's facial expressions, voice, behavior, etc.

[1978] A "brick and mortar store" is a physical retail establishment where consumers can visit and purchase products in person.

[1979] "Emotion recognition" means judging the user's emotional state by analyzing the user's facial expressions, etc.

[1980] "Emotion-based recipe suggestions" refers to taking into account the user's current emotional state and providing cooking ideas that are appropriate for that emotion.

[1981] "Managing food" means centrally monitoring and controlling food inventory, expiration dates, storage methods, etc.

[1982] "Image recognition technology" is a technology that analyzes image data and automatically recognizes specific information (such as characters or objects).

[1983] A "server" is a computer system that stores, processes, and manages data.

[1984] "Suggesting storage methods" means showing the user the best storage conditions for a particular food item.

[1985] "Providing recipe combinations" means teaching users how to combine multiple foods into one dish.

[1986] "User-specified" means that the user inputs his / her wishes and conditions into the system and makes a selection.

[1987] "Push notification" is a mechanism that notifies users of specific information in a timely manner.

[1988] This invention aims to effectively manage the expiration dates of food purchased by users and reduce food waste. Furthermore, by combining it with an emotion engine that recognizes user emotions, it can suggest recipes based on emotions, improving the user experience.

[1989] System Configuration

[1990] The system consists of a user device, a server, a database, and an emotion engine. Users use devices such as smartphones and tablets to manage expiration dates and recognize emotions. The server processes and stores data, and the database stores information about food and emotions. The emotion engine is a dedicated component for recognizing and analyzing user emotions.

[1991] Food information registration

[1992] To register the expiration date of purchased food, a user follows these steps: the user takes a photo of the food, and the device automatically extracts the expiration date using image recognition technology. If necessary, the user can enter additional information, such as the type of food and storage method. The device then sends this data to the server, which stores it in a database.

[1993] Specific examples

[1994] A user purchases milk and takes a photo of the expiration date, "December 15, 2023," with their smartphone. The device automatically extracts this date and sends it to the server as milk information along with the storage instruction, "Keep Refrigerated." The server then stores this information in a database.

[1995] Best before date notification

[1996] The server periodically checks the food information in the database and identifies foods that are approaching their expiration date. Based on the user's settings, it sends a push notification, for example, three days before the expiration date. The device receives this notification and displays a warning to the user.

[1997] Specific examples

[1998] The server checks the database and finds that the milk's expiration date is three days away. Because the notification period set by the user is three days in advance, the server notifies the device that "The milk's expiration date is approaching in three days." The device displays this notification to the user.

[1999] Preservation method

[2000] When a user registers a food item, the device requests the server for the optimal storage method for that food. The server searches the database for the storage method based on the type of food and sends it to the device. The device then displays this storage method to the user.

[2001] Specific examples

[2002] When a user registers a tomato, the device requests the server for "how to store tomatoes." The server sends the storage instructions, such as "Store tomatoes at room temperature, or refrigerate if cut," and displays them on the device.

[2003] Emotion Recognition and Recipe Suggestion

[2004] The app uses cameras and sensors installed on the device to recognize the user's emotions. While the user is using the app, the emotion engine analyzes the user's facial expressions and voice to generate emotion data. This data is sent to the server for analysis. Based on the analysis results, the server uses generative AI to suggest recipes that take into account the day's weather, seasonal ingredients, the user's physical condition, and emotions.

[2005] Specific examples

[2006] While the user is using the app, the emotion engine recognizes the user's facial expressions and generates emotional data such as "tired." Based on this information, the server uses generative AI to suggest recipes such as "chicken and carrot soup with fatigue-relieving effects." These are displayed on the user's device, and the user can use the recipe to cook the dish.

[2007] Examples of prompts to input to a generative AI model:

[2008] Generate a recipe for someone who is feeling tired and has the following items: carrots, chicken broth, onions, garlic.

[2009] Application in physical stores

[2010] This system can also be applied in physical stores. Employees and customers can use their smartphones to manage food in the store and suggest recipes based on their emotions. This will make it easier to manage expiration dates in the store and is expected to reduce food waste.

[2011] In this way, the system of the present invention automates food expiration date management, improves user experience through emotion-based recipe suggestions, reduces food waste, and supports safe and efficient use of ingredients.

[2012] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2013] Step 1:

[2014] The user takes a photo of the food they have purchased. Using the device, they take a photo of the label containing the expiration date with a camera, and the image data is used as input.

[2015] Step 2:

[2016] The device automatically extracts the expiration date using image recognition technology. It uses an image recognition algorithm (e.g., OpenCV) to analyze the characters in the image and identify the expiration date. The identified expiration date is the output.

[2017] Step 3:

[2018] If necessary, the user inputs additional information, such as the type of food or storage method. The information is supplemented through the device's user interface with text input or selection menus. The input additional information is output.

[2019] Step 4:

[2020] The device sends this data (best before date, food type, storage method) to the server. The data is transferred to the server through the network. The raw data sent to the server is the output.

[2021] Step 5:

[2022] The server stores the received data in a database. The data is structured and stored using a database management system (e.g., SQLite). The food information stored in the database is the output.

[2023] Step 6:

[2024] The server periodically checks the food information in the database to identify foods that are approaching their expiration date. It then runs a query to extract foods with an expiration date within three days. The extracted list of foods is the output.

[2025] Step 7:

[2026] The server sends push notifications for the identified foods based on the user settings. If the notification condition (e.g., three days before) is met, the server generates the notification content and sends it to the user's device. The notification message displayed on the device is the output.

[2027] Step 8:

[2028] The terminal receives the notification and displays a warning to the user. The notification message is displayed in a popup format to get the user's immediate attention. The warning displayed to the user is the output.

[2029] Step 9:

[2030] When a user registers a food item, the device requests the server for the optimal storage method for that food item. The storage method request is sent to the server. The sent request data is the output.

[2031] Step 10:

[2032] The server searches the database for a storage method based on the type of food and sends it to the terminal. It searches for a record that stores the appropriate storage method and transfers the storage method data. The storage method sent to the terminal is the output.

[2033] Step 11:

[2034] The terminal displays this saving method to the user. The appropriate saving conditions are displayed as text on the screen and provided to the user. The saving method displayed to the user is the output.

[2035] Step 12:

[2036] While the user is using the app, the emotion engine analyzes the user's facial expressions and voice to generate emotion data. Camera and microphone data is used as input to run emotion analysis algorithms (e.g., OpenCV, dlib) to identify the user's emotional state. The identified emotion data is the output.

[2037] Step 13:

[2038] Emotion data is sent to the server, where it is analyzed. The generated emotion data is sent to the server over the network and processed by the analysis component. The analyzed emotion information is the output.

[2039] Step 14:

[2040] Based on the analysis results, the server uses generative AI to suggest recipes that take into account the day's weather, seasonal ingredients, the user's physical condition, and emotions. A prompt is entered into the generative AI model, and the recipe generation results are received.

[2041] Example prompt sentence:

[2042] Generate a recipe for someone who is...

Claims

1. A way to photograph the expiration dates of purchased food and save them as data, a means of notifying consumers of food products approaching their expiration date; A means of suggesting optimal food preservation methods; A means to provide recipes that combine foods that are close to their expiration date, A method to suggest recipes taking into account the weather, seasonality, and the user's physical condition on that day; A system including:

2. 2. The system of claim 1, further comprising means for a user to set notification conditions.

3. The system according to claim 1, further comprising means for automatically extracting the expiration date of food using image recognition technology.

Images