System

Abstract

A system is provided.SOLUTION: An input unit configured to allow a user to input food ingredient information, a transmission unit configured to transmit the input food ingredient information to a server, a storage unit configured to store the transmitted food ingredient information in a database, a check unit configured to periodically check expiration dates of food ingredients stored in the database, a generation unit configured to generate notification data about a food ingredient whose expiration date is close, a notification unit configured to transmit the generated notification data to a terminal, a display unit configured to display the notification data to the user, a search unit configured to search for a recipe based on food ingredients in the database, and a recipe suggestion unit configured to transmit the found recipe to the terminal; A system comprising: SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's world, it is important to know the expiration dates of ingredients and use them without waste. However, many people forget the expiration dates of purchased ingredients or fail to manage them properly, resulting in food waste. It is also not easy to quickly find recipes that use leftover ingredients. To solve these problems, an efficient food management system is required. [Means for solving the problem]

[0005] The system of this invention solves the above problems by including the following means: an input means for a user to input information about ingredients, a transmission means for transmitting the input information about ingredients to a server, a storage means for saving the saved information about ingredients in a database, a check means for periodically checking the expiration dates of ingredients saved in the database, a generation means for generating notification data about ingredients whose expiration dates are approaching, a notification means for sending the generated notification data to a terminal, a display means for displaying the notification data to the user, a search means for searching for recipes based on ingredients in the database, and a recipe suggestion means for sending the searched recipes to a terminal. This allows users to effectively manage the expiration dates of purchased ingredients and easily receive suggestions for dishes using remaining ingredients.

[0006] A "user" is a person who uses this system to input and manage information about ingredients.

[0007] "Input means" refers to an interface or device that allows a user to input information about ingredients into a terminal.

[0008] "Transmission means" refers to a function or method for transmitting the input ingredient information to the server.

[0009] "Storage means" refers to a function or device for storing transmitted information about ingredients in a database.

[0010] "Checking means" refers to a function or process for periodically checking the expiration dates of ingredients stored in the database.

[0011] "Generation means" refers to a function or algorithm for generating notification data about food ingredients that are close to their expiration date.

[0012] The "notification means" refers to a function or method for transmitting the generated notification data to the terminal.

[0013] "Display means" refers to an interface or device for visually displaying notification data to a user.

[0014] "Search tool" refers to the functionality or algorithm used to search for recipes based on ingredients in the database.

[0015] The "recipe suggestion means" refers to a function or method for sending the searched recipe to the terminal and suggesting it to the user. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a system that efficiently manages the expiration dates of ingredients purchased by users and suggests recipes to make effective use of remaining ingredients. This system is realized by users inputting ingredient information and the server managing and processing that data.

[0038] To implement this system, the following main components are required:

[0039] 1. Register ingredients

[0040] Device:

[0041] It provides an interface for users to input information about purchased ingredients (such as name, purchase date, expiration date, etc.). For example, a smartphone app or web application can be used.

[0042] As a specific example, a user purchases "milk" and enters information such as the name "milk," the purchase date "October 5, 2023," and the expiration date "October 20, 2023" into the app.

[0043] 2. Sending and saving food information

[0044] Device:

[0045] It has a function to send the input ingredient information to the server using a protocol such as HTTP POST request.

[0046] Example: The information "milk" entered by the user is sent to the server.

[0047] server:

[0048] The received ingredient information is saved in a database. The server registers this information in the database in JSON format, SQL format, or other formats.

[0049] Example: The server stores the information about "milk" in a database.

[0050] 3. Best before date management and notification

[0051] server:

[0052] Regularly check the ingredient information in the database and list ingredients that are close to their expiration date. For example, run a script at a fixed time every day to list ingredients that will expire in the next three days.

[0053] Example: The server detects that the expiration date of "Milk" (October 20, 2023) is approaching.

[0054] server:

[0055] Notification data about food items nearing their expiration date is generated and sent to the user's device via email or push notification.

[0056] Example: The server generates a notification saying "The milk's expiration date is approaching" and sends it to the device.

[0057] Device:

[0058] Display the received notification data to the user, such as via a popup or notification bar.

[0059] Example: The device notifies the user that "The milk is nearing its expiration date."

[0060] 4. Recipe suggestions

[0061] server:

[0062] The system searches for available recipes based on ingredient information stored in the database, with priority given to recipes containing ingredients with an approaching expiration date.

[0063] Example: The server finds the recipe "Cream Stew" that uses "Milk".

[0064] Device:

[0065] The recipe information obtained from the server is displayed to the user, and multiple recipes are suggested so that the user can select the recipe that interests them.

[0066] Example: The terminal displays "Cream Stew Recipe" and the user can select it to view details.

[0067] summary

[0068] The system of this invention aims to streamline food management for users and reduce waste. Each processing step works in tandem, allowing users to easily understand the expiration date of ingredients and receive suggestions for recipes using remaining ingredients. This reduces food waste and improves the quality of their diet.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] The user inputs the ingredient information.

[0072] The user uses the device to input information such as the name of the ingredient, purchase date, expiration date, etc. Alternatively, the barcode scanning function can be used to automatically input ingredient information if necessary.

[0073] Step 2:

[0074] The terminal transmits the input ingredient information.

[0075] The device converts the ingredient information entered by the user into JSON format and sends it as an HTTP POST request to the server's API endpoint.

[0076] Step 3:

[0077] The server stores the ingredient information.

[0078] The server parses the received request, extracts the ingredient information, and saves it in the database. Once the saving is complete, it returns a successful save response to the device.

[0079] Step 4:

[0080] The server checks the expiration date.

[0081] The server runs a scheduled task every day at a fixed time, checking the expiration date of each ingredient stored in the database, comparing it with the current date, and listing ingredients that are close to expiry (e.g., within 3 days).

[0082] Step 5:

[0083] The server generates the notification data.

[0084] The server generates notification data about ingredients whose expiration date is approaching and prepares it for transmission to the terminal. The notification data includes the ingredient name and expiration date.

[0085] Step 6:

[0086] The device displays a notification to the user.

[0087] The device receives notification data from the server and displays it to the user as a pop-up or alert, allowing the user to check which ingredients are nearing their expiration date.

[0088] Step 7:

[0089] The server searches for available recipes.

[0090] When a user requests it, or the server periodically checks the database, it searches for recipes based on ingredients in stock from the recipe database. Recipes that include ingredients with an approaching expiration date are given priority.

[0091] Step 8:

[0092] The server transmits the recipe information to the terminal.

[0093] The server sends the searched recipe information to the device in JSON format, including the ingredients needed and cooking instructions.

[0094] Step 9:

[0095] The device displays recipe suggestions to the user.

[0096] The device displays the recipe information received from the server on the user interface, allowing the user to select a recipe of interest, check the details, and start cooking.

[0097] Example 1

[0098] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0099] Managing the expiration dates of ingredients is an important task in the home, and if not managed properly, food waste will occur. Furthermore, in order to efficiently utilize ingredients that are nearing their expiration date, appropriate recipe suggestions are required. However, manually managing expiration dates and finding recipes that make effective use of ingredients is time-consuming. There is a need to provide a smart system that solves these problems.

[0100] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0101] In this invention, the server includes an input means for a user to input ingredient information, a transmission means for transmitting the input ingredient information to the server, a storage means for saving the transmitted ingredient information in a database, a check means for periodically checking the expiration dates of the ingredients saved in the database, a generation means for generating notification data about ingredients with upcoming expiration dates, a notification means for sending the generated notification data to a terminal, a display means for displaying the notification data to the user, a search means for searching for recipes based on ingredients in the database, a recipe suggestion means for sending the searched recipes to the terminal, a check means including a script for periodically checking ingredient information in the database, and a search means for preferentially searching for recipes containing ingredients with upcoming expiration dates. This allows users to easily manage the expiration dates of ingredients and enables the system to suggest appropriate recipes for efficiently using ingredients with upcoming expiration dates.

[0102] "User" refers to a person who uses the system.

[0103] "Ingredient information" refers to detailed data about ingredients, such as the name of the ingredient, purchase date, and expiration date.

[0104] "Input means" refers to the interface through which the user inputs information about ingredients, such as a smartphone app or web application.

[0105] "Transmission means" refers to a function for transmitting the input information about ingredients to the server.

[0106] "Storage means" refers to a function for storing transmitted information about ingredients in a database.

[0107] "Checking means" refers to a function that periodically checks the expiration dates of ingredients stored in the database.

[0108] "Generation means" refers to a function that creates notification data about food ingredients that are close to their expiration date.

[0109] The "notification means" refers to a function for transmitting the generated notification data to the user's terminal.

[0110] The "display means" refers to a function for displaying received notification data to the user.

[0111] "Search means" refers to the function for searching for recipes based on ingredients in the database.

[0112] "Recipe suggestion means" refers to a function for sending searched recipes to the user's terminal and suggesting them.

[0113] "Script" refers to program code that runs on the server to periodically check ingredient information in the database.

[0114] A "push notification" is a notification sent directly from the server to the user's device, and can be received even if the user does not have the app open.

[0115] "Database" refers to the storage system where information about ingredients is stored.

[0116] MODE FOR CARRYING OUT THE INVENTION

[0117] This invention is a system that efficiently manages the expiration dates of ingredients purchased by users and suggests recipes for making effective use of remaining ingredients. This system is realized by users inputting ingredient information and a server managing and processing that data. Specific embodiments of this system are described below.

[0118] First, the user uses a smartphone app or web application to input information about ingredients (such as name, purchase date, expiration date, etc.) For example, if a user purchases "milk," they enter information such as "milk," "October 5, 2023," and "October 20, 2023" into the app's input form and press the "Submit" button.

[0119] The device has a means of sending the input ingredient information to the server. This is done using an HTTP POST request, and the information is sent to the server. The server has a means of saving the received ingredient information to a database, and registers the received information in the database as a new record. For example, information such as "Milk," "October 5, 2023," and "October 20, 2023" is saved in the database.

[0120] The server periodically checks the ingredient information in the database and has a means of checking to list ingredients that are close to their expiration date. To do this, a batch process is executed at a fixed time every day using a script such as Python. For example, a script is executed every day at 9:00 AM to check ingredients that have an expiration date of less than three days.

[0121] The server has a generating means for generating notification data regarding ingredients whose expiration date is approaching. The generated notification data is sent by a notification means that sends the data to the user's device using push notification or email. For example, notification data stating "Milk's expiration date is approaching" is generated and sent to the device. The user can check the received notification on the device, which is displayed in a notification bar or a pop-up window.

[0122] Next, the server has a search means for searching available recipes based on the ingredient information stored in the database. It gives priority to searching for recipes that contain ingredients with an approaching expiration date. For example, it is possible to find a recipe called "cream stew" that uses "milk." The searched recipes are sent from the server to the user's terminal and are then sent via the recipe suggestion means. The user can check multiple suggested recipes on the terminal, select the recipe of interest, and view the details.

[0123] As a concrete example, the following shows the process from when a user purchases "milk" and enters and saves the information in the app. The user enters "milk," "October 5, 2023," and "October 20, 2023" into the app's input form and presses the "Submit" button. The app sends an HTTP POST request, and the server receives this request and stores the information in a database. The server runs a script every day at 9:00 AM to check ingredients with a best-by date of within three days, generates a notification stating "The milk's expiration date is approaching," and sends it as a push notification to the user's device. The user receives the notification and checks it on their device. The server searches for recipes using "milk," finds dishes such as "cream stew," and suggests them to the user. The user can then view the suggested recipes in the app and check their details.

[0124] An example prompt is, "I want to develop an app that allows you to enter the name, purchase date, and expiration date of purchased ingredients, manages that information, and notifies you when the expiration date is approaching. This app will provide recommended recipes based on the saved data. Please tell me the specific steps and data flow model."

[0125] In this way, the entire system works together, allowing users to easily manage the expiration dates of ingredients and efficiently use ingredients that are close to their expiration date when cooking, thereby reducing food waste and improving the quality of their diet.

[0126] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0127] Step 1:

[0128] The user inputs information about the ingredients.

[0129] Specific operation: The user opens a smartphone app or web application, enters the name of the purchased food, the purchase date, and the expiration date. For example, enter "Milk," "October 5, 2023," and "October 20, 2023," and presses the "Submit" button.

[0130] Input: Name of ingredient, purchase date, and expiration date.

[0131] Output: The input ingredient information is stored on the terminal.

[0132] Step 2:

[0133] The terminal transmits the input ingredient information to the server.

[0134] Specific operation: The input information is converted into JSON format and sent to the server using an HTTP POST request.

[0135] Input: Ingredient information stored on the device (JSON format).

[0136] Output: Ingredient information sent to the server.

[0137] Step 3:

[0138] The server stores the received ingredient information in a database.

[0139] What happens: The server parses the received JSON data, extracts the ingredient name, purchase date, and expiration date, and stores this information as a new record in the database.

[0140] Input: JSON formatted ingredient information sent to the server.

[0141] Output: A new record of ingredient information stored in the database.

[0142] Step 4:

[0143] The server periodically checks the food ingredient information in the database and lists ingredients that are close to their expiration date.

[0144] What it does: The server runs a Python script every day at 9 AM, which checks the expiration date field in the database and lists ingredients that have an expiration date within the next 3 days.

[0145] Input: Ingredient information stored in the database.

[0146] Output: A list of ingredients that are close to expiry.

[0147] Step 5:

[0148] The server generates notification data regarding food ingredients whose expiration date is approaching.

[0149] Specific behavior: Generate notification content based on the listed ingredients. For example, create a message saying "Milk is nearing its expiration date."

[0150] Input: A list of ingredients that are close to expiry.

[0151] Output: The generated notification data.

[0152] Step 6:

[0153] The server transmits the generated notification data to the terminal.

[0154] Specific operation: The generated notification data is sent to the user's device as a push notification or email.

[0155] Input: The generated notification data.

[0156] Output: Notification data sent to the user's device.

[0157] Step 7:

[0158] The terminal displays the received notification data to the user.

[0159] Specific behavior: The device receives the notification and displays it to the user in a pop-up or notification bar, for example, a notification saying "Your milk is nearing its expiration date."

[0160] Input: Notification data sent to the terminal.

[0161] Output: The notification data displayed to the user.

[0162] Step 8:

[0163] The server searches for available recipes based on the ingredient information stored in the database.

[0164] Specific behavior: To prioritize recipes that contain ingredients with an approaching expiration date, the name of the ingredient is used as a query to search for related recipes. For example, searching for recipes that use "milk" will find "cream stew."

[0165] Input: Ingredient information stored in the database.

[0166] Output: A list of searched recipes.

[0167] Step 9:

[0168] The server sends the searched recipe to the terminal.

[0169] Specific operation: The searched recipe information is sent to the user's terminal.

[0170] Input: A list of searched recipes.

[0171] Output: Recipe information sent to the user's device.

[0172] Step 10:

[0173] The terminal displays the recipe information to the user.

[0174] Specific operation: The device displays the received recipe information and allows the user to select a recipe they are interested in and view its details. For example, the device can display "cream stew recipe" and view its details.

[0175] Input: Recipe information sent to the device.

[0176] Output: The recipe information displayed to the user.

[0177] (Application example 1)

[0178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0179] In modern society, busy users often find it difficult to keep track of the expiration dates of ingredients they purchase, resulting in food waste. Furthermore, they often lack the time to find appropriate recipes that make effective use of ingredients. To solve these problems, a system is needed that efficiently manages expiration dates and automatically suggests recipes that use ingredients with approaching expiration dates.

[0180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0181] In this invention, the server includes an input means for a user to input ingredient information, a transmission means for transmitting the input ingredient information to the server, a storage means for saving the transmitted ingredient information in a database, a check means for periodically checking the expiration dates of the ingredients saved in the database, a generation means for generating notification data for ingredients whose expiration dates are approaching, a notification means for sending the generated notification data to a terminal, a display means for displaying the notification data to the user, a search means for searching for recipes based on ingredients in the database, a recipe suggestion means for sending the searched recipes to the terminal, a scanning means for scanning receipts to automatically recognize ingredient information, a calculation means for calculating the expiration date from the recognized ingredient information, and a notification function for sending expiration date notifications to the user's terminal. This allows users to efficiently manage ingredient expiration dates and receive suggested recipes that utilize ingredients without waste.

[0182] The "input means" is an interface that allows the user to input information about ingredients into the terminal.

[0183] The "transmission means" is a device or software function that transmits the input information about ingredients to the server.

[0184] "Storage means" refers to a device or software function for storing the transmitted information about ingredients in a database.

[0185] The "checking means" is a device or software function that periodically checks the expiration dates of ingredients stored in the database.

[0186] The "generation means" is a device or software function that generates notification data about food ingredients that are close to their expiration date.

[0187] The "notification means" is a device or software function that transmits the generated notification data to the user's terminal.

[0188] The "display means" is an interface for displaying notification data to the user.

[0189] A "search means" is a device or software function that searches for recipes based on ingredients in a database.

[0190] The "recipe suggestion means" is a device or software function that sends the searched recipes to the user's terminal.

[0191] "Scanning means" refers to a device or software function that scans receipts and automatically recognizes ingredient information.

[0192] The "calculation means" is a device or software function that calculates the expiration date from the information of the recognized ingredients.

[0193] A "notification function" is a device or software function that sends expiration date notifications to a user's terminal.

[0194] This invention provides a system for efficiently managing expiration dates of ingredients purchased by a user and proposing recipes for making effective use of remaining ingredients. This system includes the following means.

[0195] 1. Input Method

[0196] Users enter food information using a smartphone app or web application, either manually entering the name of the purchased food, the purchase date, and the expiration date, or automatically by scanning the receipt.

[0197] 2. Transmission Method

[0198] The ingredient information entered by the user is sent to the server using an HTTP POST request. Data is sent from the smartphone app or web application to the server.

[0199] 3. Preservation means

[0200] The server receives the transmitted ingredient information and stores it in a database, which records the ingredient name, purchase date, and expiration date.

[0201] 4. Checking Methods

[0202] The server periodically runs a script to check expiration dates in the database, for example, listing ingredients that are close to expiry at a set time each day.

[0203] 5. Generation means

[0204] The server generates notification data for ingredients that are nearing their expiration date, including the name of the ingredient, its expiration date, and a warning message.

[0205] 6. Means of notification

[0206] The generated notification data is sent to the user's device via email or push notification. SMS notifications can also be sent using the Twilio API.

[0207] 7. Display means

[0208] The user's device will display the received notification data in a pop-up or notification bar, informing the user that the expiration date of the food item is approaching.

[0209] 8. Search Methods

[0210] The server searches for available recipes based on the ingredient information stored in the database, with priority given to recipes that contain ingredients with an approaching expiration date.

[0211] 9. Recipe suggestion methods

[0212] The recipe information obtained from the server is sent to the user's device, where the user can select the recipe they are interested in and check the detailed information.

[0213] 10. Scanning Method

[0214] Use your smartphone camera and OCR (optical character recognition) software (e.g., pytesseract) to scan receipts and automatically recognize ingredient information.

[0215] 11. Means of calculation

[0216] Use data analysis algorithms to calculate expiration dates from recognized ingredient information. Calculate detailed information about ingredients that are nearing their expiration date.

[0217] 12. Notification function

[0218] The notification function uses the Twilio API, Firebase Cloud Messaging, etc. to send expiration date notifications to users' devices, allowing them to take immediate action.

[0219] Specific examples

[0220] Suppose a user purchases "milk" and scans the receipt with a smartphone app. The app uses OCR technology to extract information about the "milk," parses / enters the purchase date and expiration date, and sends it to the server. The server stores this information in a database and periodically checks the expiration date. When the expiration date approaches, the server sends a notification to the user's smartphone saying, "Your milk is about to expire." At the same time, a recipe using milk (e.g., cream stew) is suggested, and the user can refer to the recipe to cook.

[0221] Example prompts for generative AI models

[0222] To create a food management application, we manage the expiration dates of ingredients purchased by the user and automatically register the ingredient information through receipt scanning and OCR analysis. It also includes a function to send notifications to a smartphone when ingredients are approaching their expiration date. For example, if a receipt says "Milk, Purchase Date 2023-10-05, Expiration Date 2023-10-20," the relevant information is registered in the application, and a notification is sent when the expiration date approaches within three days.

[0223] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0224] Step 1:

[0225] The user inputs the ingredient information.

[0226] Users use a smartphone app to input the name, purchase date, and expiration date of the ingredients they purchased, or scan a receipt to automatically recognize the ingredient information. The input data is the ingredient name, purchase date, and expiration date.

[0227] Step 2:

[0228] The terminal transmits the input ingredient information to the server.

[0229] The smartphone app sends the entered ingredient data to the server using an HTTP POST request. The device sends the ingredient name, purchase date, and expiration date as parameters. The output is a communication success or failure status.

[0230] Step 3:

[0231] The server stores the received ingredient information in a database.

[0232] The server parses the received JSON format data and registers the ingredient name, purchase date, and expiration date in the database. The input data is the ingredient information, and the output is the result saved in the database.

[0233] Step 4:

[0234] The server periodically checks the expiration dates in the database.

[0235] Run the script periodically (for example, every day at a set time) to list ingredients that are close to expiry. The input data is all the ingredients in the database, and the output is a list of ingredients that are close to expiry.

[0236] Step 5:

[0237] The server generates notification data about food ingredients that are close to their expiration date.

[0238] Based on a list of ingredients whose expiration date is approaching, notification data for each ingredient is generated. The notification data includes the ingredient name, expiration date, and warning message. The input data is a list of ingredients whose expiration date is approaching, and the output is the generated notification data.

[0239] Step 6:

[0240] The server transmits the generated notification data to the terminal.

[0241] The server sends the notification data to the user's smartphone as an email or push notification. The input data is the notification data, and the output is the transmission result.

[0242] Step 7:

[0243] The terminal displays the received notification data.

[0244] The user's smartphone displays the notification data in a pop-up or notification bar to notify the user that the expiration date is approaching. The input data is the notification data, and the output is the displayed notification message.

[0245] Step 8:

[0246] The server searches for recipes based on the ingredient information in the database.

[0247] The system prioritizes the search for recipes containing ingredients with upcoming expiration dates and suggests them to the user. The input data is information about ingredients with upcoming expiration dates, and the output is a list of searched recipes.

[0248] Step 9:

[0249] The server transmits the searched recipe information to the terminal.

[0250] The server sends the searched recipe list to the user's smartphone, where the user can check the details. The input data is the recipe list, and the output is the transmission result and displayed recipe information.

[0251] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0252] This invention is a system that efficiently manages the expiration dates of ingredients purchased by the user, suggests recipes to make effective use of remaining ingredients, and recognizes the user's emotions to make appropriate suggestions.By combining an emotion engine with the ingredient management function, this system can make suggestions based on the user's emotional state.

[0253] To implement this system, the following main components are required:

[0254] 1. Register ingredients

[0255] Device:

[0256] It provides an interface for users to input information about ingredients they have purchased (such as name, purchase date, and expiration date). As a specific example, a user purchases "milk" and enters information such as the name "Milk," purchase date "October 5, 2023," and expiration date "October 20, 2023" into the app. It is also possible to input this information automatically using the barcode scanning function.

[0257] 2. Sending and saving food information

[0258] Device:

[0259] It has a function to send the inputted ingredient information to the server using a protocol such as HTTP POST request.

[0260] Example: The information "milk" entered by the user is sent to the server.

[0261] server:

[0262] The received ingredient information is saved in a database. The server registers this information in the database in JSON format, SQL format, or other formats.

[0263] Example: The server stores the information about "milk" in a database.

[0264] 3. Best before date management and notification

[0265] server:

[0266] Regularly check the ingredient information in the database and list ingredients that are close to their expiration date. For example, run a script at a fixed time every day to list ingredients that will expire in the next three days.

[0267] Example: The server detects that the expiration date of "Milk" (October 20, 2023) is approaching.

[0268] server:

[0269] Generate notification data about food items that are nearing their expiration date and prepare to notify users. This can be done via email or push notification.

[0270] Example: The server generates a notification saying "The milk's expiration date is approaching" and sends it to the device.

[0271] Device:

[0272] Display the received notification data to the user, such as via a popup or notification bar.

[0273] Example: The device notifies the user that "The milk is nearing its expiration date."

[0274] 4. Recipe suggestions

[0275] server:

[0276] The system searches for available recipes based on ingredient information stored in the database, with priority given to recipes containing ingredients with an approaching expiration date.

[0277] Example: The server finds the recipe "Cream Stew" that uses "Milk".

[0278] Device:

[0279] The recipe information obtained from the server is displayed to the user, and multiple recipes are suggested so that the user can select the recipe that interests them.

[0280] Example: The terminal displays "Cream Stew Recipe" and the user can select it to view details.

[0281] 5. Operation of the Emotion Engine

[0282] Device:

[0283] It provides a function for recognizing the user's emotions in real time, using facial recognition via a camera and analysis based on the user's input behavior.

[0284] server:

[0285] The system receives user emotional data and optimizes recipe suggestions and notification content. For example, if the user is tired, it will suggest easy-to-make dishes.

[0286] Example: If the user is recognized as "tired," the server will suggest "easy-to-make tamagoyaki (rolled omelet)."

[0287] Device:

[0288] The emotional engine displays optimized recipes and notifications to users, providing optimal suggestions tailored to the emotional state the user is facing.

[0289] Example: The device will display "Easy recipe recommended for when you're tired: Tamagoyaki."

[0290] summary

[0291] The system of this invention not only effectively manages expiration dates of ingredients, but also enhances user convenience and satisfaction by suggesting and notifying recipes that take the user's emotional state into consideration. Various elements work together to reduce food waste and improve the user's quality of life.

[0292] The processing flow will be explained below.

[0293] Step 1:

[0294] The user inputs the ingredient information.

[0295] The user enters information about the ingredients they purchased using an application on their device. The information to be entered includes the name of the ingredient, the purchase date, and the expiration date. This information can also be entered automatically using the barcode scanning function. As a specific example, a user purchases "milk" and enters the name "milk," the purchase date "October 5, 2023," and the expiration date "October 20, 2023."

[0296] Step 2:

[0297] The terminal transmits the ingredient information to the server.

[0298] The device converts the entered ingredient information into JSON format and sends it as an HTTP POST request to the server's API endpoint. This request includes the ingredient name, purchase date, and expiration date. For example, the information for "milk" is sent to the server.

[0299] Step 3:

[0300] The server stores the ingredient information in a database.

[0301] The server parses the received request, extracts the ingredient information, and saves it in the database. The server returns a save completion response to the terminal. For example, the server saves the information for "milk" in the database.

[0302] Step 4:

[0303] The server checks the expiration date.

[0304] The server runs a scheduled task every day at a fixed time, checking the expiration date of each ingredient in the database. It compares this to the current date and lists ingredients that are close to expiring (for example, within 3 days). For example, the server detects that "Milk" is close to expiring.

[0305] Step 5:

[0306] The server generates the notification data.

[0307] The server generates notification data about ingredients whose expiration date is approaching and prepares to send it to the user. This notification data includes the ingredient name and expiration date. For example, a notification saying "Milk's expiration date is approaching" is generated.

[0308] Step 6:

[0309] The device displays the notification to the user.

[0310] The device receives notification data from the server and displays it in a pop-up or notification bar. The user can check which ingredients are nearing their expiration date. For example, the device may notify the user that "The expiration date of your milk is approaching."

[0311] Step 7:

[0312] The server searches for available recipes.

[0313] The server searches for available recipes based on the ingredient information stored in the database. It prioritizes recipes that contain ingredients with an approaching expiration date. For example, the server finds a recipe called "cream stew" that uses "milk."

[0314] Step 8:

[0315] The server transmits the recipe information to the terminal.

[0316] The server sends the searched recipe information to the device in JSON format. The recipe information includes the necessary ingredients and cooking instructions. For example, a "cream stew recipe" is sent to the device.

[0317] Step 9:

[0318] The device displays recipe suggestions to the user.

[0319] The device displays the recipe information received from the server and allows the user to select the recipe they are interested in. For example, the device displays "cream stew recipe" and the user can select it and view the details.

[0320] Step 10:

[0321] The device recognizes the user's emotions.

[0322] The device's camera and sensors are used to analyze the user's facial expressions and voice to obtain emotional data, detecting whether the user is "tired" or "stressed."

[0323] Step 11:

[0324] The server optimizes the recipe based on emotional data.

[0325] The server receives the user's emotional data and adjusts the recipe suggestions and notification content based on that data. For example, if the user is tired, it will prioritize suggestions of easy-to-make dishes. For example, if the user is tired, it will suggest "easy-to-make tamagoyaki" (rolled omelet).

[0326] Step 12:

[0327] The device displays recipes and notifications optimized by the emotion engine.

[0328] The device uses the emotion engine sent from the server to display optimized recipe information and notifications to the user. For example, it displays information such as "Easy recipe recommended for when you're tired: Tamagoyaki."

[0329] Through these steps, this system can provide users with a more personalized experience by combining food expiration date management and emotion recognition.

[0330] Example 2

[0331] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0332] Many conventional food management systems are specialized in managing and notifying expiration dates, and have the problem of not being able to make suggestions that take the user's emotions into consideration. As a result, the recipes provided may not be suitable for the user's situation, which can result in a decrease in user convenience and satisfaction. Another problem is that ingredients are often entered manually, which is a time-consuming and labor-intensive process.

[0333] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0334] In this invention, the server includes input means for a user to input food information, transmission means for transmitting the input food information to the server, storage means for saving the transmitted food information in a database, check means for periodically checking the expiration dates of the foods saved in the database, generation means for generating notification data for foods with upcoming expiration dates, notification means for sending the generated notification data to a terminal, display means for displaying the notification data to the user, search means for searching for recipes based on foods in the database, recipe suggestion means for sending the searched recipes to the terminal, emotion recognition means for recognizing the user's emotion, emotion data transmission means for sending emotion data recognized by the emotion recognition means to the server, and optimization means for optimizing recipe suggestions based on the emotion data. This not only enables food expiration date management but also makes it possible to suggest optimal recipes tailored to the user's emotions.

[0335] "Input means" refers to a device or interface that allows a user to input information about a food product.

[0336] "Transmission means" refers to a device or protocol that transmits the input food information to the server.

[0337] "Storage means" means a device or process that stores the submitted food information in a database.

[0338] A "checking means" is a device or script that periodically checks the expiration dates of food items stored in the database.

[0339] A "generator" is a device or algorithm that generates notification data about food products that are nearing their expiration date.

[0340] The "notification means" is a device or protocol that transmits the generated notification data to the terminal.

[0341] A "display means" is a device or interface that displays notification data to a user.

[0342] A "search tool" is a device or algorithm that searches for recipes based on foods in the database.

[0343] A "recipe suggestion means" is a device or protocol that sends searched recipes to a terminal.

[0344] An "emotion recognition means" is a device or algorithm that recognizes a user's emotions.

[0345] The "emotion data transmission means" is a device or protocol that transmits the emotion data recognized by the emotion recognition means to the server.

[0346] An "optimization means" is a device or algorithm that optimizes recipe suggestions based on emotion data.

[0347] This invention is a system that efficiently manages the expiration dates of food purchased by a user, suggests recipes for making effective use of remaining food, and recognizes the user's emotions to make appropriate suggestions. Specific embodiments of this invention are described below.

[0348] Hardware and Software Configuration

[0349] Input method: Provides an interface for users to input food information. Specifically, this corresponds to an application installed on a device such as a smartphone or tablet. Users can manually enter the food name, purchase date, and expiration date, or use a barcode scanning function to automatically enter them.

[0350] Transmission method: HTTP POST request is used as the protocol to send the entered food information to the server. For example, when a user enters information about "milk," this information is sent to the server in JSON format.

[0351] Storage method: The server stores the received food information in a database using SQL Server or NoSQL database, and the food information is stored in an appropriate format.

[0352] Check method: The server periodically runs a script to check the food information in the database. This process can be automated using a cron job or similar, and lists foods that have a shelf life of less than three days.

[0353] Generation method: Generates notification data for foods in the database that are close to their expiration date. The generated notification data is prepared in a format that notifies the user via email, push notification, etc.

[0354] Notification method: The generated notification data is sent to the device. This is done using a mail server or push notification service. The notification data is displayed in a pop-up or notification bar so that the user can check it immediately.

[0355] Display method: The device displays the notification data received from the server to the user. Specifically, the device uses the smartphone's notification bar or push notification to allow the user to quickly check the information.

[0356] Search method: The server searches for available recipes based on the food information stored in the database, with priority given to searching and suggesting recipes that include food items with an approaching expiration date.

[0357] Recipe suggestion method: The server sends the searched recipes to the terminal and suggests multiple recipes, allowing the user to select the recipe they are interested in and check the details.

[0358] Emotion recognition: The device provides a function to recognize the user's emotions in real time. Emotion recognition can be achieved by facial recognition via a camera or by analyzing the user's input behavior.

[0359] Emotion data transmission method: The recognized emotion data is transmitted from the device to the server. Based on this data, the server optimizes the next recipe suggestion.

[0360] Optimization method: The server receives the emotional data and suggests recipes according to the user's emotional state. For example, if the server detects that the user is "tired," it will prioritize suggestions for easy-to-make dishes.

[0361] Specific examples and prompts for the generative AI model

[0362] Examples:

[0363] It explains how the system notifies users when the milk they have purchased is nearing its expiration date and suggests suitable recipes.

[0364] Specific operation example: A user purchases "milk" and enters the name "milk," the purchase date "October 5, 2023," and the expiration date "October 20, 2023" into the device. The entered information is sent to the server and stored in a database. The server periodically checks the database and lists foods with an expiration date of three days or less. It then generates notification data stating "The milk's expiration date is approaching" and sends it to the user's device. When the user receives the notification, the server provides a recipe for "cream stew."

[0365] Example prompt for a generative AI model:

[0366] Explain how the system will notify the user when the "Milk" they have purchased is nearing its expiration date and suggest suitable recipes.

[0367] This invention not only enables the expiration date of food products to be managed, but also makes it possible to propose optimal recipes that take into account the user's feelings, thereby improving user convenience and satisfaction.

[0368] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0369] Specific processing steps of the system program

[0370] Step 1: User enters food information

[0371] The user inputs information about the food they have purchased into the terminal, such as the name of the food, the date of purchase, the expiration date, etc. This input is performed using a keyboard or a barcode scanner.

[0372] Input: food name, purchase date, expiration date

[0373] Output: Input data on the terminal

[0374] What it does: A user opens the app, enters "Milk," sets the purchase date to "2023-10-05," and the expiration date to "2023-10-20." By using the barcode scanner, this information is automatically entered.

[0375] Step 2: Device sends data to server

[0376] The terminal sends the food information entered by the user to the server. The data is sent via an HTTP POST request in JSON format.

[0377] Input: Food information entered by the user into the device

[0378] Output: JSON data sent to the server

[0379] Specific operation: The device sends JSON data (e.g., "{\"name\": \"Milk\", \"purchase_date\": \"2023-10-05\", \"expiry_date\": \"2023-10-20\"}") to the server using "POST / api / food_register".

[0380] Step 3: Data received and stored by the server

[0381] The server receives the food information sent from the device and stores it in a database using SQL queries.

[0382] Input: JSON data sent from the terminal

[0383] Output: Food information stored in the database

[0384] Specific operation: The server executes the SQL query "INSERT INTO food_table (name, purchase_date, expiration_date) VALUES (\'Milk\', \'2023-10-05\', \'2023-10-20\')" and saves it in the database.

[0385] Step 4: Expiry date check by server

[0386] The server runs a script that periodically checks the food information in the database. This script is run by a cron job at midnight every day and lists foods that have an expiration date within the next three days.

[0387] Input: Food information stored in a database

[0388] Output: A list of foods that are close to expiry date

[0389] Specific operation: The server executes the SQL query "SELECT FROM food_table WHERE best-before_date <= DATE_ADD(CURDATE(), INTERVAL 3 DAY)".

[0390] Step 5: Notification data generation by the server

[0391] The server generates notification data about food products that are nearing their expiration date, and the generated data is ready to send notifications to users.

[0392] Input: List of foods that are close to expiry date

[0393] Output: Generated notification data

[0394] Specific behavior: The server generates a notification message saying "The milk is nearing its expiration date."

[0395] Step 6: Send notification data to the device

[0396] The server then sends the generated notification data to the user's device, which can take the form of email or push notification.

[0397] Input: Notification data

[0398] Output: Notification sent to the device

[0399] Specific operation: The server sends the notification data via a push notification service or mail server.

[0400] Step 7: Displaying notifications on your device

[0401] The device displays the received notification data to the user in a pop-up or notification bar format.

[0402] Input: Notification data sent from the server

[0403] Output: The notification displayed to the user

[0404] Specific behavior: The device will display a pop-up notification saying "The milk's expiration date is approaching."

[0405] Step 8: Server Lookup for Recipes

[0406] The server searches for available recipes based on the food information stored in the database, with priority given to recipes that include food items with an approaching expiration date.

[0407] Input: Food information in the database

[0408] Output: Searched recipe list

[0409] Specific behavior: The server executes the SQL query "SELECT FROM recipe WHERE ingredients LIKE \"%milk%\"".

[0410] Step 9: Send the recipe to your device

[0411] The server sends the searched recipes to the terminal, and multiple recipes are suggested to the user.

[0412] Input: Searched recipe list

[0413] Output: Recipe information sent to the device

[0414] Specific operation: The server sends recipe information to the terminal and displays it to the user.

[0415] Step 10: Emotion recognition on the device

[0416] The device will provide features for recognizing users' emotions in real time, including camera-based facial recognition and behavioral analysis.

[0417] Input: User's facial expressions and behavior data

[0418] Output: Recognized emotion data

[0419] Specific operation: The device analyzes the user's face captured by the camera and recognizes emotions such as "tired."

[0420] Step 11: Send emotion data to the server

[0421] The device sends the recognized emotion data to the server, which then uses it to suggest the next recipe.

[0422] Input: Recognized emotion data

[0423] Output: Emotion data sent to the server

[0424] Specific operation: The device sends emotion data in JSON format to the server using "POST / api / emotion_data".

[0425] Step 12: Optimizing recipe suggestions by the server

[0426] The server optimizes recipe suggestions based on emotion data, and selects recipes that suit the user's state.

[0427] Input: Emotion data, recipe information in the database

[0428] Output: Optimized recipe suggestions

[0429] Specific operation: For users who are recognized by the server as being "tired," the server will prioritize suggesting easy-to-make recipes for "tamagoyaki" (rolled omelet).

[0430] (Application example 2)

[0431] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0432] In modern consumer lifestyles, it is difficult for users to efficiently manage the expiration dates of purchased ingredients, resulting in food waste. It is also not easy for users to select the optimal recipe based on their emotional state at any given time. At the same time, there is a demand for systems that can suggest appropriate recipes based on emotions, but no such systems have existed to date. This situation has led to problems that reduce user convenience and satisfaction.

[0433] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0434] In this invention, the server includes emotion recognition means for recognizing the emotional state of the user, recipe optimization means for optimizing recipes based on the recognized emotional state, and generation means for generating notification data for ingredients whose expiration dates are approaching, thereby enabling the server to efficiently manage the expiration dates of ingredients purchased by the user and propose optimal recipes according to the user's emotional state.

[0435] The "input means" is an interface through which the user inputs information about ingredients.

[0436] The "transmission means" is a function that transmits the input information about ingredients to the server.

[0437] The "storage means" is a function that stores the transmitted information about ingredients in a database.

[0438] The "checking means" is a function that periodically checks the expiration dates of ingredients stored in the database.

[0439] The "generation means" is a function that generates notification data about food ingredients that are close to their expiration date.

[0440] The "notification means" is a function that transmits the generated notification data to the terminal.

[0441] The "display means" is a function that displays the notification data to the user.

[0442] "Search means" is a function that searches for recipes based on ingredients in the database.

[0443] The "recipe suggestion means" is a function that sends the searched recipe to the terminal.

[0444] The "emotion recognition means" is a function for recognizing the user's emotional state.

[0445] The "recipe optimizer" is a function that optimizes recipes based on the recognized emotional state.

[0446] This invention is a system that efficiently manages the expiration dates of ingredients purchased by a user and suggests optimal recipes based on the emotional state of the user. Specific embodiments for carrying out this invention are described below.

[0447] 1. Register ingredients

[0448] Program Generation

[0449] We provide a smartphone application that allows users to scan the barcode of ingredients to register information about the ingredients they have purchased (such as name, purchase date, expiration date, etc.). When a user scans the barcode, the corresponding ingredient information is automatically entered into the application.

[0450] Processing Description

[0451] In this step, the smartphone uses the barcode scanning function to obtain information about ingredients and sends the entered data to the server, which then stores the data in a database. The specific hardware used is the smartphone, and the software is a barcode scanning module.

[0452] 2. Best before date management and notification

[0453] Program Generation

[0454] A script runs on the server every day, checking the food ingredient information in the database and listing ingredients that are close to their expiration date. Notification data is generated for the listed ingredients and sent to the user via email or push notification.

[0455] Processing Description

[0456] In this step, the server checks expiration dates using Python scripts or scheduled scripts. For ingredients with an approaching expiration date, it generates notification data and sends it to smartphones or other devices. The specific software used is a scripting language (e.g., Python) and a notification service (e.g., Firebase Cloud Messaging).

[0457] 3. Recipe suggestions

[0458] Program Generation

[0459] The server searches for available recipes based on the ingredient information stored in the database, giving priority to recipes that contain ingredients with an approaching expiration date, and sends the results to the user's device to suggest recipes.

[0460] Processing Description

[0461] The server searches for recipe data based on the user's ingredient data. The recipe results are sent to the user's device, where the recipe details are displayed. The server executes a database query to search for recipes and returns the results in JSON format.

[0462] 4. Emotion Recognition and Recipe Optimization

[0463] Program Generation

[0464] We provide an emotion recognition engine that uses facial recognition and voice analysis via a camera to recognize the user's emotional state, and the emotion engine optimizes the recipe suggestions based on the recognized emotional state.

[0465] Processing Description

[0466] The system analyzes the user's emotional state using the camera and microphone on the smartphone or head-mounted display (HMD). The analyzed emotional data is sent to the server, which then provides recipes based on the user's emotional state. The specific software used is an emotion recognition library (e.g., OpenCV, Affectiva).

[0467] 5. Specific Examples

[0468] For example, if a user scans "Milk," the data is automatically entered into the app and saved in a database. When the expiration date approaches, a notification stating "Milk expiration date is approaching" is sent to the smartphone. If the emotion recognition engine detects that the user is "tired," the server will suggest "easy-to-make tamagoyaki (rolled omelet)."

[0469] 6. Examples of prompts

[0470] "I've entered the ingredients I purchased (milk, eggs, cabbage) and their expiration dates. I'm feeling tired right now and would like to know what easy meals I can make. Please tell me the best recipe based on this data and my mood."

[0471] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0472] Step 1:

[0473] Entering ingredient information

[0474] The user inputs the ingredients they purchased using the barcode scanning function of their smartphone. The smartphone reads the barcode and obtains information such as the ingredient name, purchase date, and expiration date. The barcode data is read as input, and the ingredient information is automatically entered as output.

[0475] Step 2:

[0476] Sending ingredient information

[0477] The smartphone sends the entered ingredient information to the server via an HTTP POST request. The smartphone receives ingredient information as input and sends the information as output to the server in JSON format.

[0478] Step 3:

[0479] Saving food information

[0480] The server stores the received ingredient information in a database. It analyzes the ingredient information received as input and registers the data using SQL queries etc. to store it in the database. As output, records stored in the database are generated.

[0481] Step 4:

[0482] Regular check of expiration dates

[0483] The server periodically checks the ingredient information in the database using a script and lists ingredients with an approaching expiration date. It compares the current date with the ingredient information in the database as input and extracts ingredients with an upcoming expiration date. The output is a list of ingredients with an upcoming expiration date.

[0484] Step 5:

[0485] Generate notification data

[0486] The server generates notification data about ingredients with upcoming expiration dates. It receives as input a list of ingredients with upcoming expiration dates and generates a notification message to be sent to the user. The notification message is generated as output.

[0487] Step 6:

[0488] Sending notification data

[0489] The server sends the generated notification data to the smartphone. It takes the generated notification message as input and sends it to the smartphone using a notification protocol (e.g., Firebase Cloud Messaging). It generates a notification to be sent to the smartphone as output.

[0490] Step 7:

[0491] Ingredient-based recipe search

[0492] The server searches for recipes based on ingredient information in the database, with priority given to recipes containing ingredients with an approaching expiration date. It receives ingredient information as input and searches the database for matching recipes. A list of recipes is generated as output.

[0493] Step 8:

[0494] Sending recipe information

[0495] The server sends the searched recipe information to the smartphone. It receives a list of recipes as input and transmits the data using a protocol for sending to the smartphone app. The recipe data that is sent to the smartphone is generated as output.

[0496] Step 9:

[0497] Viewing Recipes

[0498] The smartphone displays the received recipe information to the user. Using the received recipe data as input, the recipe information is displayed on the app's UI so that it can be seen by the user. As output, a recipe screen is generated that the user can view.

[0499] Step 10:

[0500] emotion recognition

[0501] Smartphones and head-mounted displays use cameras and microphones to recognize the user's emotional state. Camera images and audio data are analyzed as input, and the user's emotional state is analyzed. Recognized emotional data is generated as output.

[0502] Step 11:

[0503] Emotion-based recipe optimization

[0504] The server optimizes the recipe suggestions based on the recognized emotional state. It receives emotional data as input and selects recipes that match that state. It generates an optimized recipe list as output.

[0505] Step 12:

[0506] Displaying recipes based on emotions

[0507] The smartphone displays the optimized recipe information to the user. It receives the optimized recipe data as input and displays the recipe information visible to the user on the app's UI. As output, an optimized recipe screen is generated that the user can view.

[0508] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0509] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0510] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0511] [Second embodiment]

[0512] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0513] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0514] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0515] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0516] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0517] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0518] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0519] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0520] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0521] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0522] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0523] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0524] This invention is a system that efficiently manages the expiration dates of ingredients purchased by users and suggests recipes to make effective use of remaining ingredients. This system is realized by users inputting ingredient information and the server managing and processing that data.

[0525] To implement this system, the following main components are required:

[0526] 1. Register ingredients

[0527] Device:

[0528] It provides an interface for users to input information about purchased ingredients (such as name, purchase date, expiration date, etc.). For example, a smartphone app or web application can be used.

[0529] As a specific example, a user purchases "milk" and enters information such as the name "milk," the purchase date "October 5, 2023," and the expiration date "October 20, 2023" into the app.

[0530] 2. Sending and saving food information

[0531] Device:

[0532] It has a function to send the input ingredient information to the server using a protocol such as HTTP POST request.

[0533] Example: The information "milk" entered by the user is sent to the server.

[0534] server:

[0535] The received ingredient information is saved in a database. The server registers this information in the database in JSON format, SQL format, or other formats.

[0536] Example: The server stores the information about "milk" in a database.

[0537] 3. Best before date management and notification

[0538] server:

[0539] Regularly check the ingredient information in the database and list ingredients that are close to their expiration date. For example, run a script at a fixed time every day to list ingredients that will expire in the next three days.

[0540] Example: The server detects that the expiration date of "Milk" (October 20, 2023) is approaching.

[0541] server:

[0542] Notification data about food items nearing their expiration date is generated and sent to the user's device via email or push notification.

[0543] Example: The server generates a notification saying "The milk's expiration date is approaching" and sends it to the device.

[0544] Device:

[0545] Display the received notification data to the user, such as via a popup or notification bar.

[0546] Example: The device notifies the user that "The milk is nearing its expiration date."

[0547] 4. Recipe suggestions

[0548] server:

[0549] The system searches for available recipes based on ingredient information stored in the database, with priority given to recipes containing ingredients with an approaching expiration date.

[0550] Example: The server finds the recipe "Cream Stew" that uses "Milk".

[0551] Device:

[0552] The recipe information obtained from the server is displayed to the user, and multiple recipes are suggested so that the user can select the recipe that interests them.

[0553] Example: The terminal displays "Cream Stew Recipe" and the user can select it to view details.

[0554] summary

[0555] The system of this invention aims to streamline food management for users and reduce waste. Each processing step works in tandem, allowing users to easily understand the expiration date of ingredients and receive suggestions for recipes using remaining ingredients. This reduces food waste and improves the quality of their diet.

[0556] The processing flow will be explained below.

[0557] Step 1:

[0558] The user inputs the ingredient information.

[0559] The user uses the device to input information such as the name of the ingredient, purchase date, expiration date, etc. Alternatively, the barcode scanning function can be used to automatically input ingredient information if necessary.

[0560] Step 2:

[0561] The terminal transmits the input ingredient information.

[0562] The device converts the ingredient information entered by the user into JSON format and sends it as an HTTP POST request to the server's API endpoint.

[0563] Step 3:

[0564] The server stores the ingredient information.

[0565] The server parses the received request, extracts the ingredient information, and saves it in the database. Once the saving is complete, it returns a successful save response to the device.

[0566] Step 4:

[0567] The server checks the expiration date.

[0568] The server runs a scheduled task every day at a fixed time, checking the expiration date of each ingredient stored in the database, comparing it with the current date, and listing ingredients that are close to expiry (e.g., within 3 days).

[0569] Step 5:

[0570] The server generates the notification data.

[0571] The server generates notification data about ingredients whose expiration date is approaching and prepares it for transmission to the terminal. The notification data includes the ingredient name and expiration date.

[0572] Step 6:

[0573] The device displays a notification to the user.

[0574] The device receives notification data from the server and displays it to the user as a pop-up or alert, allowing the user to check which ingredients are nearing their expiration date.

[0575] Step 7:

[0576] The server searches for available recipes.

[0577] When a user requests it, or the server periodically checks the database, it searches for recipes based on ingredients in stock from the recipe database. Recipes that include ingredients with an approaching expiration date are given priority.

[0578] Step 8:

[0579] The server transmits the recipe information to the terminal.

[0580] The server sends the searched recipe information to the device in JSON format, including the ingredients needed and cooking instructions.

[0581] Step 9:

[0582] The device displays recipe suggestions to the user.

[0583] The device displays the recipe information received from the server on the user interface, allowing the user to select a recipe of interest, check the details, and start cooking.

[0584] Example 1

[0585] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0586] Managing the expiration dates of ingredients is an important task in the home, and if not managed properly, food waste will occur. Furthermore, in order to efficiently utilize ingredients that are nearing their expiration date, appropriate recipe suggestions are required. However, manually managing expiration dates and finding recipes that make effective use of ingredients is time-consuming. There is a need to provide a smart system that solves these problems.

[0587] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0588] In this invention, the server includes an input means for a user to input ingredient information, a transmission means for transmitting the input ingredient information to the server, a storage means for saving the transmitted ingredient information in a database, a check means for periodically checking the expiration dates of the ingredients saved in the database, a generation means for generating notification data about ingredients with upcoming expiration dates, a notification means for sending the generated notification data to a terminal, a display means for displaying the notification data to the user, a search means for searching for recipes based on ingredients in the database, a recipe suggestion means for sending the searched recipes to the terminal, a check means including a script for periodically checking ingredient information in the database, and a search means for preferentially searching for recipes containing ingredients with upcoming expiration dates. This allows users to easily manage the expiration dates of ingredients and enables the system to suggest appropriate recipes for efficiently using ingredients with upcoming expiration dates.

[0589] "User" refers to a person who uses the system.

[0590] "Ingredient information" refers to detailed data about ingredients, such as the name of the ingredient, purchase date, and expiration date.

[0591] "Input means" refers to the interface through which the user inputs information about ingredients, such as a smartphone app or web application.

[0592] "Transmission means" refers to a function for transmitting the input information about ingredients to the server.

[0593] "Storage means" refers to a function for storing transmitted information about ingredients in a database.

[0594] "Checking means" refers to a function that periodically checks the expiration dates of ingredients stored in the database.

[0595] "Generation means" refers to a function that creates notification data about food ingredients that are close to their expiration date.

[0596] The "notification means" refers to a function for transmitting the generated notification data to the user's terminal.

[0597] The "display means" refers to a function for displaying received notification data to the user.

[0598] "Search means" refers to the function for searching for recipes based on ingredients in the database.

[0599] "Recipe suggestion means" refers to a function for sending searched recipes to the user's terminal and suggesting them.

[0600] "Script" refers to program code that runs on the server to periodically check ingredient information in the database.

[0601] A "push notification" is a notification sent directly from the server to the user's device, and can be received even if the user does not have the app open.

[0602] "Database" refers to the storage system where information about ingredients is stored.

[0603] MODE FOR CARRYING OUT THE INVENTION

[0604] This invention is a system that efficiently manages the expiration dates of ingredients purchased by users and suggests recipes for making effective use of remaining ingredients. This system is realized by users inputting ingredient information and a server managing and processing that data. Specific embodiments of this system are described below.

[0605] First, the user uses a smartphone app or web application to input information about ingredients (such as name, purchase date, expiration date, etc.) For example, if a user purchases "milk," they enter information such as "milk," "October 5, 2023," and "October 20, 2023" into the app's input form and press the "Submit" button.

[0606] The device has a means of sending the input ingredient information to the server. This is done using an HTTP POST request, and the information is sent to the server. The server has a means of saving the received ingredient information to a database, and registers the received information in the database as a new record. For example, information such as "Milk," "October 5, 2023," and "October 20, 2023" is saved in the database.

[0607] The server periodically checks the ingredient information in the database and has a means of checking to list ingredients that are close to their expiration date. To do this, a batch process is executed at a fixed time every day using a script such as Python. For example, a script is executed every day at 9:00 AM to check ingredients that have an expiration date of less than three days.

[0608] The server has a generating means for generating notification data regarding ingredients whose expiration date is approaching. The generated notification data is sent by a notification means that sends the data to the user's device using push notification or email. For example, notification data stating "Milk's expiration date is approaching" is generated and sent to the device. The user can check the received notification on the device, which is displayed in a notification bar or a pop-up window.

[0609] Next, the server has a search means for searching available recipes based on the ingredient information stored in the database. It gives priority to searching for recipes that contain ingredients with an approaching expiration date. For example, it is possible to find a recipe called "cream stew" that uses "milk." The searched recipes are sent from the server to the user's terminal and are then sent via the recipe suggestion means. The user can check multiple suggested recipes on the terminal, select the recipe of interest, and view the details.

[0610] As a concrete example, the following shows the process from when a user purchases "milk" and enters and saves the information in the app. The user enters "milk," "October 5, 2023," and "October 20, 2023" into the app's input form and presses the "Submit" button. The app sends an HTTP POST request, and the server receives this request and stores the information in a database. The server runs a script every day at 9:00 AM to check ingredients with a best-by date of within three days, generates a notification stating "The milk's expiration date is approaching," and sends it as a push notification to the user's device. The user receives the notification and checks it on their device. The server searches for recipes using "milk," finds dishes such as "cream stew," and suggests them to the user. The user can then view the suggested recipes in the app and check their details.

[0611] An example prompt is, "I want to develop an app that allows you to enter the name, purchase date, and expiration date of purchased ingredients, manages that information, and notifies you when the expiration date is approaching. This app will provide recommended recipes based on the saved data. Please tell me the specific steps and data flow model."

[0612] In this way, the entire system works together, allowing users to easily manage the expiration dates of ingredients and efficiently use ingredients that are close to their expiration date when cooking, thereby reducing food waste and improving the quality of their diet.

[0613] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0614] Step 1:

[0615] The user inputs information about the ingredients.

[0616] Specific operation: The user opens a smartphone app or web application, enters the name of the purchased food, the purchase date, and the expiration date. For example, enter "Milk," "October 5, 2023," and "October 20, 2023," and presses the "Submit" button.

[0617] Input: Name of ingredient, purchase date, and expiration date.

[0618] Output: The input ingredient information is stored on the terminal.

[0619] Step 2:

[0620] The terminal transmits the input ingredient information to the server.

[0621] Specific operation: The input information is converted into JSON format and sent to the server using an HTTP POST request.

[0622] Input: Ingredient information stored on the device (JSON format).

[0623] Output: Ingredient information sent to the server.

[0624] Step 3:

[0625] The server stores the received ingredient information in a database.

[0626] What happens: The server parses the received JSON data, extracts the ingredient name, purchase date, and expiration date, and stores this information as a new record in the database.

[0627] Input: JSON formatted ingredient information sent to the server.

[0628] Output: A new record of ingredient information stored in the database.

[0629] Step 4:

[0630] The server periodically checks the food ingredient information in the database and lists ingredients that are close to their expiration date.

[0631] What it does: The server runs a Python script every day at 9 AM, which checks the expiration date field in the database and lists ingredients that have an expiration date within the next 3 days.

[0632] Input: Ingredient information stored in the database.

[0633] Output: A list of ingredients that are close to expiry.

[0634] Step 5:

[0635] The server generates notification data regarding food ingredients whose expiration date is approaching.

[0636] Specific behavior: Generate notification content based on the listed ingredients. For example, create a message saying "Milk is nearing its expiration date."

[0637] Input: A list of ingredients that are close to expiry.

[0638] Output: The generated notification data.

[0639] Step 6:

[0640] The server transmits the generated notification data to the terminal.

[0641] Specific operation: The generated notification data is sent to the user's device as a push notification or email.

[0642] Input: The generated notification data.

[0643] Output: Notification data sent to the user's device.

[0644] Step 7:

[0645] The terminal displays the received notification data to the user.

[0646] Specific behavior: The device receives the notification and displays it to the user in a pop-up or notification bar, for example, a notification saying "Your milk is nearing its expiration date."

[0647] Input: Notification data sent to the terminal.

[0648] Output: The notification data displayed to the user.

[0649] Step 8:

[0650] The server searches for available recipes based on the ingredient information stored in the database.

[0651] Specific behavior: To prioritize recipes that contain ingredients with an approaching expiration date, the name of the ingredient is used as a query to search for related recipes. For example, searching for recipes that use "milk" will find "cream stew."

[0652] Input: Ingredient information stored in the database.

[0653] Output: A list of searched recipes.

[0654] Step 9:

[0655] The server sends the searched recipe to the terminal.

[0656] Specific operation: The searched recipe information is sent to the user's terminal.

[0657] Input: A list of searched recipes.

[0658] Output: Recipe information sent to the user's device.

[0659] Step 10:

[0660] The terminal displays the recipe information to the user.

[0661] Specific operation: The device displays the received recipe information and allows the user to select a recipe they are interested in and view its details. For example, the device can display "cream stew recipe" and view its details.

[0662] Input: Recipe information sent to the device.

[0663] Output: The recipe information displayed to the user.

[0664] (Application example 1)

[0665] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0666] In modern society, busy users often find it difficult to keep track of the expiration dates of ingredients they purchase, resulting in food waste. Furthermore, they often lack the time to find appropriate recipes that make effective use of ingredients. To solve these problems, a system is needed that efficiently manages expiration dates and automatically suggests recipes that use ingredients with approaching expiration dates.

[0667] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0668] In this invention, the server includes an input means for a user to input ingredient information, a transmission means for transmitting the input ingredient information to the server, a storage means for saving the transmitted ingredient information in a database, a check means for periodically checking the expiration dates of the ingredients saved in the database, a generation means for generating notification data for ingredients whose expiration dates are approaching, a notification means for sending the generated notification data to a terminal, a display means for displaying the notification data to the user, a search means for searching for recipes based on ingredients in the database, a recipe suggestion means for sending the searched recipes to the terminal, a scanning means for scanning receipts to automatically recognize ingredient information, a calculation means for calculating the expiration date from the recognized ingredient information, and a notification function for sending expiration date notifications to the user's terminal. This allows users to efficiently manage ingredient expiration dates and receive suggested recipes that utilize ingredients without waste.

[0669] The "input means" is an interface that allows the user to input information about ingredients into the terminal.

[0670] The "transmission means" is a device or software function that transmits the input information about ingredients to the server.

[0671] "Storage means" refers to a device or software function for storing the transmitted information about ingredients in a database.

[0672] The "checking means" is a device or software function that periodically checks the expiration dates of ingredients stored in the database.

[0673] The "generation means" is a device or software function that generates notification data about food ingredients that are close to their expiration date.

[0674] The "notification means" is a device or software function that transmits the generated notification data to the user's terminal.

[0675] The "display means" is an interface for displaying notification data to the user.

[0676] A "search means" is a device or software function that searches for recipes based on ingredients in a database.

[0677] The "recipe suggestion means" is a device or software function that sends the searched recipes to the user's terminal.

[0678] "Scanning means" refers to a device or software function that scans receipts and automatically recognizes ingredient information.

[0679] The "calculation means" is a device or software function that calculates the expiration date from the information of the recognized ingredients.

[0680] A "notification function" is a device or software function that sends expiration date notifications to a user's terminal.

[0681] This invention provides a system for efficiently managing expiration dates of ingredients purchased by a user and proposing recipes for making effective use of remaining ingredients. This system includes the following means.

[0682] 1. Input Method

[0683] Users enter food information using a smartphone app or web application, either manually entering the name of the purchased food, the purchase date, and the expiration date, or automatically by scanning the receipt.

[0684] 2. Transmission Method

[0685] The ingredient information entered by the user is sent to the server using an HTTP POST request. Data is sent from the smartphone app or web application to the server.

[0686] 3. Preservation means

[0687] The server receives the transmitted ingredient information and stores it in a database, which records the ingredient name, purchase date, and expiration date.

[0688] 4. Checking Methods

[0689] The server periodically runs a script to check expiration dates in the database, for example, listing ingredients that are close to expiry at a set time each day.

[0690] 5. Generation means

[0691] The server generates notification data for ingredients that are nearing their expiration date, including the name of the ingredient, its expiration date, and a warning message.

[0692] 6. Means of notification

[0693] The generated notification data is sent to the user's device via email or push notification. SMS notifications can also be sent using the Twilio API.

[0694] 7. Display means

[0695] The user's device will display the received notification data in a pop-up or notification bar, informing the user that the expiration date of the food item is approaching.

[0696] 8. Search Methods

[0697] The server searches for available recipes based on the ingredient information stored in the database, with priority given to recipes that contain ingredients with an approaching expiration date.

[0698] 9. Recipe suggestion methods

[0699] The recipe information obtained from the server is sent to the user's device, where the user can select the recipe they are interested in and check the detailed information.

[0700] 10. Scanning Method

[0701] Use your smartphone camera and OCR (optical character recognition) software (e.g., pytesseract) to scan receipts and automatically recognize ingredient information.

[0702] 11. Means of calculation

[0703] Use data analysis algorithms to calculate expiration dates from recognized ingredient information. Calculate detailed information about ingredients that are nearing their expiration date.

[0704] 12. Notification function

[0705] The notification function uses the Twilio API, Firebase Cloud Messaging, etc. to send expiration date notifications to users' devices, allowing them to take immediate action.

[0706] Specific examples

[0707] Suppose a user purchases "milk" and scans the receipt with a smartphone app. The app uses OCR technology to extract information about the "milk," parses / enters the purchase date and expiration date, and sends it to the server. The server stores this information in a database and periodically checks the expiration date. When the expiration date approaches, the server sends a notification to the user's smartphone saying, "Your milk is about to expire." At the same time, a recipe using milk (e.g., cream stew) is suggested, and the user can refer to the recipe to cook.

[0708] Example prompts for generative AI models

[0709] To create a food management application, we manage the expiration dates of ingredients purchased by the user and automatically register the ingredient information through receipt scanning and OCR analysis. It also includes a function to send notifications to a smartphone when ingredients are approaching their expiration date. For example, if a receipt says "Milk, Purchase Date 2023-10-05, Expiration Date 2023-10-20," the relevant information is registered in the application, and a notification is sent when the expiration date approaches within three days.

[0710] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0711] Step 1:

[0712] The user inputs the ingredient information.

[0713] Users use a smartphone app to input the name, purchase date, and expiration date of the ingredients they purchased, or scan a receipt to automatically recognize the ingredient information. The input data is the ingredient name, purchase date, and expiration date.

[0714] Step 2:

[0715] The terminal transmits the input ingredient information to the server.

[0716] The smartphone app sends the entered ingredient data to the server using an HTTP POST request. The device sends the ingredient name, purchase date, and expiration date as parameters. The output is a communication success or failure status.

[0717] Step 3:

[0718] The server stores the received ingredient information in a database.

[0719] The server parses the received JSON format data and registers the ingredient name, purchase date, and expiration date in the database. The input data is the ingredient information, and the output is the result saved in the database.

[0720] Step 4:

[0721] The server periodically checks the expiration dates in the database.

[0722] Run the script periodically (for example, every day at a set time) to list ingredients that are close to expiry. The input data is all the ingredients in the database, and the output is a list of ingredients that are close to expiry.

[0723] Step 5:

[0724] The server generates notification data about food ingredients that are close to their expiration date.

[0725] Based on a list of ingredients whose expiration date is approaching, notification data for each ingredient is generated. The notification data includes the ingredient name, expiration date, and warning message. The input data is a list of ingredients whose expiration date is approaching, and the output is the generated notification data.

[0726] Step 6:

[0727] The server transmits the generated notification data to the terminal.

[0728] The server sends the notification data to the user's smartphone as an email or push notification. The input data is the notification data, and the output is the transmission result.

[0729] Step 7:

[0730] The terminal displays the received notification data.

[0731] The user's smartphone displays the notification data in a pop-up or notification bar to notify the user that the expiration date is approaching. The input data is the notification data, and the output is the displayed notification message.

[0732] Step 8:

[0733] The server searches for recipes based on the ingredient information in the database.

[0734] The system prioritizes the search for recipes containing ingredients with upcoming expiration dates and suggests them to the user. The input data is information about ingredients with upcoming expiration dates, and the output is a list of searched recipes.

[0735] Step 9:

[0736] The server transmits the searched recipe information to the terminal.

[0737] The server sends the searched recipe list to the user's smartphone, where the user can check the details. The input data is the recipe list, and the output is the transmission result and displayed recipe information.

[0738] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0739] This invention is a system that efficiently manages the expiration dates of ingredients purchased by the user, suggests recipes to make effective use of remaining ingredients, and recognizes the user's emotions to make appropriate suggestions.By combining an emotion engine with the ingredient management function, this system can make suggestions based on the user's emotional state.

[0740] To implement this system, the following main components are required:

[0741] 1. Register ingredients

[0742] Device:

[0743] It provides an interface for users to input information about ingredients they have purchased (such as name, purchase date, and expiration date). As a specific example, a user purchases "milk" and enters information such as the name "Milk," purchase date "October 5, 2023," and expiration date "October 20, 2023" into the app. It is also possible to input this information automatically using the barcode scanning function.

[0744] 2. Sending and saving food information

[0745] Device:

[0746] It has a function to send the inputted ingredient information to the server using a protocol such as HTTP POST request.

[0747] Example: The information "milk" entered by the user is sent to the server.

[0748] server:

[0749] The received ingredient information is saved in a database. The server registers this information in the database in JSON format, SQL format, or other formats.

[0750] Example: The server stores the information about "milk" in a database.

[0751] 3. Best before date management and notification

[0752] server:

[0753] Regularly check the ingredient information in the database and list ingredients that are close to their expiration date. For example, run a script at a fixed time every day to list ingredients that will expire in the next three days.

[0754] Example: The server detects that the expiration date of "Milk" (October 20, 2023) is approaching.

[0755] server:

[0756] Generate notification data about food items that are nearing their expiration date and prepare to notify users. This can be done via email or push notification.

[0757] Example: The server generates a notification saying "The milk's expiration date is approaching" and sends it to the device.

[0758] Device:

[0759] Display the received notification data to the user, such as via a popup or notification bar.

[0760] Example: The device notifies the user that "The milk is nearing its expiration date."

[0761] 4. Recipe suggestions

[0762] server:

[0763] The system searches for available recipes based on ingredient information stored in the database, with priority given to recipes containing ingredients with an approaching expiration date.

[0764] Example: The server finds the recipe "Cream Stew" that uses "Milk".

[0765] Device:

[0766] The recipe information obtained from the server is displayed to the user, and multiple recipes are suggested so that the user can select the recipe that interests them.

[0767] Example: The terminal displays "Cream Stew Recipe" and the user can select it to view details.

[0768] 5. Operation of the Emotion Engine

[0769] Device:

[0770] It provides a function for recognizing the user's emotions in real time, using facial recognition via a camera and analysis based on the user's input behavior.

[0771] server:

[0772] The system receives user emotional data and optimizes recipe suggestions and notification content. For example, if the user is tired, it will suggest easy-to-make dishes.

[0773] Example: If the user is recognized as "tired," the server will suggest "easy-to-make tamagoyaki (rolled omelet)."

[0774] Device:

[0775] The emotional engine displays optimized recipes and notifications to users, providing optimal suggestions tailored to the emotional state the user is facing.

[0776] Example: The device will display "Easy recipe recommended for when you're tired: Tamagoyaki."

[0777] summary

[0778] The system of this invention not only effectively manages expiration dates of ingredients, but also enhances user convenience and satisfaction by suggesting and notifying recipes that take the user's emotional state into consideration. Various elements work together to reduce food waste and improve the user's quality of life.

[0779] The processing flow will be explained below.

[0780] Step 1:

[0781] The user inputs the ingredient information.

[0782] The user enters information about the ingredients they purchased using an application on their device. The information to be entered includes the name of the ingredient, the purchase date, and the expiration date. This information can also be entered automatically using the barcode scanning function. As a specific example, a user purchases "milk" and enters the name "milk," the purchase date "October 5, 2023," and the expiration date "October 20, 2023."

[0783] Step 2:

[0784] The terminal transmits the ingredient information to the server.

[0785] The device converts the entered ingredient information into JSON format and sends it as an HTTP POST request to the server's API endpoint. This request includes the ingredient name, purchase date, and expiration date. For example, the information for "milk" is sent to the server.

[0786] Step 3:

[0787] The server stores the ingredient information in a database.

[0788] The server parses the received request, extracts the ingredient information, and saves it in the database. The server returns a save completion response to the terminal. For example, the server saves the information for "milk" in the database.

[0789] Step 4:

[0790] The server checks the expiration date.

[0791] The server runs a scheduled task every day at a fixed time, checking the expiration date of each ingredient in the database. It compares this to the current date and lists ingredients that are close to expiring (for example, within 3 days). For example, the server detects that "Milk" is close to expiring.

[0792] Step 5:

[0793] The server generates the notification data.

[0794] The server generates notification data about ingredients whose expiration date is approaching and prepares to send it to the user. This notification data includes the ingredient name and expiration date. For example, a notification saying "Milk's expiration date is approaching" is generated.

[0795] Step 6:

[0796] The device displays the notification to the user.

[0797] The device receives notification data from the server and displays it in a pop-up or notification bar. The user can check which ingredients are nearing their expiration date. For example, the device may notify the user that "The expiration date of your milk is approaching."

[0798] Step 7:

[0799] The server searches for available recipes.

[0800] The server searches for available recipes based on the ingredient information stored in the database. It prioritizes recipes that contain ingredients with an approaching expiration date. For example, the server finds a recipe called "cream stew" that uses "milk."

[0801] Step 8:

[0802] The server transmits the recipe information to the terminal.

[0803] The server sends the searched recipe information to the device in JSON format. The recipe information includes the necessary ingredients and cooking instructions. For example, a "cream stew recipe" is sent to the device.

[0804] Step 9:

[0805] The device displays recipe suggestions to the user.

[0806] The device displays the recipe information received from the server and allows the user to select the recipe they are interested in. For example, the device displays "cream stew recipe" and the user can select it and view the details.

[0807] Step 10:

[0808] The device recognizes the user's emotions.

[0809] The device's camera and sensors are used to analyze the user's facial expressions and voice to obtain emotional data, detecting whether the user is "tired" or "stressed."

[0810] Step 11:

[0811] The server optimizes the recipe based on emotional data.

[0812] The server receives the user's emotional data and adjusts the recipe suggestions and notification content based on that data. For example, if the user is tired, it will prioritize suggestions of easy-to-make dishes. For example, if the user is tired, it will suggest "easy-to-make tamagoyaki" (rolled omelet).

[0813] Step 12:

[0814] The device displays recipes and notifications optimized by the emotion engine.

[0815] The device uses the emotion engine sent from the server to display optimized recipe information and notifications to the user. For example, it displays information such as "Easy recipe recommended for when you're tired: Tamagoyaki."

[0816] Through these steps, this system can provide users with a more personalized experience by combining food expiration date management and emotion recognition.

[0817] Example 2

[0818] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0819] Many conventional food management systems are specialized in managing and notifying expiration dates, and have the problem of not being able to make suggestions that take the user's emotions into consideration. As a result, the recipes provided may not be suitable for the user's situation, which can result in a decrease in user convenience and satisfaction. Another problem is that ingredients are often entered manually, which is a time-consuming and labor-intensive process.

[0820] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0821] In this invention, the server includes input means for a user to input food information, transmission means for transmitting the input food information to the server, storage means for saving the transmitted food information in a database, check means for periodically checking the expiration dates of the foods saved in the database, generation means for generating notification data for foods with upcoming expiration dates, notification means for sending the generated notification data to a terminal, display means for displaying the notification data to the user, search means for searching for recipes based on foods in the database, recipe suggestion means for sending the searched recipes to the terminal, emotion recognition means for recognizing the user's emotion, emotion data transmission means for sending emotion data recognized by the emotion recognition means to the server, and optimization means for optimizing recipe suggestions based on the emotion data. This not only enables food expiration date management but also makes it possible to suggest optimal recipes tailored to the user's emotions.

[0822] "Input means" refers to a device or interface that allows a user to input information about a food product.

[0823] "Transmission means" refers to a device or protocol that transmits the input food information to the server.

[0824] "Storage means" means a device or process that stores the submitted food information in a database.

[0825] A "checking means" is a device or script that periodically checks the expiration dates of food items stored in the database.

[0826] A "generator" is a device or algorithm that generates notification data about food products that are nearing their expiration date.

[0827] The "notification means" is a device or protocol that transmits the generated notification data to the terminal.

[0828] A "display means" is a device or interface that displays notification data to a user.

[0829] A "search tool" is a device or algorithm that searches for recipes based on foods in the database.

[0830] A "recipe suggestion means" is a device or protocol that sends searched recipes to a terminal.

[0831] An "emotion recognition means" is a device or algorithm that recognizes a user's emotions.

[0832] The "emotion data transmission means" is a device or protocol that transmits the emotion data recognized by the emotion recognition means to the server.

[0833] An "optimization means" is a device or algorithm that optimizes recipe suggestions based on emotion data.

[0834] This invention is a system that efficiently manages the expiration dates of food purchased by a user, suggests recipes for making effective use of remaining food, and recognizes the user's emotions to make appropriate suggestions. Specific embodiments of this invention are described below.

[0835] Hardware and Software Configuration

[0836] Input method: Provides an interface for users to input food information. Specifically, this corresponds to an application installed on a device such as a smartphone or tablet. Users can manually enter the food name, purchase date, and expiration date, or use a barcode scanning function to automatically enter them.

[0837] Transmission method: HTTP POST request is used as the protocol to send the entered food information to the server. For example, when a user enters information about "milk," this information is sent to the server in JSON format.

[0838] Storage method: The server stores the received food information in a database using SQL Server or NoSQL database, and the food information is stored in an appropriate format.

[0839] Check method: The server periodically runs a script to check the food information in the database. This process can be automated using a cron job or similar, and lists foods that have a shelf life of less than three days.

[0840] Generation method: Generates notification data for foods in the database that are close to their expiration date. The generated notification data is prepared in a format that notifies the user via email, push notification, etc.

[0841] Notification method: The generated notification data is sent to the device. This is done using a mail server or push notification service. The notification data is displayed in a pop-up or notification bar so that the user can check it immediately.

[0842] Display method: The device displays the notification data received from the server to the user. Specifically, the device uses the smartphone's notification bar or push notification to allow the user to quickly check the information.

[0843] Search method: The server searches for available recipes based on the food information stored in the database, with priority given to searching and suggesting recipes that include food items with an approaching expiration date.

[0844] Recipe suggestion method: The server sends the searched recipes to the terminal and suggests multiple recipes, allowing the user to select the recipe they are interested in and check the details.

[0845] Emotion recognition: The device provides a function to recognize the user's emotions in real time. Emotion recognition can be achieved by facial recognition via a camera or by analyzing the user's input behavior.

[0846] Emotion data transmission method: The recognized emotion data is transmitted from the device to the server. Based on this data, the server optimizes the next recipe suggestion.

[0847] Optimization method: The server receives the emotional data and suggests recipes according to the user's emotional state. For example, if the server detects that the user is "tired," it will prioritize suggestions for easy-to-make dishes.

[0848] Specific examples and prompts for the generative AI model

[0849] Examples:

[0850] It explains how the system notifies users when the milk they have purchased is nearing its expiration date and suggests suitable recipes.

[0851] Specific operation example: A user purchases "milk" and enters the name "milk," the purchase date "October 5, 2023," and the expiration date "October 20, 2023" into the device. The entered information is sent to the server and stored in a database. The server periodically checks the database and lists foods with an expiration date of three days or less. It then generates notification data stating "The milk's expiration date is approaching" and sends it to the user's device. When the user receives the notification, the server provides a recipe for "cream stew."

[0852] Example prompt for a generative AI model:

[0853] Explain how the system will notify the user when the "Milk" they have purchased is nearing its expiration date and suggest suitable recipes.

[0854] This invention not only enables the expiration date of food products to be managed, but also makes it possible to propose optimal recipes that take into account the user's feelings, thereby improving user convenience and satisfaction.

[0855] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0856] Specific processing steps of the system program

[0857] Step 1: User enters food information

[0858] The user inputs information about the food they have purchased into the terminal, such as the name of the food, the date of purchase, the expiration date, etc. This input is performed using a keyboard or a barcode scanner.

[0859] Input: food name, purchase date, expiration date

[0860] Output: Input data on the terminal

[0861] What it does: A user opens the app, enters "Milk," sets the purchase date to "2023-10-05," and the expiration date to "2023-10-20." By using the barcode scanner, this information is automatically entered.

[0862] Step 2: Device sends data to server

[0863] The terminal sends the food information entered by the user to the server. The data is sent via an HTTP POST request in JSON format.

[0864] Input: Food information entered by the user into the device

[0865] Output: JSON data sent to the server

[0866] Specific operation: The device sends JSON data (e.g., "{\"name\": \"Milk\", \"purchase_date\": \"2023-10-05\", \"expiry_date\": \"2023-10-20\"}") to the server using "POST / api / food_register".

[0867] Step 3: Data received and stored by the server

[0868] The server receives the food information sent from the device and stores it in a database using SQL queries.

[0869] Input: JSON data sent from the terminal

[0870] Output: Food information stored in the database

[0871] Specific operation: The server executes the SQL query "INSERT INTO food_table (name, purchase_date, expiration_date) VALUES (\'Milk\', \'2023-10-05\', \'2023-10-20\')" and saves it in the database.

[0872] Step 4: Expiry date check by server

[0873] The server runs a script that periodically checks the food information in the database. This script is run by a cron job at midnight every day and lists foods that have an expiration date within the next three days.

[0874] Input: Food information stored in a database

[0875] Output: A list of foods that are close to expiry date

[0876] Specific operation: The server executes the SQL query "SELECT FROM food_table WHERE best-before_date <= DATE_ADD(CURDATE(), INTERVAL 3 DAY)".

[0877] Step 5: Notification data generation by the server

[0878] The server generates notification data about food products that are nearing their expiration date, and the generated data is ready to send notifications to users.

[0879] Input: List of foods that are close to expiry date

[0880] Output: Generated notification data

[0881] Specific behavior: The server generates a notification message saying "The milk is nearing its expiration date."

[0882] Step 6: Send notification data to the device

[0883] The server then sends the generated notification data to the user's device, which can take the form of email or push notification.

[0884] Input: Notification data

[0885] Output: Notification sent to the device

[0886] Specific operation: The server sends the notification data via a push notification service or mail server.

[0887] Step 7: Displaying notifications on your device

[0888] The device displays the received notification data to the user in a pop-up or notification bar format.

[0889] Input: Notification data sent from the server

[0890] Output: The notification displayed to the user

[0891] Specific behavior: The device will display a pop-up notification saying "The milk's expiration date is approaching."

[0892] Step 8: Server Lookup for Recipes

[0893] The server searches for available recipes based on the food information stored in the database, with priority given to recipes that include food items with an approaching expiration date.

[0894] Input: Food information in the database

[0895] Output: Searched recipe list

[0896] Specific behavior: The server executes the SQL query "SELECT FROM recipe WHERE ingredients LIKE \"%milk%\"".

[0897] Step 9: Send the recipe to your device

[0898] The server sends the searched recipes to the terminal, and multiple recipes are suggested to the user.

[0899] Input: Searched recipe list

[0900] Output: Recipe information sent to the device

[0901] Specific operation: The server sends recipe information to the terminal and displays it to the user.

[0902] Step 10: Emotion recognition on the device

[0903] The device will provide features for recognizing users' emotions in real time, including camera-based facial recognition and behavioral analysis.

[0904] Input: User's facial expressions and behavior data

[0905] Output: Recognized emotion data

[0906] Specific operation: The device analyzes the user's face captured by the camera and recognizes emotions such as "tired."

[0907] Step 11: Send emotion data to the server

[0908] The device sends the recognized emotion data to the server, which then uses it to suggest the next recipe.

[0909] Input: Recognized emotion data

[0910] Output: Emotion data sent to the server

[0911] Specific operation: The device sends emotion data in JSON format to the server using "POST / api / emotion_data".

[0912] Step 12: Optimizing recipe suggestions by the server

[0913] The server optimizes recipe suggestions based on emotion data, and selects recipes that suit the user's state.

[0914] Input: Emotion data, recipe information in the database

[0915] Output: Optimized recipe suggestions

[0916] Specific operation: For users who are recognized by the server as being "tired," the server will prioritize suggesting easy-to-make recipes for "tamagoyaki" (rolled omelet).

[0917] (Application example 2)

[0918] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0919] In modern consumer lifestyles, it is difficult for users to efficiently manage the expiration dates of purchased ingredients, resulting in food waste. It is also not easy for users to select the optimal recipe based on their emotional state at any given time. At the same time, there is a demand for systems that can suggest appropriate recipes based on emotions, but no such systems have existed to date. This situation has led to problems that reduce user convenience and satisfaction.

[0920] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0921] In this invention, the server includes emotion recognition means for recognizing the emotional state of the user, recipe optimization means for optimizing recipes based on the recognized emotional state, and generation means for generating notification data for ingredients whose expiration dates are approaching, thereby enabling the server to efficiently manage the expiration dates of ingredients purchased by the user and propose optimal recipes according to the user's emotional state.

[0922] The "input means" is an interface through which the user inputs information about ingredients.

[0923] The "transmission means" is a function that transmits the input information about ingredients to the server.

[0924] The "storage means" is a function that stores the transmitted information about ingredients in a database.

[0925] The "checking means" is a function that periodically checks the expiration dates of ingredients stored in the database.

[0926] The "generation means" is a function that generates notification data about food ingredients that are close to their expiration date.

[0927] The "notification means" is a function that transmits the generated notification data to the terminal.

[0928] The "display means" is a function that displays the notification data to the user.

[0929] "Search means" is a function that searches for recipes based on ingredients in the database.

[0930] The "recipe suggestion means" is a function that sends the searched recipe to the terminal.

[0931] The "emotion recognition means" is a function for recognizing the user's emotional state.

[0932] The "recipe optimizer" is a function that optimizes recipes based on the recognized emotional state.

[0933] This invention is a system that efficiently manages the expiration dates of ingredients purchased by a user and suggests optimal recipes based on the emotional state of the user. Specific embodiments for carrying out this invention are described below.

[0934] 1. Register ingredients

[0935] Program Generation

[0936] We provide a smartphone application that allows users to scan the barcode of ingredients to register information about the ingredients they have purchased (such as name, purchase date, expiration date, etc.). When a user scans the barcode, the corresponding ingredient information is automatically entered into the application.

[0937] Processing Description

[0938] In this step, the smartphone uses the barcode scanning function to obtain information about ingredients and sends the entered data to the server, which then stores the data in a database. The specific hardware used is the smartphone, and the software is a barcode scanning module.

[0939] 2. Best before date management and notification

[0940] Program Generation

[0941] A script runs on the server every day, checking the food ingredient information in the database and listing ingredients that are close to their expiration date. Notification data is generated for the listed ingredients and sent to the user via email or push notification.

[0942] Processing Description

[0943] In this step, the server checks expiration dates using Python scripts or scheduled scripts. For ingredients with an approaching expiration date, it generates notification data and sends it to smartphones or other devices. The specific software used is a scripting language (e.g., Python) and a notification service (e.g., Firebase Cloud Messaging).

[0944] 3. Recipe suggestions

[0945] Program Generation

[0946] The server searches for available recipes based on the ingredient information stored in the database, giving priority to recipes that contain ingredients with an approaching expiration date, and sends the results to the user's device to suggest recipes.

[0947] Processing Description

[0948] The server searches for recipe data based on the user's ingredient data. The recipe results are sent to the user's device, where the recipe details are displayed. The server executes a database query to search for recipes and returns the results in JSON format.

[0949] 4. Emotion Recognition and Recipe Optimization

[0950] Program Generation

[0951] We provide an emotion recognition engine that uses facial recognition and voice analysis via a camera to recognize the user's emotional state, and the emotion engine optimizes the recipe suggestions based on the recognized emotional state.

[0952] Processing Description

[0953] The system analyzes the user's emotional state using the camera and microphone on the smartphone or head-mounted display (HMD). The analyzed emotional data is sent to the server, which then provides recipes based on the user's emotional state. The specific software used is an emotion recognition library (e.g., OpenCV, Affectiva).

[0954] 5. Specific Examples

[0955] For example, if a user scans "Milk," the data is automatically entered into the app and saved in a database. When the expiration date approaches, a notification stating "Milk expiration date is approaching" is sent to the smartphone. If the emotion recognition engine detects that the user is "tired," the server will suggest "easy-to-make tamagoyaki (rolled omelet)."

[0956] 6. Examples of prompts

[0957] "I've entered the ingredients I purchased (milk, eggs, cabbage) and their expiration dates. I'm feeling tired right now and would like to know what easy meals I can make. Please tell me the best recipe based on this data and my mood."

[0958] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0959] Step 1:

[0960] Entering ingredient information

[0961] The user inputs the ingredients they purchased using the barcode scanning function of their smartphone. The smartphone reads the barcode and obtains information such as the ingredient name, purchase date, and expiration date. The barcode data is read as input, and the ingredient information is automatically entered as output.

[0962] Step 2:

[0963] Sending ingredient information

[0964] The smartphone sends the entered ingredient information to the server via an HTTP POST request. The smartphone receives ingredient information as input and sends the information as output to the server in JSON format.

[0965] Step 3:

[0966] Saving food information

[0967] The server stores the received ingredient information in a database. It analyzes the ingredient information received as input and registers the data using SQL queries etc. to store it in the database. As output, records stored in the database are generated.

[0968] Step 4:

[0969] Regular check of expiration dates

[0970] The server periodically checks the ingredient information in the database using a script and lists ingredients with an approaching expiration date. It compares the current date with the ingredient information in the database as input and extracts ingredients with an upcoming expiration date. The output is a list of ingredients with an upcoming expiration date.

[0971] Step 5:

[0972] Generate notification data

[0973] The server generates notification data about ingredients with upcoming expiration dates. It receives as input a list of ingredients with upcoming expiration dates and generates a notification message to be sent to the user. The notification message is generated as output.

[0974] Step 6:

[0975] Sending notification data

[0976] The server sends the generated notification data to the smartphone. It takes the generated notification message as input and sends it to the smartphone using a notification protocol (e.g., Firebase Cloud Messaging). It generates a notification to be sent to the smartphone as output.

[0977] Step 7:

[0978] Ingredient-based recipe search

[0979] The server searches for recipes based on ingredient information in the database, with priority given to recipes containing ingredients with an approaching expiration date. It receives ingredient information as input and searches the database for matching recipes. A list of recipes is generated as output.

[0980] Step 8:

[0981] Sending recipe information

[0982] The server sends the searched recipe information to the smartphone. It receives a list of recipes as input and transmits the data using a protocol for sending to the smartphone app. The recipe data that is sent to the smartphone is generated as output.

[0983] Step 9:

[0984] Viewing Recipes

[0985] The smartphone displays the received recipe information to the user. Using the received recipe data as input, the recipe information is displayed on the app's UI so that it can be seen by the user. As output, a recipe screen is generated that the user can view.

[0986] Step 10:

[0987] emotion recognition

[0988] Smartphones and head-mounted displays use cameras and microphones to recognize the user's emotional state. Camera images and audio data are analyzed as input, and the user's emotional state is analyzed. Recognized emotional data is generated as output.

[0989] Step 11:

[0990] Emotion-based recipe optimization

[0991] The server optimizes the recipe suggestions based on the recognized emotional state. It receives emotional data as input and selects recipes that match that state. It generates an optimized recipe list as output.

[0992] Step 12:

[0993] Displaying recipes based on emotions

[0994] The smartphone displays the optimized recipe information to the user. It receives the optimized recipe data as input and displays the recipe information visible to the user on the app's UI. As output, an optimized recipe screen is generated that the user can view.

[0995] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0996] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0997] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0998] [Third embodiment]

[0999] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1000] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1001] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1002] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1003] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1004] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1005] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1006] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1007] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1008] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1009] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1010] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1011] This invention is a system that efficiently manages the expiration dates of ingredients purchased by users and suggests recipes to make effective use of remaining ingredients. This system is realized by users inputting ingredient information and the server managing and processing that data.

[1012] To implement this system, the following main components are required:

[1013] 1. Register ingredients

[1014] Device:

[1015] It provides an interface for users to input information about purchased ingredients (such as name, purchase date, expiration date, etc.). For example, a smartphone app or web application can be used.

[1016] As a specific example, a user purchases "milk" and enters information such as the name "milk," the purchase date "October 5, 2023," and the expiration date "October 20, 2023" into the app.

[1017] 2. Sending and saving food information

[1018] Device:

[1019] It has a function to send the input ingredient information to the server using a protocol such as HTTP POST request.

[1020] Example: The information "milk" entered by the user is sent to the server.

[1021] server:

[1022] The received ingredient information is saved in a database. The server registers this information in the database in JSON format, SQL format, or other formats.

[1023] Example: The server stores the information about "milk" in a database.

[1024] 3. Best before date management and notification

[1025] server:

[1026] Regularly check the ingredient information in the database and list ingredients that are close to their expiration date. For example, run a script at a fixed time every day to list ingredients that will expire in the next three days.

[1027] Example: The server detects that the expiration date of "Milk" (October 20, 2023) is approaching.

[1028] server:

[1029] Notification data about food items nearing their expiration date is generated and sent to the user's device via email or push notification.

[1030] Example: The server generates a notification saying "The milk's expiration date is approaching" and sends it to the device.

[1031] Device:

[1032] Display the received notification data to the user, such as via a popup or notification bar.

[1033] Example: The device notifies the user that "The milk is nearing its expiration date."

[1034] 4. Recipe suggestions

[1035] server:

[1036] The system searches for available recipes based on ingredient information stored in the database, with priority given to recipes containing ingredients with an approaching expiration date.

[1037] Example: The server finds the recipe "Cream Stew" that uses "Milk".

[1038] Device:

[1039] The recipe information obtained from the server is displayed to the user, and multiple recipes are suggested so that the user can select the recipe that interests them.

[1040] Example: The terminal displays "Cream Stew Recipe" and the user can select it to view details.

[1041] summary

[1042] The system of this invention aims to streamline food management for users and reduce waste. Each processing step works in tandem, allowing users to easily understand the expiration date of ingredients and receive suggestions for recipes using remaining ingredients. This reduces food waste and improves the quality of their diet.

[1043] The processing flow will be explained below.

[1044] Step 1:

[1045] The user inputs the ingredient information.

[1046] The user uses the device to input information such as the name of the ingredient, purchase date, expiration date, etc. Alternatively, the barcode scanning function can be used to automatically input ingredient information if necessary.

[1047] Step 2:

[1048] The terminal transmits the input ingredient information.

[1049] The device converts the ingredient information entered by the user into JSON format and sends it as an HTTP POST request to the server's API endpoint.

[1050] Step 3:

[1051] The server stores the ingredient information.

[1052] The server parses the received request, extracts the ingredient information, and saves it in the database. Once the saving is complete, it returns a successful save response to the device.

[1053] Step 4:

[1054] The server checks the expiration date.

[1055] The server runs a scheduled task every day at a fixed time, checking the expiration date of each ingredient stored in the database, comparing it with the current date, and listing ingredients that are close to expiry (e.g., within 3 days).

[1056] Step 5:

[1057] The server generates the notification data.

[1058] The server generates notification data about ingredients whose expiration date is approaching and prepares it for transmission to the terminal. The notification data includes the ingredient name and expiration date.

[1059] Step 6:

[1060] The device displays a notification to the user.

[1061] The device receives notification data from the server and displays it to the user as a pop-up or alert, allowing the user to check which ingredients are nearing their expiration date.

[1062] Step 7:

[1063] The server searches for available recipes.

[1064] When a user requests it, or the server periodically checks the database, it searches for recipes based on ingredients in stock from the recipe database. Recipes that include ingredients with an approaching expiration date are given priority.

[1065] Step 8:

[1066] The server transmits the recipe information to the terminal.

[1067] The server sends the searched recipe information to the device in JSON format, including the ingredients needed and cooking instructions.

[1068] Step 9:

[1069] The device displays recipe suggestions to the user.

[1070] The device displays the recipe information received from the server on the user interface, allowing the user to select a recipe of interest, check the details, and start cooking.

[1071] Example 1

[1072] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1073] Managing the expiration dates of ingredients is an important task in the home, and if not managed properly, food waste will occur. Furthermore, in order to efficiently utilize ingredients that are nearing their expiration date, appropriate recipe suggestions are required. However, manually managing expiration dates and finding recipes that make effective use of ingredients is time-consuming. There is a need to provide a smart system that solves these problems.

[1074] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1075] In this invention, the server includes an input means for a user to input ingredient information, a transmission means for transmitting the input ingredient information to the server, a storage means for saving the transmitted ingredient information in a database, a check means for periodically checking the expiration dates of the ingredients saved in the database, a generation means for generating notification data about ingredients with upcoming expiration dates, a notification means for sending the generated notification data to a terminal, a display means for displaying the notification data to the user, a search means for searching for recipes based on ingredients in the database, a recipe suggestion means for sending the searched recipes to the terminal, a check means including a script for periodically checking ingredient information in the database, and a search means for preferentially searching for recipes containing ingredients with upcoming expiration dates. This allows users to easily manage the expiration dates of ingredients and enables the system to suggest appropriate recipes for efficiently using ingredients with upcoming expiration dates.

[1076] "User" refers to a person who uses the system.

[1077] "Ingredient information" refers to detailed data about ingredients, such as the name of the ingredient, purchase date, and expiration date.

[1078] "Input means" refers to the interface through which the user inputs information about ingredients, such as a smartphone app or web application.

[1079] "Transmission means" refers to a function for transmitting the input information about ingredients to the server.

[1080] "Storage means" refers to a function for storing transmitted information about ingredients in a database.

[1081] "Checking means" refers to a function that periodically checks the expiration dates of ingredients stored in the database.

[1082] "Generation means" refers to a function that creates notification data about food ingredients that are close to their expiration date.

[1083] The "notification means" refers to a function for transmitting the generated notification data to the user's terminal.

[1084] The "display means" refers to a function for displaying received notification data to the user.

[1085] "Search means" refers to the function for searching for recipes based on ingredients in the database.

[1086] "Recipe suggestion means" refers to a function for sending searched recipes to the user's terminal and suggesting them.

[1087] "Script" refers to program code that runs on the server to periodically check ingredient information in the database.

[1088] A "push notification" is a notification sent directly from the server to the user's device, and can be received even if the user does not have the app open.

[1089] "Database" refers to the storage system where information about ingredients is stored.

[1090] MODE FOR CARRYING OUT THE INVENTION

[1091] This invention is a system that efficiently manages the expiration dates of ingredients purchased by users and suggests recipes for making effective use of remaining ingredients. This system is realized by users inputting ingredient information and a server managing and processing that data. Specific embodiments of this system are described below.

[1092] First, the user uses a smartphone app or web application to input information about ingredients (such as name, purchase date, expiration date, etc.) For example, if a user purchases "milk," they enter information such as "milk," "October 5, 2023," and "October 20, 2023" into the app's input form and press the "Submit" button.

[1093] The device has a means of sending the input ingredient information to the server. This is done using an HTTP POST request, and the information is sent to the server. The server has a means of saving the received ingredient information to a database, and registers the received information in the database as a new record. For example, information such as "Milk," "October 5, 2023," and "October 20, 2023" is saved in the database.

[1094] The server periodically checks the ingredient information in the database and has a means of checking to list ingredients that are close to their expiration date. To do this, a batch process is executed at a fixed time every day using a script such as Python. For example, a script is executed every day at 9:00 AM to check ingredients that have an expiration date of less than three days.

[1095] The server has a generating means for generating notification data regarding ingredients whose expiration date is approaching. The generated notification data is sent by a notification means that sends the data to the user's device using push notification or email. For example, notification data stating "Milk's expiration date is approaching" is generated and sent to the device. The user can check the received notification on the device, which is displayed in a notification bar or a pop-up window.

[1096] Next, the server has a search means for searching available recipes based on the ingredient information stored in the database. It gives priority to searching for recipes that contain ingredients with an approaching expiration date. For example, it is possible to find a recipe called "cream stew" that uses "milk." The searched recipes are sent from the server to the user's terminal and are then sent via the recipe suggestion means. The user can check multiple suggested recipes on the terminal, select the recipe of interest, and view the details.

[1097] As a concrete example, the following shows the process from when a user purchases "milk" and enters and saves the information in the app. The user enters "milk," "October 5, 2023," and "October 20, 2023" into the app's input form and presses the "Submit" button. The app sends an HTTP POST request, and the server receives this request and stores the information in a database. The server runs a script every day at 9:00 AM to check ingredients with a best-by date of within three days, generates a notification stating "The milk's expiration date is approaching," and sends it as a push notification to the user's device. The user receives the notification and checks it on their device. The server searches for recipes using "milk," finds dishes such as "cream stew," and suggests them to the user. The user can then view the suggested recipes in the app and check their details.

[1098] An example prompt is, "I want to develop an app that allows you to enter the name, purchase date, and expiration date of purchased ingredients, manages that information, and notifies you when the expiration date is approaching. This app will provide recommended recipes based on the saved data. Please tell me the specific steps and data flow model."

[1099] In this way, the entire system works together, allowing users to easily manage the expiration dates of ingredients and efficiently use ingredients that are close to their expiration date when cooking, thereby reducing food waste and improving the quality of their diet.

[1100] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1101] Step 1:

[1102] The user inputs information about the ingredients.

[1103] Specific operation: The user opens a smartphone app or web application, enters the name of the purchased food, the purchase date, and the expiration date. For example, enter "Milk," "October 5, 2023," and "October 20, 2023," and presses the "Submit" button.

[1104] Input: Name of ingredient, purchase date, and expiration date.

[1105] Output: The input ingredient information is stored on the terminal.

[1106] Step 2:

[1107] The terminal transmits the input ingredient information to the server.

[1108] Specific operation: The input information is converted into JSON format and sent to the server using an HTTP POST request.

[1109] Input: Ingredient information stored on the device (JSON format).

[1110] Output: Ingredient information sent to the server.

[1111] Step 3:

[1112] The server stores the received ingredient information in a database.

[1113] What happens: The server parses the received JSON data, extracts the ingredient name, purchase date, and expiration date, and stores this information as a new record in the database.

[1114] Input: JSON formatted ingredient information sent to the server.

[1115] Output: A new record of ingredient information stored in the database.

[1116] Step 4:

[1117] The server periodically checks the food ingredient information in the database and lists ingredients that are close to their expiration date.

[1118] What it does: The server runs a Python script every day at 9 AM, which checks the expiration date field in the database and lists ingredients that have an expiration date within the next 3 days.

[1119] Input: Ingredient information stored in the database.

[1120] Output: A list of ingredients that are close to expiry.

[1121] Step 5:

[1122] The server generates notification data regarding food ingredients whose expiration date is approaching.

[1123] Specific behavior: Generate notification content based on the listed ingredients. For example, create a message saying "Milk is nearing its expiration date."

[1124] Input: A list of ingredients that are close to expiry.

[1125] Output: The generated notification data.

[1126] Step 6:

[1127] The server transmits the generated notification data to the terminal.

[1128] Specific operation: The generated notification data is sent to the user's device as a push notification or email.

[1129] Input: The generated notification data.

[1130] Output: Notification data sent to the user's device.

[1131] Step 7:

[1132] The terminal displays the received notification data to the user.

[1133] Specific behavior: The device receives the notification and displays it to the user in a pop-up or notification bar, for example, a notification saying "Your milk is nearing its expiration date."

[1134] Input: Notification data sent to the terminal.

[1135] Output: The notification data displayed to the user.

[1136] Step 8:

[1137] The server searches for available recipes based on the ingredient information stored in the database.

[1138] Specific behavior: To prioritize recipes that contain ingredients with an approaching expiration date, the name of the ingredient is used as a query to search for related recipes. For example, searching for recipes that use "milk" will find "cream stew."

[1139] Input: Ingredient information stored in the database.

[1140] Output: A list of searched recipes.

[1141] Step 9:

[1142] The server sends the searched recipe to the terminal.

[1143] Specific operation: The searched recipe information is sent to the user's terminal.

[1144] Input: A list of searched recipes.

[1145] Output: Recipe information sent to the user's device.

[1146] Step 10:

[1147] The terminal displays the recipe information to the user.

[1148] Specific operation: The device displays the received recipe information and allows the user to select a recipe they are interested in and view its details. For example, the device can display "cream stew recipe" and view its details.

[1149] Input: Recipe information sent to the device.

[1150] Output: The recipe information displayed to the user.

[1151] (Application example 1)

[1152] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1153] In modern society, busy users often find it difficult to keep track of the expiration dates of ingredients they purchase, resulting in food waste. Furthermore, they often lack the time to find appropriate recipes that make effective use of ingredients. To solve these problems, a system is needed that efficiently manages expiration dates and automatically suggests recipes that use ingredients with approaching expiration dates.

[1154] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1155] In this invention, the server includes an input means for a user to input ingredient information, a transmission means for transmitting the input ingredient information to the server, a storage means for saving the transmitted ingredient information in a database, a check means for periodically checking the expiration dates of the ingredients saved in the database, a generation means for generating notification data for ingredients whose expiration dates are approaching, a notification means for sending the generated notification data to a terminal, a display means for displaying the notification data to the user, a search means for searching for recipes based on ingredients in the database, a recipe suggestion means for sending the searched recipes to the terminal, a scanning means for scanning receipts to automatically recognize ingredient information, a calculation means for calculating the expiration date from the recognized ingredient information, and a notification function for sending expiration date notifications to the user's terminal. This allows users to efficiently manage ingredient expiration dates and receive suggested recipes that utilize ingredients without waste.

[1156] The "input means" is an interface that allows the user to input information about ingredients into the terminal.

[1157] The "transmission means" is a device or software function that transmits the input information about ingredients to the server.

[1158] "Storage means" refers to a device or software function for storing the transmitted information about ingredients in a database.

[1159] The "checking means" is a device or software function that periodically checks the expiration dates of ingredients stored in the database.

[1160] The "generation means" is a device or software function that generates notification data about food ingredients that are close to their expiration date.

[1161] The "notification means" is a device or software function that transmits the generated notification data to the user's terminal.

[1162] The "display means" is an interface for displaying notification data to the user.

[1163] A "search means" is a device or software function that searches for recipes based on ingredients in a database.

[1164] The "recipe suggestion means" is a device or software function that sends the searched recipes to the user's terminal.

[1165] "Scanning means" refers to a device or software function that scans receipts and automatically recognizes ingredient information.

[1166] The "calculation means" is a device or software function that calculates the expiration date from the information of the recognized ingredients.

[1167] A "notification function" is a device or software function that sends expiration date notifications to a user's terminal.

[1168] This invention provides a system for efficiently managing expiration dates of ingredients purchased by a user and proposing recipes for making effective use of remaining ingredients. This system includes the following means.

[1169] 1. Input Method

[1170] Users enter food information using a smartphone app or web application, either manually entering the name of the purchased food, the purchase date, and the expiration date, or automatically by scanning the receipt.

[1171] 2. Transmission Method

[1172] The ingredient information entered by the user is sent to the server using an HTTP POST request. Data is sent from the smartphone app or web application to the server.

[1173] 3. Preservation means

[1174] The server receives the transmitted ingredient information and stores it in a database, which records the ingredient name, purchase date, and expiration date.

[1175] 4. Checking Methods

[1176] The server periodically runs a script to check expiration dates in the database, for example, listing ingredients that are close to expiry at a set time each day.

[1177] 5. Generation means

[1178] The server generates notification data for ingredients that are nearing their expiration date, including the name of the ingredient, its expiration date, and a warning message.

[1179] 6. Means of notification

[1180] The generated notification data is sent to the user's device via email or push notification. SMS notifications can also be sent using the Twilio API.

[1181] 7. Display means

[1182] The user's device will display the received notification data in a pop-up or notification bar, informing the user that the expiration date of the food item is approaching.

[1183] 8. Search Methods

[1184] The server searches for available recipes based on the ingredient information stored in the database, with priority given to recipes that contain ingredients with an approaching expiration date.

[1185] 9. Recipe suggestion methods

[1186] The recipe information obtained from the server is sent to the user's device, where the user can select the recipe they are interested in and check the detailed information.

[1187] 10. Scanning Method

[1188] Use your smartphone camera and OCR (optical character recognition) software (e.g., pytesseract) to scan receipts and automatically recognize ingredient information.

[1189] 11. Means of calculation

[1190] Use data analysis algorithms to calculate expiration dates from recognized ingredient information. Calculate detailed information about ingredients that are nearing their expiration date.

[1191] 12. Notification function

[1192] The notification function uses the Twilio API, Firebase Cloud Messaging, etc. to send expiration date notifications to users' devices, allowing them to take immediate action.

[1193] Specific examples

[1194] Suppose a user purchases "milk" and scans the receipt with a smartphone app. The app uses OCR technology to extract information about the "milk," parses / enters the purchase date and expiration date, and sends it to the server. The server stores this information in a database and periodically checks the expiration date. When the expiration date approaches, the server sends a notification to the user's smartphone saying, "Your milk is about to expire." At the same time, a recipe using milk (e.g., cream stew) is suggested, and the user can refer to the recipe to cook.

[1195] Example prompts for generative AI models

[1196] To create a food management application, we manage the expiration dates of ingredients purchased by the user and automatically register the ingredient information through receipt scanning and OCR analysis. It also includes a function to send notifications to a smartphone when ingredients are approaching their expiration date. For example, if a receipt says "Milk, Purchase Date 2023-10-05, Expiration Date 2023-10-20," the relevant information is registered in the application, and a notification is sent when the expiration date approaches within three days.

[1197] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1198] Step 1:

[1199] The user inputs the ingredient information.

[1200] Users use a smartphone app to input the name, purchase date, and expiration date of the ingredients they purchased, or scan a receipt to automatically recognize the ingredient information. The input data is the ingredient name, purchase date, and expiration date.

[1201] Step 2:

[1202] The terminal transmits the input ingredient information to the server.

[1203] The smartphone app sends the entered ingredient data to the server using an HTTP POST request. The device sends the ingredient name, purchase date, and expiration date as parameters. The output is a communication success or failure status.

[1204] Step 3:

[1205] The server stores the received ingredient information in a database.

[1206] The server parses the received JSON format data and registers the ingredient name, purchase date, and expiration date in the database. The input data is the ingredient information, and the output is the result saved in the database.

[1207] Step 4:

[1208] The server periodically checks the expiration dates in the database.

[1209] Run the script periodically (for example, every day at a set time) to list ingredients that are close to expiry. The input data is all the ingredients in the database, and the output is a list of ingredients that are close to expiry.

[1210] Step 5:

[1211] The server generates notification data about food ingredients that are close to their expiration date.

[1212] Based on a list of ingredients whose expiration date is approaching, notification data for each ingredient is generated. The notification data includes the ingredient name, expiration date, and warning message. The input data is a list of ingredients whose expiration date is approaching, and the output is the generated notification data.

[1213] Step 6:

[1214] The server transmits the generated notification data to the terminal.

[1215] The server sends the notification data to the user's smartphone as an email or push notification. The input data is the notification data, and the output is the transmission result.

[1216] Step 7:

[1217] The terminal displays the received notification data.

[1218] The user's smartphone displays the notification data in a pop-up or notification bar to notify the user that the expiration date is approaching. The input data is the notification data, and the output is the displayed notification message.

[1219] Step 8:

[1220] The server searches for recipes based on the ingredient information in the database.

[1221] The system prioritizes the search for recipes containing ingredients with upcoming expiration dates and suggests them to the user. The input data is information about ingredients with upcoming expiration dates, and the output is a list of searched recipes.

[1222] Step 9:

[1223] The server transmits the searched recipe information to the terminal.

[1224] The server sends the searched recipe list to the user's smartphone, where the user can check the details. The input data is the recipe list, and the output is the transmission result and displayed recipe information.

[1225] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1226] This invention is a system that efficiently manages the expiration dates of ingredients purchased by the user, suggests recipes to make effective use of remaining ingredients, and recognizes the user's emotions to make appropriate suggestions.By combining an emotion engine with the ingredient management function, this system can make suggestions based on the user's emotional state.

[1227] To implement this system, the following main components are required:

[1228] 1. Register ingredients

[1229] Device:

[1230] It provides an interface for users to input information about ingredients they have purchased (such as name, purchase date, and expiration date). As a specific example, a user purchases "milk" and enters information such as the name "Milk," purchase date "October 5, 2023," and expiration date "October 20, 2023" into the app. It is also possible to input this information automatically using the barcode scanning function.

[1231] 2. Sending and saving food information

[1232] Device:

[1233] It has a function to send the inputted ingredient information to the server using a protocol such as HTTP POST request.

[1234] Example: The information "milk" entered by the user is sent to the server.

[1235] server:

[1236] The received ingredient information is saved in a database. The server registers this information in the database in JSON format, SQL format, or other formats.

[1237] Example: The server stores the information about "milk" in a database.

[1238] 3. Best before date management and notification

[1239] server:

[1240] Regularly check the ingredient information in the database and list ingredients that are close to their expiration date. For example, run a script at a fixed time every day to list ingredients that will expire in the next three days.

[1241] Example: The server detects that the expiration date of "Milk" (October 20, 2023) is approaching.

[1242] server:

[1243] Generate notification data about food items that are nearing their expiration date and prepare to notify users. This can be done via email or push notification.

[1244] Example: The server generates a notification saying "The milk's expiration date is approaching" and sends it to the device.

[1245] Device:

[1246] Display the received notification data to the user, such as via a popup or notification bar.

[1247] Example: The device notifies the user that "The milk is nearing its expiration date."

[1248] 4. Recipe suggestions

[1249] server:

[1250] The system searches for available recipes based on ingredient information stored in the database, with priority given to recipes containing ingredients with an approaching expiration date.

[1251] Example: The server finds the recipe "Cream Stew" that uses "Milk".

[1252] Device:

[1253] The recipe information obtained from the server is displayed to the user, and multiple recipes are suggested so that the user can select the recipe that interests them.

[1254] Example: The terminal displays "Cream Stew Recipe" and the user can select it to view details.

[1255] 5. Operation of the Emotion Engine

[1256] Device:

[1257] It provides a function for recognizing the user's emotions in real time, using facial recognition via a camera and analysis based on the user's input behavior.

[1258] server:

[1259] The system receives user emotional data and optimizes recipe suggestions and notification content. For example, if the user is tired, it will suggest easy-to-make dishes.

[1260] Example: If the user is recognized as "tired," the server will suggest "easy-to-make tamagoyaki (rolled omelet)."

[1261] Device:

[1262] The emotional engine displays optimized recipes and notifications to users, providing optimal suggestions tailored to the emotional state the user is facing.

[1263] Example: The device will display "Easy recipe recommended for when you're tired: Tamagoyaki."

[1264] summary

[1265] The system of this invention not only effectively manages expiration dates of ingredients, but also enhances user convenience and satisfaction by suggesting and notifying recipes that take the user's emotional state into consideration. Various elements work together to reduce food waste and improve the user's quality of life.

[1266] The processing flow will be explained below.

[1267] Step 1:

[1268] The user inputs the ingredient information.

[1269] The user enters information about the ingredients they purchased using an application on their device. The information to be entered includes the name of the ingredient, the purchase date, and the expiration date. This information can also be entered automatically using the barcode scanning function. As a specific example, a user purchases "milk" and enters the name "milk," the purchase date "October 5, 2023," and the expiration date "October 20, 2023."

[1270] Step 2:

[1271] The terminal transmits the ingredient information to the server.

[1272] The device converts the entered ingredient information into JSON format and sends it as an HTTP POST request to the server's API endpoint. This request includes the ingredient name, purchase date, and expiration date. For example, the information for "milk" is sent to the server.

[1273] Step 3:

[1274] The server stores the ingredient information in a database.

[1275] The server parses the received request, extracts the ingredient information, and saves it in the database. The server returns a save completion response to the terminal. For example, the server saves the information for "milk" in the database.

[1276] Step 4:

[1277] The server checks the expiration date.

[1278] The server runs a scheduled task every day at a fixed time, checking the expiration date of each ingredient in the database. It compares this to the current date and lists ingredients that are close to expiring (for example, within 3 days). For example, the server detects that "Milk" is close to expiring.

[1279] Step 5:

[1280] The server generates the notification data.

[1281] The server generates notification data about ingredients whose expiration date is approaching and prepares to send it to the user. This notification data includes the ingredient name and expiration date. For example, a notification saying "Milk's expiration date is approaching" is generated.

[1282] Step 6:

[1283] The device displays the notification to the user.

[1284] The device receives notification data from the server and displays it in a pop-up or notification bar. The user can check which ingredients are nearing their expiration date. For example, the device may notify the user that "The expiration date of your milk is approaching."

[1285] Step 7:

[1286] The server searches for available recipes.

[1287] The server searches for available recipes based on the ingredient information stored in the database. It prioritizes recipes that contain ingredients with an approaching expiration date. For example, the server finds a recipe called "cream stew" that uses "milk."

[1288] Step 8:

[1289] The server transmits the recipe information to the terminal.

[1290] The server sends the searched recipe information to the device in JSON format. The recipe information includes the necessary ingredients and cooking instructions. For example, a "cream stew recipe" is sent to the device.

[1291] Step 9:

[1292] The device displays recipe suggestions to the user.

[1293] The device displays the recipe information received from the server and allows the user to select the recipe they are interested in. For example, the device displays "cream stew recipe" and the user can select it and view the details.

[1294] Step 10:

[1295] The device recognizes the user's emotions.

[1296] The device's camera and sensors are used to analyze the user's facial expressions and voice to obtain emotional data, detecting whether the user is "tired" or "stressed."

[1297] Step 11:

[1298] The server optimizes the recipe based on emotional data.

[1299] The server receives the user's emotional data and adjusts the recipe suggestions and notification content based on that data. For example, if the user is tired, it will prioritize suggestions of easy-to-make dishes. For example, if the user is tired, it will suggest "easy-to-make tamagoyaki" (rolled omelet).

[1300] Step 12:

[1301] The device displays recipes and notifications optimized by the emotion engine.

[1302] The device uses the emotion engine sent from the server to display optimized recipe information and notifications to the user. For example, it displays information such as "Easy recipe recommended for when you're tired: Tamagoyaki."

[1303] Through these steps, this system can provide users with a more personalized experience by combining food expiration date management and emotion recognition.

[1304] Example 2

[1305] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1306] Many conventional food management systems are specialized in managing and notifying expiration dates, and have the problem of not being able to make suggestions that take the user's emotions into consideration. As a result, the recipes provided may not be suitable for the user's situation, which can result in a decrease in user convenience and satisfaction. Another problem is that ingredients are often entered manually, which is a time-consuming and labor-intensive process.

[1307] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1308] In this invention, the server includes input means for a user to input food information, transmission means for transmitting the input food information to the server, storage means for saving the transmitted food information in a database, check means for periodically checking the expiration dates of the foods saved in the database, generation means for generating notification data for foods with upcoming expiration dates, notification means for sending the generated notification data to a terminal, display means for displaying the notification data to the user, search means for searching for recipes based on foods in the database, recipe suggestion means for sending the searched recipes to the terminal, emotion recognition means for recognizing the user's emotion, emotion data transmission means for sending emotion data recognized by the emotion recognition means to the server, and optimization means for optimizing recipe suggestions based on the emotion data. This not only enables food expiration date management but also makes it possible to suggest optimal recipes tailored to the user's emotions.

[1309] "Input means" refers to a device or interface that allows a user to input information about a food product.

[1310] "Transmission means" refers to a device or protocol that transmits the input food information to the server.

[1311] "Storage means" means a device or process that stores the submitted food information in a database.

[1312] A "checking means" is a device or script that periodically checks the expiration dates of food items stored in the database.

[1313] A "generator" is a device or algorithm that generates notification data about food products that are nearing their expiration date.

[1314] The "notification means" is a device or protocol that transmits the generated notification data to the terminal.

[1315] A "display means" is a device or interface that displays notification data to a user.

[1316] A "search tool" is a device or algorithm that searches for recipes based on foods in the database.

[1317] A "recipe suggestion means" is a device or protocol that sends searched recipes to a terminal.

[1318] An "emotion recognition means" is a device or algorithm that recognizes a user's emotions.

[1319] The "emotion data transmission means" is a device or protocol that transmits the emotion data recognized by the emotion recognition means to the server.

[1320] An "optimization means" is a device or algorithm that optimizes recipe suggestions based on emotion data.

[1321] This invention is a system that efficiently manages the expiration dates of food purchased by a user, suggests recipes for making effective use of remaining food, and recognizes the user's emotions to make appropriate suggestions. Specific embodiments of this invention are described below.

[1322] Hardware and Software Configuration

[1323] Input method: Provides an interface for users to input food information. Specifically, this corresponds to an application installed on a device such as a smartphone or tablet. Users can manually enter the food name, purchase date, and expiration date, or use a barcode scanning function to automatically enter them.

[1324] Transmission method: HTTP POST request is used as the protocol to send the entered food information to the server. For example, when a user enters information about "milk," this information is sent to the server in JSON format.

[1325] Storage method: The server stores the received food information in a database using SQL Server or NoSQL database, and the food information is stored in an appropriate format.

[1326] Check method: The server periodically runs a script to check the food information in the database. This process can be automated using a cron job or similar, and lists foods that have a shelf life of less than three days.

[1327] Generation method: Generates notification data for foods in the database that are close to their expiration date. The generated notification data is prepared in a format that notifies the user via email, push notification, etc.

[1328] Notification method: The generated notification data is sent to the device. This is done using a mail server or push notification service. The notification data is displayed in a pop-up or notification bar so that the user can check it immediately.

[1329] Display method: The device displays the notification data received from the server to the user. Specifically, the device uses the smartphone's notification bar or push notification to allow the user to quickly check the information.

[1330] Search method: The server searches for available recipes based on the food information stored in the database, with priority given to searching and suggesting recipes that include food items with an approaching expiration date.

[1331] Recipe suggestion method: The server sends the searched recipes to the terminal and suggests multiple recipes, allowing the user to select the recipe they are interested in and check the details.

[1332] Emotion recognition: The device provides a function to recognize the user's emotions in real time. Emotion recognition can be achieved by facial recognition via a camera or by analyzing the user's input behavior.

[1333] Emotion data transmission method: The recognized emotion data is transmitted from the device to the server. Based on this data, the server optimizes the next recipe suggestion.

[1334] Optimization method: The server receives the emotional data and suggests recipes according to the user's emotional state. For example, if the server detects that the user is "tired," it will prioritize suggestions for easy-to-make dishes.

[1335] Specific examples and prompts for the generative AI model

[1336] Examples:

[1337] It explains how the system notifies users when the milk they have purchased is nearing its expiration date and suggests suitable recipes.

[1338] Specific operation example: A user purchases "milk" and enters the name "milk," the purchase date "October 5, 2023," and the expiration date "October 20, 2023" into the device. The entered information is sent to the server and stored in a database. The server periodically checks the database and lists foods with an expiration date of three days or less. It then generates notification data stating "The milk's expiration date is approaching" and sends it to the user's device. When the user receives the notification, the server provides a recipe for "cream stew."

[1339] Example prompt for a generative AI model:

[1340] Explain how the system will notify the user when the "Milk" they have purchased is nearing its expiration date and suggest suitable recipes.

[1341] This invention not only enables the expiration date of food products to be managed, but also makes it possible to propose optimal recipes that take into account the user's feelings, thereby improving user convenience and satisfaction.

[1342] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1343] Specific processing steps of the system program

[1344] Step 1: User enters food information

[1345] The user inputs information about the food they have purchased into the terminal, such as the name of the food, the date of purchase, the expiration date, etc. This input is performed using a keyboard or a barcode scanner.

[1346] Input: food name, purchase date, expiration date

[1347] Output: Input data on the terminal

[1348] What it does: A user opens the app, enters "Milk," sets the purchase date to "2023-10-05," and the expiration date to "2023-10-20." By using the barcode scanner, this information is automatically entered.

[1349] Step 2: Device sends data to server

[1350] The terminal sends the food information entered by the user to the server. The data is sent via an HTTP POST request in JSON format.

[1351] Input: Food information entered by the user into the device

[1352] Output: JSON data sent to the server

[1353] Specific operation: The device sends JSON data (e.g., "{\"name\": \"Milk\", \"purchase_date\": \"2023-10-05\", \"expiry_date\": \"2023-10-20\"}") to the server using "POST / api / food_register".

[1354] Step 3: Data received and stored by the server

[1355] The server receives the food information sent from the device and stores it in a database using SQL queries.

[1356] Input: JSON data sent from the terminal

[1357] Output: Food information stored in the database

[1358] Specific operation: The server executes the SQL query "INSERT INTO food_table (name, purchase_date, expiration_date) VALUES (\'Milk\', \'2023-10-05\', \'2023-10-20\')" and saves it in the database.

[1359] Step 4: Expiry date check by server

[1360] The server runs a script that periodically checks the food information in the database. This script is run by a cron job at midnight every day and lists foods that have an expiration date within the next three days.

[1361] Input: Food information stored in a database

[1362] Output: A list of foods that are close to expiry date

[1363] Specific operation: The server executes the SQL query "SELECT FROM food_table WHERE best-before_date <= DATE_ADD(CURDATE(), INTERVAL 3 DAY)".

[1364] Step 5: Notification data generation by the server

[1365] The server generates notification data about food products that are nearing their expiration date, and the generated data is ready to send notifications to users.

[1366] Input: List of foods that are close to expiry date

[1367] Output: Generated notification data

[1368] Specific behavior: The server generates a notification message saying "The milk is nearing its expiration date."

[1369] Step 6: Send notification data to the device

[1370] The server then sends the generated notification data to the user's device, which can take the form of email or push notification.

[1371] Input: Notification data

[1372] Output: Notification sent to the device

[1373] Specific operation: The server sends the notification data via a push notification service or mail server.

[1374] Step 7: Displaying notifications on your device

[1375] The device displays the received notification data to the user in a pop-up or notification bar format.

[1376] Input: Notification data sent from the server

[1377] Output: The notification displayed to the user

[1378] Specific behavior: The device will display a pop-up notification saying "The milk's expiration date is approaching."

[1379] Step 8: Server Lookup for Recipes

[1380] The server searches for available recipes based on the food information stored in the database, with priority given to recipes that include food items with an approaching expiration date.

[1381] Input: Food information in the database

[1382] Output: Searched recipe list

[1383] Specific behavior: The server executes the SQL query "SELECT FROM recipe WHERE ingredients LIKE \"%milk%\"".

[1384] Step 9: Send the recipe to your device

[1385] The server sends the searched recipes to the terminal, and multiple recipes are suggested to the user.

[1386] Input: Searched recipe list

[1387] Output: Recipe information sent to the device

[1388] Specific operation: The server sends recipe information to the terminal and displays it to the user.

[1389] Step 10: Emotion recognition on the device

[1390] The device will provide features for recognizing users' emotions in real time, including camera-based facial recognition and behavioral analysis.

[1391] Input: User's facial expressions and behavior data

[1392] Output: Recognized emotion data

[1393] Specific operation: The device analyzes the user's face captured by the camera and recognizes emotions such as "tired."

[1394] Step 11: Send emotion data to the server

[1395] The device sends the recognized emotion data to the server, which then uses it to suggest the next recipe.

[1396] Input: Recognized emotion data

[1397] Output: Emotion data sent to the server

[1398] Specific operation: The device sends emotion data in JSON format to the server using "POST / api / emotion_data".

[1399] Step 12: Optimizing recipe suggestions by the server

[1400] The server optimizes recipe suggestions based on emotion data, and selects recipes that suit the user's state.

[1401] Input: Emotion data, recipe information in the database

[1402] Output: Optimized recipe suggestions

[1403] Specific operation: For users who are recognized by the server as being "tired," the server will prioritize suggesting easy-to-make recipes for "tamagoyaki" (rolled omelet).

[1404] (Application example 2)

[1405] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1406] In modern consumer lifestyles, it is difficult for users to efficiently manage the expiration dates of purchased ingredients, resulting in food waste. It is also not easy for users to select the optimal recipe based on their emotional state at any given time. At the same time, there is a demand for systems that can suggest appropriate recipes based on emotions, but no such systems have existed to date. This situation has led to problems that reduce user convenience and satisfaction.

[1407] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1408] In this invention, the server includes emotion recognition means for recognizing the emotional state of the user, recipe optimization means for optimizing recipes based on the recognized emotional state, and generation means for generating notification data for ingredients whose expiration dates are approaching, thereby enabling the server to efficiently manage the expiration dates of ingredients purchased by the user and propose optimal recipes according to the user's emotional state.

[1409] The "input means" is an interface through which the user inputs information about ingredients.

[1410] The "transmission means" is a function that transmits the input information about ingredients to the server.

[1411] The "storage means" is a function that stores the transmitted information about ingredients in a database.

[1412] The "checking means" is a function that periodically checks the expiration dates of ingredients stored in the database.

[1413] The "generation means" is a function that generates notification data about food ingredients that are close to their expiration date.

[1414] The "notification means" is a function that transmits the generated notification data to the terminal.

[1415] The "display means" is a function that displays the notification data to the user.

[1416] "Search means" is a function that searches for recipes based on ingredients in the database.

[1417] The "recipe suggestion means" is a function that sends the searched recipe to the terminal.

[1418] The "emotion recognition means" is a function for recognizing the user's emotional state.

[1419] The "recipe optimizer" is a function that optimizes recipes based on the recognized emotional state.

[1420] This invention is a system that efficiently manages the expiration dates of ingredients purchased by a user and suggests optimal recipes based on the emotional state of the user. Specific embodiments for carrying out this invention are described below.

[1421] 1. Register ingredients

[1422] Program Generation

[1423] We provide a smartphone application that allows users to scan the barcode of ingredients to register information about the ingredients they have purchased (such as name, purchase date, expiration date, etc.). When a user scans the barcode, the corresponding ingredient information is automatically entered into the application.

[1424] Processing Description

[1425] In this step, the smartphone uses the barcode scanning function to obtain information about ingredients and sends the entered data to the server, which then stores the data in a database. The specific hardware used is the smartphone, and the software is a barcode scanning module.

[1426] 2. Best before date management and notification

[1427] Program Generation

[1428] A script runs on the server every day, checking the food ingredient information in the database and listing ingredients that are close to their expiration date. Notification data is generated for the listed ingredients and sent to the user via email or push notification.

[1429] Processing Description

[1430] In this step, the server checks expiration dates using Python scripts or scheduled scripts. For ingredients with an approaching expiration date, it generates notification data and sends it to smartphones or other devices. The specific software used is a scripting language (e.g., Python) and a notification service (e.g., Firebase Cloud Messaging).

[1431] 3. Recipe suggestions

[1432] Program Generation

[1433] The server searches for available recipes based on the ingredient information stored in the database, giving priority to recipes that contain ingredients with an approaching expiration date, and sends the results to the user's device to suggest recipes.

[1434] Processing Description

[1435] The server searches for recipe data based on the user's ingredient data. The recipe results are sent to the user's device, where the recipe details are displayed. The server executes a database query to search for recipes and returns the results in JSON format.

[1436] 4. Emotion Recognition and Recipe Optimization

[1437] Program Generation

[1438] We provide an emotion recognition engine that uses facial recognition and voice analysis via a camera to recognize the user's emotional state, and the emotion engine optimizes the recipe suggestions based on the recognized emotional state.

[1439] Processing Description

[1440] The system analyzes the user's emotional state using the camera and microphone on the smartphone or head-mounted display (HMD). The analyzed emotional data is sent to the server, which then provides recipes based on the user's emotional state. The specific software used is an emotion recognition library (e.g., OpenCV, Affectiva).

[1441] 5. Specific Examples

[1442] For example, if a user scans "Milk," the data is automatically entered into the app and saved in a database. When the expiration date approaches, a notification stating "Milk expiration date is approaching" is sent to the smartphone. If the emotion recognition engine detects that the user is "tired," the server will suggest "easy-to-make tamagoyaki (rolled omelet)."

[1443] 6. Examples of prompts

[1444] "I've entered the ingredients I purchased (milk, eggs, cabbage) and their expiration dates. I'm feeling tired right now and would like to know what easy meals I can make. Please tell me the best recipe based on this data and my mood."

[1445] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1446] Step 1:

[1447] Entering ingredient information

[1448] The user inputs the ingredients they purchased using the barcode scanning function of their smartphone. The smartphone reads the barcode and obtains information such as the ingredient name, purchase date, and expiration date. The barcode data is read as input, and the ingredient information is automatically entered as output.

[1449] Step 2:

[1450] Sending ingredient information

[1451] The smartphone sends the entered ingredient information to the server via an HTTP POST request. The smartphone receives ingredient information as input and sends the information as output to the server in JSON format.

[1452] Step 3:

[1453] Saving food information

[1454] The server stores the received ingredient information in a database. It analyzes the ingredient information received as input and registers the data using SQL queries etc. to store it in the database. As output, records stored in the database are generated.

[1455] Step 4:

[1456] Regular check of expiration dates

[1457] The server periodically checks the ingredient information in the database using a script and lists ingredients with an approaching expiration date. It compares the current date with the ingredient information in the database as input and extracts ingredients with an upcoming expiration date. The output is a list of ingredients with an upcoming expiration date.

[1458] Step 5:

[1459] Generate notification data

[1460] The server generates notification data about ingredients with upcoming expiration dates. It receives as input a list of ingredients with upcoming expiration dates and generates a notification message to be sent to the user. The notification message is generated as output.

[1461] Step 6:

[1462] Sending notification data

[1463] The server sends the generated notification data to the smartphone. It takes the generated notification message as input and sends it to the smartphone using a notification protocol (e.g., Firebase Cloud Messaging). It generates a notification to be sent to the smartphone as output.

[1464] Step 7:

[1465] Ingredient-based recipe search

[1466] The server searches for recipes based on ingredient information in the database, with priority given to recipes containing ingredients with an approaching expiration date. It receives ingredient information as input and searches the database for matching recipes. A list of recipes is generated as output.

[1467] Step 8:

[1468] Sending recipe information

[1469] The server sends the searched recipe information to the smartphone. It receives a list of recipes as input and transmits the data using a protocol for sending to the smartphone app. The recipe data that is sent to the smartphone is generated as output.

[1470] Step 9:

[1471] Viewing Recipes

[1472] The smartphone displays the received recipe information to the user. Using the received recipe data as input, the recipe information is displayed on the app's UI so that it can be seen by the user. As output, a recipe screen is generated that the user can view.

[1473] Step 10:

[1474] emotion recognition

[1475] Smartphones and head-mounted displays use cameras and microphones to recognize the user's emotional state. Camera images and audio data are analyzed as input, and the user's emotional state is analyzed. Recognized emotional data is generated as output.

[1476] Step 11:

[1477] Emotion-based recipe optimization

[1478] The server optimizes the recipe suggestions based on the recognized emotional state. It receives emotional data as input and selects recipes that match that state. It generates an optimized recipe list as output.

[1479] Step 12:

[1480] Displaying recipes based on emotions

[1481] The smartphone displays the optimized recipe information to the user. It receives the optimized recipe data as input and displays the recipe information visible to the user on the app's UI. As output, an optimized recipe screen is generated that the user can view.

[1482] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1483] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1484] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1485] [Fourth embodiment]

[1486] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1487] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1488] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1489] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1490] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1491] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1492] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1493] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1494] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1495] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1496] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1497] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1498] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1499] This invention is a system that efficiently manages the expiration dates of ingredients purchased by users and suggests recipes to make effective use of remaining ingredients. This system is realized by users inputting ingredient information and the server managing and processing that data.

[1500] To implement this system, the following main components are required:

[1501] 1. Register ingredients

[1502] Device:

[1503] It provides an interface for users to input information about purchased ingredients (such as name, purchase date, expiration date, etc.). For example, a smartphone app or web application can be used.

[1504] As a specific example, a user purchases "milk" and enters information such as the name "milk," the purchase date "October 5, 2023," and the expiration date "October 20, 2023" into the app.

[1505] 2. Sending and saving food information

[1506] Device:

[1507] It has a function to send the input ingredient information to the server using a protocol such as HTTP POST request.

[1508] Example: The information "milk" entered by the user is sent to the server.

[1509] server:

[1510] The received ingredient information is saved in a database. The server registers this information in the database in JSON format, SQL format, or other formats.

[1511] Example: The server stores the information about "milk" in a database.

[1512] 3. Best before date management and notification

[1513] server:

[1514] Regularly check the ingredient information in the database and list ingredients that are close to their expiration date. For example, run a script at a fixed time every day to list ingredients that will expire in the next three days.

[1515] Example: The server detects that the expiration date of "Milk" (October 20, 2023) is approaching.

[1516] server:

[1517] Notification data about food items nearing their expiration date is generated and sent to the user's device via email or push notification.

[1518] Example: The server generates a notification saying "The milk's expiration date is approaching" and sends it to the device.

[1519] Device:

[1520] Display the received notification data to the user, such as via a popup or notification bar.

[1521] Example: The device notifies the user that "The milk is nearing its expiration date."

[1522] 4. Recipe suggestions

[1523] server:

[1524] The system searches for available recipes based on ingredient information stored in the database, with priority given to recipes containing ingredients with an approaching expiration date.

[1525] Example: The server finds the recipe "Cream Stew" that uses "Milk".

[1526] Device:

[1527] The recipe information obtained from the server is displayed to the user, and multiple recipes are suggested so that the user can select the recipe that interests them.

[1528] Example: The terminal displays "Cream Stew Recipe" and the user can select it to view details.

[1529] summary

[1530] The system of this invention aims to streamline food management for users and reduce waste. Each processing step works in tandem, allowing users to easily understand the expiration date of ingredients and receive suggestions for recipes using remaining ingredients. This reduces food waste and improves the quality of their diet.

[1531] The processing flow will be explained below.

[1532] Step 1:

[1533] The user inputs the ingredient information.

[1534] The user uses the device to input information such as the name of the ingredient, purchase date, expiration date, etc. Alternatively, the barcode scanning function can be used to automatically input ingredient information if necessary.

[1535] Step 2:

[1536] The terminal transmits the input ingredient information.

[1537] The device converts the ingredient information entered by the user into JSON format and sends it as an HTTP POST request to the server's API endpoint.

[1538] Step 3:

[1539] The server stores the ingredient information.

[1540] The server parses the received request, extracts the ingredient information, and saves it in the database. Once the saving is complete, it returns a successful save response to the device.

[1541] Step 4:

[1542] The server checks the expiration date.

[1543] The server runs a scheduled task every day at a fixed time, checking the expiration date of each ingredient stored in the database, comparing it with the current date, and listing ingredients that are close to expiry (e.g., within 3 days).

[1544] Step 5:

[1545] The server generates the notification data.

[1546] The server generates notification data about ingredients whose expiration date is approaching and prepares it for transmission to the terminal. The notification data includes the ingredient name and expiration date.

[1547] Step 6:

[1548] The device displays a notification to the user.

[1549] The device receives notification data from the server and displays it to the user as a pop-up or alert, allowing the user to check which ingredients are nearing their expiration date.

[1550] Step 7:

[1551] The server searches for available recipes.

[1552] When a user requests it, or the server periodically checks the database, it searches for recipes based on ingredients in stock from the recipe database. Recipes that include ingredients with an approaching expiration date are given priority.

[1553] Step 8:

[1554] The server transmits the recipe information to the terminal.

[1555] The server sends the searched recipe information to the device in JSON format, including the ingredients needed and cooking instructions.

[1556] Step 9:

[1557] The device displays recipe suggestions to the user.

[1558] The device displays the recipe information received from the server on the user interface, allowing the user to select a recipe of interest, check the details, and start cooking.

[1559] Example 1

[1560] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1561] Managing the expiration dates of ingredients is an important task in the home, and if not managed properly, food waste will occur. Furthermore, in order to efficiently utilize ingredients that are nearing their expiration date, appropriate recipe suggestions are required. However, manually managing expiration dates and finding recipes that make effective use of ingredients is time-consuming. There is a need to provide a smart system that solves these problems.

[1562] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1563] In this invention, the server includes an input means for a user to input ingredient information, a transmission means for transmitting the input ingredient information to the server, a storage means for saving the transmitted ingredient information in a database, a check means for periodically checking the expiration dates of the ingredients saved in the database, a generation means for generating notification data about ingredients with upcoming expiration dates, a notification means for sending the generated notification data to a terminal, a display means for displaying the notification data to the user, a search means for searching for recipes based on ingredients in the database, a recipe suggestion means for sending the searched recipes to the terminal, a check means including a script for periodically checking ingredient information in the database, and a search means for preferentially searching for recipes containing ingredients with upcoming expiration dates. This allows users to easily manage the expiration dates of ingredients and enables the system to suggest appropriate recipes for efficiently using ingredients with upcoming expiration dates.

[1564] "User" refers to a person who uses the system.

[1565] "Ingredient information" refers to detailed data about ingredients, such as the name of the ingredient, purchase date, and expiration date.

[1566] "Input means" refers to the interface through which the user inputs information about ingredients, such as a smartphone app or web application.

[1567] "Transmission means" refers to a function for transmitting the input information about ingredients to the server.

[1568] "Storage means" refers to a function for storing transmitted information about ingredients in a database.

[1569] "Checking means" refers to a function that periodically checks the expiration dates of ingredients stored in the database.

[1570] "Generation means" refers to a function that creates notification data about food ingredients that are close to their expiration date.

[1571] The "notification means" refers to a function for transmitting the generated notification data to the user's terminal.

[1572] The "display means" refers to a function for displaying received notification data to the user.

[1573] "Search means" refers to the function for searching for recipes based on ingredients in the database.

[1574] "Recipe suggestion means" refers to a function for sending searched recipes to the user's terminal and suggesting them.

[1575] "Script" refers to program code that runs on the server to periodically check ingredient information in the database.

[1576] A "push notification" is a notification sent directly from the server to the user's device, and can be received even if the user does not have the app open.

[1577] "Database" refers to the storage system where information about ingredients is stored.

[1578] MODE FOR CARRYING OUT THE INVENTION

[1579] This invention is a system that efficiently manages the expiration dates of ingredients purchased by users and suggests recipes for making effective use of remaining ingredients. This system is realized by users inputting ingredient information and a server managing and processing that data. Specific embodiments of this system are described below.

[1580] First, the user uses a smartphone app or web application to input information about ingredients (such as name, purchase date, expiration date, etc.) For example, if a user purchases "milk," they enter information such as "milk," "October 5, 2023," and "October 20, 2023" into the app's input form and press the "Submit" button.

[1581] The device has a means of sending the input ingredient information to the server. This is done using an HTTP POST request, and the information is sent to the server. The server has a means of saving the received ingredient information to a database, and registers the received information in the database as a new record. For example, information such as "Milk," "October 5, 2023," and "October 20, 2023" is saved in the database.

[1582] The server periodically checks the ingredient information in the database and has a means of checking to list ingredients that are close to their expiration date. To do this, a batch process is executed at a fixed time every day using a script such as Python. For example, a script is executed every day at 9:00 AM to check ingredients that have an expiration date of less than three days.

[1583] The server has a generating means for generating notification data regarding ingredients whose expiration date is approaching. The generated notification data is sent by a notification means that sends the data to the user's device using push notification or email. For example, notification data stating "Milk's expiration date is approaching" is generated and sent to the device. The user can check the received notification on the device, which is displayed in a notification bar or a pop-up window.

[1584] Next, the server has a search means for searching available recipes based on the ingredient information stored in the database. It gives priority to searching for recipes that contain ingredients with an approaching expiration date. For example, it is possible to find a recipe called "cream stew" that uses "milk." The searched recipes are sent from the server to the user's terminal and are then sent via the recipe suggestion means. The user can check multiple suggested recipes on the terminal, select the recipe of interest, and view the details.

[1585] As a concrete example, the following shows the process from when a user purchases "milk" and enters and saves the information in the app. The user enters "milk," "October 5, 2023," and "October 20, 2023" into the app's input form and presses the "Submit" button. The app sends an HTTP POST request, and the server receives this request and stores the information in a database. The server runs a script every day at 9:00 AM to check ingredients with a best-by date of within three days, generates a notification stating "The milk's expiration date is approaching," and sends it as a push notification to the user's device. The user receives the notification and checks it on their device. The server searches for recipes using "milk," finds dishes such as "cream stew," and suggests them to the user. The user can then view the suggested recipes in the app and check their details.

[1586] An example prompt is, "I want to develop an app that allows you to enter the name, purchase date, and expiration date of purchased ingredients, manages that information, and notifies you when the expiration date is approaching. This app will provide recommended recipes based on the saved data. Please tell me the specific steps and data flow model."

[1587] In this way, the entire system works together, allowing users to easily manage the expiration dates of ingredients and efficiently use ingredients that are close to their expiration date when cooking, thereby reducing food waste and improving the quality of their diet.

[1588] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1589] Step 1:

[1590] The user inputs information about the ingredients.

[1591] Specific operation: The user opens a smartphone app or web application, enters the name of the purchased food, the purchase date, and the expiration date. For example, enter "Milk," "October 5, 2023," and "October 20, 2023," and presses the "Submit" button.

[1592] Input: Name of ingredient, purchase date, and expiration date.

[1593] Output: The input ingredient information is stored on the terminal.

[1594] Step 2:

[1595] The terminal transmits the input ingredient information to the server.

[1596] Specific operation: The input information is converted into JSON format and sent to the server using an HTTP POST request.

[1597] Input: Ingredient information stored on the device (JSON format).

[1598] Output: Ingredient information sent to the server.

[1599] Step 3:

[1600] The server stores the received ingredient information in a database.

[1601] What happens: The server parses the received JSON data, extracts the ingredient name, purchase date, and expiration date, and stores this information as a new record in the database.

[1602] Input: JSON formatted ingredient information sent to the server.

[1603] Output: A new record of ingredient information stored in the database.

[1604] Step 4:

[1605] The server periodically checks the food ingredient information in the database and lists ingredients that are close to their expiration date.

[1606] What it does: The server runs a Python script every day at 9 AM, which checks the expiration date field in the database and lists ingredients that have an expiration date within the next 3 days.

[1607] Input: Ingredient information stored in the database.

[1608] Output: A list of ingredients that are close to expiry.

[1609] Step 5:

[1610] The server generates notification data regarding food ingredients whose expiration date is approaching.

[1611] Specific behavior: Generate notification content based on the listed ingredients. For example, create a message saying "Milk is nearing its expiration date."

[1612] Input: A list of ingredients that are close to expiry.

[1613] Output: The generated notification data.

[1614] Step 6:

[1615] The server transmits the generated notification data to the terminal.

[1616] Specific operation: The generated notification data is sent to the user's device as a push notification or email.

[1617] Input: The generated notification data.

[1618] Output: Notification data sent to the user's device.

[1619] Step 7:

[1620] The terminal displays the received notification data to the user.

[1621] Specific behavior: The device receives the notification and displays it to the user in a pop-up or notification bar, for example, a notification saying "Your milk is nearing its expiration date."

[1622] Input: Notification data sent to the terminal.

[1623] Output: The notification data displayed to the user.

[1624] Step 8:

[1625] The server searches for available recipes based on the ingredient information stored in the database.

[1626] Specific behavior: To prioritize recipes that contain ingredients with an approaching expiration date, the name of the ingredient is used as a query to search for related recipes. For example, searching for recipes that use "milk" will find "cream stew."

[1627] Input: Ingredient information stored in the database.

[1628] Output: A list of searched recipes.

[1629] Step 9:

[1630] The server sends the searched recipe to the terminal.

[1631] Specific operation: The searched recipe information is sent to the user's terminal.

[1632] Input: A list of searched recipes.

[1633] Output: Recipe information sent to the user's device.

[1634] Step 10:

[1635] The terminal displays the recipe information to the user.

[1636] Specific operation: The device displays the received recipe information and allows the user to select a recipe they are interested in and view its details. For example, the device can display "cream stew recipe" and view its details.

[1637] Input: Recipe information sent to the device.

[1638] Output: The recipe information displayed to the user.

[1639] (Application example 1)

[1640] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1641] In modern society, busy users often find it difficult to keep track of the expiration dates of ingredients they purchase, resulting in food waste. Furthermore, they often lack the time to find appropriate recipes that make effective use of ingredients. To solve these problems, a system is needed that efficiently manages expiration dates and automatically suggests recipes that use ingredients with approaching expiration dates.

[1642] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1643] In this invention, the server includes an input means for a user to input ingredient information, a transmission means for transmitting the input ingredient information to the server, a storage means for saving the transmitted ingredient information in a database, a check means for periodically checking the expiration dates of the ingredients saved in the database, a generation means for generating notification data for ingredients whose expiration dates are approaching, a notification means for sending the generated notification data to a terminal, a display means for displaying the notification data to the user, a search means for searching for recipes based on ingredients in the database, a recipe suggestion means for sending the searched recipes to the terminal, a scanning means for scanning receipts to automatically recognize ingredient information, a calculation means for calculating the expiration date from the recognized ingredient information, and a notification function for sending expiration date notifications to the user's terminal. This allows users to efficiently manage ingredient expiration dates and receive suggested recipes that utilize ingredients without waste.

[1644] The "input means" is an interface that allows the user to input information about ingredients into the terminal.

[1645] The "transmission means" is a device or software function that transmits the input information about ingredients to the server.

[1646] "Storage means" refers to a device or software function for storing the transmitted information about ingredients in a database.

[1647] The "checking means" is a device or software function that periodically checks the expiration dates of ingredients stored in the database.

[1648] The "generation means" is a device or software function that generates notification data about food ingredients that are close to their expiration date.

[1649] The "notification means" is a device or software function that transmits the generated notification data to the user's terminal.

[1650] The "display means" is an interface for displaying notification data to the user.

[1651] A "search means" is a device or software function that searches for recipes based on ingredients in a database.

[1652] The "recipe suggestion means" is a device or software function that sends the searched recipes to the user's terminal.

[1653] "Scanning means" refers to a device or software function that scans receipts and automatically recognizes ingredient information.

[1654] The "calculation means" is a device or software function that calculates the expiration date from the information of the recognized ingredients.

[1655] A "notification function" is a device or software function that sends expiration date notifications to a user's terminal.

[1656] This invention provides a system for efficiently managing expiration dates of ingredients purchased by a user and proposing recipes for making effective use of remaining ingredients. This system includes the following means.

[1657] 1. Input Method

[1658] Users enter food information using a smartphone app or web application, either manually entering the name of the purchased food, the purchase date, and the expiration date, or automatically by scanning the receipt.

[1659] 2. Transmission Method

[1660] The ingredient information entered by the user is sent to the server using an HTTP POST request. Data is sent from the smartphone app or web application to the server.

[1661] 3. Preservation means

[1662] The server receives the transmitted ingredient information and stores it in a database, which records the ingredient name, purchase date, and expiration date.

[1663] 4. Checking Methods

[1664] The server periodically runs a script to check expiration dates in the database, for example, listing ingredients that are close to expiry at a set time each day.

[1665] 5. Generation means

[1666] The server generates notification data for ingredients that are nearing their expiration date, including the name of the ingredient, its expiration date, and a warning message.

[1667] 6. Means of notification

[1668] The generated notification data is sent to the user's device via email or push notification. SMS notifications can also be sent using the Twilio API.

[1669] 7. Display means

[1670] The user's device will display the received notification data in a pop-up or notification bar, informing the user that the expiration date of the food item is approaching.

[1671] 8. Search Methods

[1672] The server searches for available recipes based on the ingredient information stored in the database, with priority given to recipes that contain ingredients with an approaching expiration date.

[1673] 9. Recipe suggestion methods

[1674] The recipe information obtained from the server is sent to the user's device, where the user can select the recipe they are interested in and check the detailed information.

[1675] 10. Scanning Method

[1676] Use your smartphone camera and OCR (optical character recognition) software (e.g., pytesseract) to scan receipts and automatically recognize ingredient information.

[1677] 11. Means of calculation

[1678] Use data analysis algorithms to calculate expiration dates from recognized ingredient information. Calculate detailed information about ingredients that are nearing their expiration date.

[1679] 12. Notification function

[1680] The notification function uses the Twilio API, Firebase Cloud Messaging, etc. to send expiration date notifications to users' devices, allowing them to take immediate action.

[1681] Specific examples

[1682] Suppose a user purchases "milk" and scans the receipt with a smartphone app. The app uses OCR technology to extract information about the "milk," parses / enters the purchase date and expiration date, and sends it to the server. The server stores this information in a database and periodically checks the expiration date. When the expiration date approaches, the server sends a notification to the user's smartphone saying, "Your milk is about to expire." At the same time, a recipe using milk (e.g., cream stew) is suggested, and the user can refer to the recipe to cook.

[1683] Example prompts for generative AI models

[1684] To create a food management application, we manage the expiration dates of ingredients purchased by the user and automatically register the ingredient information through receipt scanning and OCR analysis. It also includes a function to send notifications to a smartphone when ingredients are approaching their expiration date. For example, if a receipt says "Milk, Purchase Date 2023-10-05, Expiration Date 2023-10-20," the relevant information is registered in the application, and a notification is sent when the expiration date approaches within three days.

[1685] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1686] Step 1:

[1687] The user inputs the ingredient information.

[1688] Users use a smartphone app to input the name, purchase date, and expiration date of the ingredients they purchased, or scan a receipt to automatically recognize the ingredient information. The input data is the ingredient name, purchase date, and expiration date.

[1689] Step 2:

[1690] The terminal transmits the input ingredient information to the server.

[1691] The smartphone app sends the entered ingredient data to the server using an HTTP POST request. The device sends the ingredient name, purchase date, and expiration date as parameters. The output is a communication success or failure status.

[1692] Step 3:

[1693] The server stores the received ingredient information in a database.

[1694] The server parses the received JSON format data and registers the ingredient name, purchase date, and expiration date in the database. The input data is the ingredient information, and the output is the result saved in the database.

[1695] Step 4:

[1696] The server periodically checks the expiration dates in the database.

[1697] Run the script periodically (for example, every day at a set time) to list ingredients that are close to expiry. The input data is all the ingredients in the database, and the output is a list of ingredients that are close to expiry.

[1698] Step 5:

[1699] The server generates notification data about food ingredients that are close to their expiration date.

[1700] Based on a list of ingredients whose expiration date is approaching, notification data for each ingredient is generated. The notification data includes the ingredient name, expiration date, and warning message. The input data is a list of ingredients whose expiration date is approaching, and the output is the generated notification data.

[1701] Step 6:

[1702] The server transmits the generated notification data to the terminal.

[1703] The server sends the notification data to the user's smartphone as an email or push notification. The input data is the notification data, and the output is the transmission result.

[1704] Step 7:

[1705] The terminal displays the received notification data.

[1706] The user's smartphone displays the notification data in a pop-up or notification bar to notify the user that the expiration date is approaching. The input data is the notification data, and the output is the displayed notification message.

[1707] Step 8:

[1708] The server searches for recipes based on the ingredient information in the database.

[1709] The system prioritizes the search for recipes containing ingredients with upcoming expiration dates and suggests them to the user. The input data is information about ingredients with upcoming expiration dates, and the output is a list of searched recipes.

[1710] Step 9:

[1711] The server transmits the searched recipe information to the terminal.

[1712] The server sends the searched recipe list to the user's smartphone, where the user can check the details. The input data is the recipe list, and the output is the transmission result and displayed recipe information.

[1713] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1714] This invention is a system that efficiently manages the expiration dates of ingredients purchased by the user, suggests recipes to make effective use of remaining ingredients, and recognizes the user's emotions to make appropriate suggestions.By combining an emotion engine with the ingredient management function, this system can make suggestions based on the user's emotional state.

[1715] To implement this system, the following main components are required:

[1716] 1. Register ingredients

[1717] Device:

[1718] It provides an interface for users to input information about ingredients they have purchased (such as name, purchase date, and expiration date). As a specific example, a user purchases "milk" and enters information such as the name "Milk," purchase date "October 5, 2023," and expiration date "October 20, 2023" into the app. It is also possible to input this information automatically using the barcode scanning function.

[1719] 2. Sending and saving food information

[1720] Device:

[1721] It has a function to send the inputted ingredient information to the server using a protocol such as HTTP POST request.

[1722] Example: The information "milk" entered by the user is sent to the server.

[1723] server:

[1724] The received ingredient information is saved in a database. The server registers this information in the database in JSON format, SQL format, or other formats.

[1725] Example: The server stores the information about "milk" in a database.

[1726] 3. Best before date management and notification

[1727] server:

[1728] Regularly check the ingredient information in the database and list ingredients that are close to their expiration date. For example, run a script at a fixed time every day to list ingredients that will expire in the next three days.

[1729] Example: The server detects that the expiration date of "Milk" (October 20, 2023) is approaching.

[1730] server:

[1731] Generate notification data about food items that are nearing their expiration date and prepare to notify users. This can be done via email or push notification.

[1732] Example: The server generates a notification saying "The milk's expiration date is approaching" and sends it to the device.

[1733] Device:

[1734] Display the received notification data to the user, such as via a popup or notification bar.

[1735] Example: The device notifies the user that "The milk is nearing its expiration date."

[1736] 4. Recipe suggestions

[1737] server:

[1738] The system searches for available recipes based on ingredient information stored in the database, with priority given to recipes containing ingredients with an approaching expiration date.

[1739] Example: The server finds the recipe "Cream Stew" that uses "Milk".

[1740] Device:

[1741] The recipe information obtained from the server is displayed to the user, and multiple recipes are suggested so that the user can select the recipe that interests them.

[1742] Example: The terminal displays "Cream Stew Recipe" and the user can select it to view details.

[1743] 5. Operation of the Emotion Engine

[1744] Device:

[1745] It provides a function for recognizing the user's emotions in real time, using facial recognition via a camera and analysis based on the user's input behavior.

[1746] server:

[1747] The system receives user emotional data and optimizes recipe suggestions and notification content. For example, if the user is tired, it will suggest easy-to-make dishes.

[1748] Example: If the user is recognized as "tired," the server will suggest "easy-to-make tamagoyaki (rolled omelet)."

[1749] Device:

[1750] The emotional engine displays optimized recipes and notifications to users, providing optimal suggestions tailored to the emotional state the user is facing.

[1751] Example: The device will display "Easy recipe recommended for when you're tired: Tamagoyaki."

[1752] summary

[1753] The system of this invention not only effectively manages expiration dates of ingredients, but also enhances user convenience and satisfaction by suggesting and notifying recipes that take the user's emotional state into consideration. Various elements work together to reduce food waste and improve the user's quality of life.

[1754] The processing flow will be explained below.

[1755] Step 1:

[1756] The user inputs the ingredient information.

[1757] The user enters information about the ingredients they purchased using an application on their device. The information to be entered includes the name of the ingredient, the purchase date, and the expiration date. This information can also be entered automatically using the barcode scanning function. As a specific example, a user purchases "milk" and enters the name "milk," the purchase date "October 5, 2023," and the expiration date "October 20, 2023."

[1758] Step 2:

[1759] The terminal transmits the ingredient information to the server.

[1760] The device converts the entered ingredient information into JSON format and sends it as an HTTP POST request to the server's API endpoint. This request includes the ingredient name, purchase date, and expiration date. For example, the information for "milk" is sent to the server.

[1761] Step 3:

[1762] The server stores the ingredient information in a database.

[1763] The server parses the received request, extracts the ingredient information, and saves it in the database. The server returns a save completion response to the terminal. For example, the server saves the information for "milk" in the database.

[1764] Step 4:

[1765] The server checks the expiration date.

[1766] The server runs a scheduled task every day at a fixed time, checking the expiration date of each ingredient in the database. It compares this to the current date and lists ingredients that are close to expiring (for example, within 3 days). For example, the server detects that "Milk" is close to expiring.

[1767] Step 5:

[1768] The server generates the notification data.

[1769] The server generates notification data about ingredients whose expiration date is approaching and prepares to send it to the user. This notification data includes the ingredient name and expiration date. For example, a notification saying "Milk's expiration date is approaching" is generated.

[1770] Step 6:

[1771] The device displays the notification to the user.

[1772] The device receives notification data from the server and displays it in a pop-up or notification bar. The user can check which ingredients are nearing their expiration date. For example, the device may notify the user that "The expiration date of your milk is approaching."

[1773] Step 7:

[1774] The server searches for available recipes.

[1775] The server searches for available recipes based on the ingredient information stored in the database. It prioritizes recipes that contain ingredients with an approaching expiration date. For example, the server finds a recipe called "cream stew" that uses "milk."

[1776] Step 8:

[1777] The server transmits the recipe information to the terminal.

[1778] The server sends the searched recipe information to the device in JSON format. The recipe information includes the necessary ingredients and cooking instructions. For example, a "cream stew recipe" is sent to the device.

[1779] Step 9:

[1780] The device displays recipe suggestions to the user.

[1781] The device displays the recipe information received from the server and allows the user to select the recipe they are interested in. For example, the device displays "cream stew recipe" and the user can select it and view the details.

[1782] Step 10:

[1783] The device recognizes the user's emotions.

[1784] The device's camera and sensors are used to analyze the user's facial expressions and voice to obtain emotional data, detecting whether the user is "tired" or "stressed."

[1785] Step 11:

[1786] The server optimizes the recipe based on emotional data.

[1787] The server receives the user's emotional data and adjusts the recipe suggestions and notification content based on that data. For example, if the user is tired, it will prioritize suggestions of easy-to-make dishes. For example, if the user is tired, it will suggest "easy-to-make tamagoyaki" (rolled omelet).

[1788] Step 12:

[1789] The device displays recipes and notifications optimized by the emotion engine.

[1790] The device uses the emotion engine sent from the server to display optimized recipe information and notifications to the user. For example, it displays information such as "Easy recipe recommended for when you're tired: Tamagoyaki."

[1791] Through these steps, this system can provide users with a more personalized experience by combining food expiration date management and emotion recognition.

[1792] Example 2

[1793] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1794] Many conventional food management systems are specialized in managing and notifying expiration dates, and have the problem of not being able to make suggestions that take the user's emotions into consideration. As a result, the recipes provided may not be suitable for the user's situation, which can result in a decrease in user convenience and satisfaction. Another problem is that ingredients are often entered manually, which is a time-consuming and labor-intensive process.

[1795] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1796] In this invention, the server includes input means for a user to input food information, transmission means for transmitting the input food information to the server, storage means for saving the transmitted food information in a database, check means for periodically checking the expiration dates of the foods saved in the database, generation means for generating notification data for foods with upcoming expiration dates, notification means for sending the generated notification data to a terminal, display means for displaying the notification data to the user, search means for searching for recipes based on foods in the database, recipe suggestion means for sending the searched recipes to the terminal, emotion recognition means for recognizing the user's emotion, emotion data transmission means for sending emotion data recognized by the emotion recognition means to the server, and optimization means for optimizing recipe suggestions based on the emotion data. This not only enables food expiration date management but also makes it possible to suggest optimal recipes tailored to the user's emotions.

[1797] "Input means" refers to a device or interface that allows a user to input information about a food product.

[1798] "Transmission means" refers to a device or protocol that transmits the input food information to the server.

[1799] "Storage means" means a device or process that stores the submitted food information in a database.

[1800] A "checking means" is a device or script that periodically checks the expiration dates of food items stored in the database.

[1801] A "generator" is a device or algorithm that generates notification data about food products that are nearing their expiration date.

[1802] The "notification means" is a device or protocol that transmits the generated notification data to the terminal.

[1803] A "display means" is a device or interface that displays notification data to a user.

[1804] A "search tool" is a device or algorithm that searches for recipes based on foods in the database.

[1805] A "recipe suggestion means" is a device or protocol that sends searched recipes to a terminal.

[1806] An "emotion recognition means" is a device or algorithm that recognizes a user's emotions.

[1807] The "emotion data transmission means" is a device or protocol that transmits the emotion data recognized by the emotion recognition means to the server.

[1808] An "optimization means" is a device or algorithm that optimizes recipe suggestions based on emotion data.

[1809] This invention is a system that efficiently manages the expiration dates of food purchased by a user, suggests recipes for making effective use of remaining food, and recognizes the user's emotions to make appropriate suggestions. Specific embodiments of this invention are described below.

[1810] Hardware and Software Configuration

[1811] Input method: Provides an interface for users to input food information. Specifically, this corresponds to an application installed on a device such as a smartphone or tablet. Users can manually enter the food name, purchase date, and expiration date, or use a barcode scanning function to automatically enter them.

[1812] Transmission method: HTTP POST request is used as the protocol to send the entered food information to the server. For example, when a user enters information about "milk," this information is sent to the server in JSON format.

[1813] Storage method: The server stores the received food information in a database using SQL Server or NoSQL database, and the food information is stored in an appropriate format.

[1814] Check method: The server periodically runs a script to check the food information in the database. This process can be automated using a cron job or similar, and lists foods that have a shelf life of less than three days.

[1815] Generation method: Generates notification data for foods in the database that are close to their expiration date. The generated notification data is prepared in a format that notifies the user via email, push notification, etc.

[1816] Notification method: The generated notification data is sent to the device. This is done using a mail server or push notification service. The notification data is displayed in a pop-up or notification bar so that the user can check it immediately.

[1817] Display method: The device displays the notification data received from the server to the user. Specifically, the device uses the smartphone's notification bar or push notification to allow the user to quickly check the information.

[1818] Search method: The server searches for available recipes based on the food information stored in the database, with priority given to searching and suggesting recipes that include food items with an approaching expiration date.

[1819] Recipe suggestion method: The server sends the searched recipes to the terminal and suggests multiple recipes, allowing the user to select the recipe they are interested in and check the details.

[1820] Emotion recognition: The device provides a function to recognize the user's emotions in real time. Emotion recognition can be achieved by facial recognition via a camera or by analyzing the user's input behavior.

[1821] Emotion data transmission method: The recognized emotion data is transmitted from the device to the server. Based on this data, the server optimizes the next recipe suggestion.

[1822] Optimization method: The server receives the emotional data and suggests recipes according to the user's emotional state. For example, if the server detects that the user is "tired," it will prioritize suggestions for easy-to-make dishes.

[1823] Specific examples and prompts for the generative AI model

[1824] Examples:

[1825] It explains how the system notifies users when the milk they have purchased is nearing its expiration date and suggests suitable recipes.

[1826] Specific operation example: A user purchases "milk" and enters the name "milk," the purchase date "October 5, 2023," and the expiration date "October 20, 2023" into the device. The entered information is sent to the server and stored in a database. The server periodically checks the database and lists foods with an expiration date of three days or less. It then generates notification data stating "The milk's expiration date is approaching" and sends it to the user's device. When the user receives the notification, the server provides a recipe for "cream stew."

[1827] Example prompt for a generative AI model:

[1828] Explain how the system will notify the user when the "Milk" they have purchased is nearing its expiration date and suggest suitable recipes.

[1829] This invention not only enables the expiration date of food products to be managed, but also makes it possible to propose optimal recipes that take into account the user's feelings, thereby improving user convenience and satisfaction.

[1830] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1831] Specific processing steps of the system program

[1832] Step 1: User enters food information

[1833] The user inputs information about the food they have purchased into the terminal, such as the name of the food, the date of purchase, the expiration date, etc. This input is performed using a keyboard or a barcode scanner.

[1834] Input: food name, purchase date, expiration date

[1835] Output: Input data on the terminal

[1836] What it does: A user opens the app, enters "Milk," sets the purchase date to "2023-10-05," and the expiration date to "2023-10-20." By using the barcode scanner, this information is automatically entered.

[1837] Step 2: Device sends data to server

[1838] The terminal sends the food information entered by the user to the server. The data is sent via an HTTP POST request in JSON format.

[1839] Input: Food information entered by the user into the device

[1840] Output: JSON data sent to the server

[1841] Specific operation: The device sends JSON data (e.g., "{\"name\": \"Milk\", \"purchase_date\": \"2023-10-05\", \"expiry_date\": \"2023-10-20\"}") to the server using "POST / api / food_register".

[1842] Step 3: Data received and stored by the server

[1843] The server receives the food information sent from the device and stores it in a database using SQL queries.

[1844] Input: JSON data sent from the terminal

[1845] Output: Food information stored in the database

[1846] Specific operation: The server executes the SQL query "INSERT INTO food_table (name, purchase_date, expiration_date) VALUES (\'Milk\', \'2023-10-05\', \'2023-10-20\')" and saves it in the database.

[1847] Step 4: Expiry date check by server

[1848] The server runs a script that periodically checks the food information in the database. This script is run by a cron job at midnight every day and lists foods that have an expiration date within the next three days.

[1849] Input: Food information stored in a database

[1850] Output: A list of foods that are close to expiry date

[1851] Specific operation: The server executes the SQL query "SELECT FROM food_table WHERE best-before_date <= DATE_ADD(CURDATE(), INTERVAL 3 DAY)".

[1852] Step 5: Notification data generation by the server

[1853] The server generates notification data about food products that are nearing their expiration date, and the generated data is ready to send notifications to users.

[1854] Input: List of foods that are close to expiry date

[1855] Output: Generated notification data

[1856] Specific behavior: The server generates a notification message saying "The milk is nearing its expiration date."

[1857] Step 6: Send notification data to the device

[1858] The server then sends the generated notification data to the user's device, which can take the form of email or push notification.

[1859] Input: Notification data

[1860] Output: Notification sent to the device

[1861] Specific operation: The server sends the notification data via a push notification service or mail server.

[1862] Step 7: Displaying notifications on your device

[1863] The device displays the received notification data to the user in a pop-up or notification bar format.

[1864] Input: Notification data sent from the server

[1865] Output: The notification displayed to the user

[1866] Specific behavior: The device will display a pop-up notification saying "The milk's expiration date is approaching."

[1867] Step 8: Server Lookup for Recipes

[1868] The server searches for available recipes based on the food information stored in the database, with priority given to recipes that include food items with an approaching expiration date.

[1869] Input: Food information in the database

[1870] Output: Searched recipe list

[1871] Specific behavior: The server executes the SQL query "SELECT FROM recipe WHERE ingredients LIKE \"%milk%\"".

[1872] Step 9: Send the recipe to your device

[1873] The server sends the searched recipes to the terminal, and multiple recipes are suggested to the user.

[1874] Input: Searched recipe list

[1875] Output: Recipe information sent to the device

[1876] Specific operation: The server sends recipe information to the terminal and displays it to the user.

[1877] Step 10: Emotion recognition on the device

[1878] The device will provide features for recognizing users' emotions in real time, including camera-based facial recognition and behavioral analysis.

[1879] Input: User's facial expressions and behavior data

[1880] Output: Recognized emotion data

[1881] Specific operation: The device analyzes the user's face captured by the camera and recognizes emotions such as "tired."

[1882] Step 11: Send emotion data to the server

[1883] The device sends the recognized emotion data to the server, which then uses it to suggest the next recipe.

[1884] Input: Recognized emotion data

[1885] Output: Emotion data sent to the server

[1886] Specific operation: The device sends emotion data in JSON format to the server using "POST / api / emotion_data".

[1887] Step 12: Optimizing recipe suggestions by the server

[1888] The server optimizes recipe suggestions based on emotion data, and selects recipes that suit the user's state.

[1889] Input: Emotion data, recipe information in the database

[1890] Output: Optimized recipe suggestions

[1891] Specific operation: For users who are recognized by the server as being "tired," the server will prioritize suggesting easy-to-make recipes for "tamagoyaki" (rolled omelet).

[1892] (Application example 2)

[1893] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1894] In modern consumer lifestyles, it is difficult for users to efficiently manage the expiration dates of purchased ingredients, resulting in food waste. It is also not easy for users to select the optimal recipe based on their emotional state at any given time. At the same time, there is a demand for systems that can suggest appropriate recipes based on emotions, but no such systems have existed to date. This situation has led to problems that reduce user convenience and satisfaction.

[1895] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1896] In this invention, the server includes emotion recognition means for recognizing the emotional state of the user, recipe optimization means for optimizing recipes based on the recognized emotional state, and generation means for generating notification data for ingredients whose expiration dates are approaching, thereby enabling the server to efficiently manage the expiration dates of ingredients purchased by the user and propose optimal recipes according to the user's emotional state.

[1897] The "input means" is an interface through which the user inputs information about ingredients.

[1898] The "transmission means" is a function that transmits the input information about ingredients to the server.

[1899] The "storage means" is a function that stores the transmitted information about ingredients in a database.

[1900] The "checking means" is a function that periodically checks the expiration dates of ingredients stored in the database.

[1901] The "generation means" is a function that generates notification data about food ingredients that are close to their expiration date.

[1902] The "notification means" is a function that transmits the generated notification data to the terminal.

[1903] The "display means" is a function that displays the notification data to the user.

[1904] "Search means" is a function that searches for recipes based on ingredients in the database.

[1905] The "recipe suggestion means" is a function that sends the searched recipe to the terminal.

[1906] The "emotion recognition means" is a function for recognizing the user's emotional state.

[1907] The "recipe optimizer" is a function that optimizes recipes based on the recognized emotional state.

[1908] This invention is a system that efficiently manages the expiration dates of ingredients purchased by a user and suggests optimal recipes based on the emotional state of the user. Specific embodiments for carrying out this invention are described below.

[1909] 1. Register ingredients

[1910] Program Generation

[1911] We provide a smartphone application that allows users to scan the barcode of ingredients to register information about the ingredients they have purchased (such as name, purchase date, expiration date, etc.). When a user scans the barcode, the corresponding ingredient information is automatically entered into the application.

[1912] Processing Description

[1913] In this step, the smartphone uses the barcode scanning function to obtain information about ingredients and sends the entered data to the server, which then stores the data in a database. The specific hardware used is the smartphone, and the software is a barcode scanning module.

[1914] 2. Best before date management and notification

[1915] Program Generation

[1916] A script runs on the server every day, checking the food ingredient information in the database and listing ingredients that are close to their expiration date. Notification data is generated for the listed ingredients and sent to the user via email or push notification.

[1917] Processing Description

[1918] In this step, the server checks expiration dates using Python scripts or scheduled scripts. For ingredients with an approaching expiration date, it generates notification data and sends it to smartphones or other devices. The specific software used is a scripting language (e.g., Python) and a notification service (e.g., Firebase Cloud Messaging).

[1919] 3. Recipe suggestions

[1920] Program Generation

[1921] The server searches for available recipes based on the ingredient information stored in the database, giving priority to recipes that contain ingredients with an approaching expiration date, and sends the results to the user's device to suggest recipes.

[1922] Processing Description

[1923] The server searches for recipe data based on the user's ingredient data. The recipe results are sent to the user's device, where the recipe details are displayed. The server executes a database query to search for recipes and returns the results in JSON format.

[1924] 4. Emotion Recognition and Recipe Optimization

[1925] Program Generation

[1926] We provide an emotion recognition engine that uses facial recognition and voice analysis via a camera to recognize the user's emotional state, and the emotion engine optimizes the recipe suggestions based on the recognized emotional state.

[1927] Processing Description

[1928] The system analyzes the user's emotional state using the camera and microphone on the smartphone or head-mounted display (HMD). The analyzed emotional data is sent to the server, which then provides recipes based on the user's emotional state. The specific software used is an emotion recognition library (e.g., OpenCV, Affectiva).

[1929] 5. Specific Examples

[1930] For example, if a user scans "Milk," the data is automatically entered into the app and saved in a database. When the expiration date approaches, a notification stating "Milk expiration date is approaching" is sent to the smartphone. If the emotion recognition engine detects that the user is "tired," the server will suggest "easy-to-make tamagoyaki (rolled omelet)."

[1931] 6. Examples of prompts

[1932] "I've entered the ingredients I purchased (milk, eggs, cabbage) and their expiration dates. I'm feeling tired right now and would like to know what easy meals I can make. Please tell me the best recipe based on this data and my mood."

[1933] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1934] Step 1:

[1935] Entering ingredient information

[1936] The user inputs the ingredients they purchased using the barcode scanning function of their smartphone. The smartphone reads the barcode and obtains information such as the ingredient name, purchase date, and expiration date. The barcode data is read as input, and the ingredient information is automatically entered as output.

[1937] Step 2:

[1938] Sending ingredient information

[1939] The smartphone sends the entered ingredient information to the server via an HTTP POST request. The smartphone receives ingredient information as input and sends the information as output to the server in JSON format.

[1940] Step 3:

[1941] Saving food information

[1942] The server stores the received ingredient information in a database. It analyzes the ingredient information received as input and registers the data using SQL queries etc. to store it in the database. As output, records stored in the database are generated.

[1943] Step 4:

[1944] Regular check of expiration dates

[1945] The server periodically checks the ingredient information in the database using a script and lists ingredients with an approaching expiration date. It compares the current date with the ingredient information in the database as input and extracts ingredients with an upcoming expiration date. The output is a list of ingredients with an upcoming expiration date.

[1946] Step 5:

[1947] Generate notification data

[1948] The server generates notification data about ingredients with upcoming expiration dates. It receives as input a list of ingredients with upcoming expiration dates and generates a notification message to be sent to the user. The notification message is generated as output.

[1949] Step 6:

[1950] Sending notification data

[1951] The server sends the generated notification data to the smartphone. It takes the generated notification message as input and sends it to the smartphone using a notification protocol (e.g., Firebase Cloud Messaging). It generates a notification to be sent to the smartphone as output.

[1952] Step 7:

[1953] Ingredient-based recipe search

[1954] The server searches for recipes based on ingredient information in the database, with priority given to recipes containing ingredients with an approaching expiration date. It receives ingredient information as input and searches the database for matching recipes. A list of recipes is generated as output.

[1955] Step 8:

[1956] Sending recipe information

[1957] The server sends the searched recipe information to the smartphone. It receives a list of recipes as input and transmits the data using a protocol for sending to the smartphone app. The recipe data that is sent to the smartphone is generated as output.

[1958] Step 9:

[1959] Viewing Recipes

[1960] The smartphone displays the received recipe information to the user. Using the received recipe data as input, the recipe information is displayed on the app's UI so that it can be seen by the user. As output, a recipe screen is generated that the user can view.

[1961] Step 10:

[1962] emotion recognition

[1963] Smartphones and head-mounted displays use cameras and microphones to recognize the user's emotional state. Camera images and audio data are analyzed as input, and the user's emotional state is analyzed. Recognized emotional data is generated as output.

[1964] Step 11:

[1965] Emotion-based recipe optimization

[1966] The server optimizes the recipe suggestions based on the recognized emotional state. It receives emotional data as input and selects recipes that match that state. It generates an optimized recipe list as output.

[1967] Step 12:

[1968] Displaying recipes based on emotions

[1969] The smartphone displays the optimized recipe information to the user. It receives the optimized recipe data as input and displays the recipe information visible to the user on the app's UI. As output, an optimized recipe screen is generated that the user can view.

[1970] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1971] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1972] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1973] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1974] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1975] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1976] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1977] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1978] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1979] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1980] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1981] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1982] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1983] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1984] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1985] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1986] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1987] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific...

Claims

1. an input means for a user to input information about ingredients; a transmitting means for transmitting the input information on ingredients to a server; a storage means for storing the transmitted information on ingredients in a database; A checking means for periodically checking the expiration dates of ingredients stored in the database; A generating means for generating notification data about food ingredients whose expiration date is approaching; a notification means for transmitting the generated notification data to the terminal; a display means for displaying the notification data to a user; A search method to search for recipes based on ingredients in the database; a recipe suggestion means for sending the searched recipe to a terminal; A system including:

2. 2. The system according to claim 1, wherein information about ingredients is automatically input by scanning a barcode.

3. The system of claim 1 , wherein the notification data is sent to the user as an email or a push notification.

Images