System

Abstract

A system is provided.SOLUTION: This system includes a means for photographing the best-before date of food and storing it as data, a means for automatically notifying that the best-before date approaches, a means for proposing a food storage method, and a means for providing a recipe obtained by combining foods whose best-before dates are close to each other.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's busy lifestyles, people often forget to track food expiration dates. This results in food waste and disposal, resulting in economic losses and increased environmental impact. In addition, improper storage methods can accelerate food deterioration. Furthermore, a lack of recipe suggestions for efficiently using food approaching its expiration date often leads to food waste. The present invention aims to solve these problems and provide a system for managing food expiration dates, suggesting storage methods, and promoting efficient use. [Means for solving the problem]

[0005] The present invention solves the above problems by the following means.

[0006] 1. A method for photographing food expiration dates and saving them as data. Users use their smartphone camera to photograph the expiration date of food products, and the image is sent to a server where date information is extracted using OCR technology. The extracted date information is then stored in a database.

[0007] 2. Automatically notify users when their expiration date is approaching. The server periodically checks the expiration date information in the database and identifies foods that are approaching their expiration date. If any are identified, the server generates a notification message and sends it to the user via a push notification on their device.

[0008] 3. As a means of suggesting food preservation methods. For food that is nearing its expiration date, information on the optimal preservation method is retrieved from the database and the user is notified of the preservation method based on that information.

[0009] 4. As a means to provide recipes that combine foods with close expiration dates, the server searches for recipes in the database based on foods with close expiration dates, selects appropriate recipes, and notifies the user.

[0010] This makes it easier for users to manage food expiration dates, reducing food waste and disposal, and helping them use food more efficiently through suggestions for appropriate storage methods and recipes.

[0011] The "best before" date refers to the date by which food can maintain its flavor and quality, and is the date by which consumers can safely and deliciously eat the food.

[0012] "Photography" refers to the act of recording an object or information as an image using a camera.

[0013] "Data" refers to codes, numbers, etc. used to organize, record, and process information, and refers to anything that contains specific information.

[0014] "Preservation" means to maintain information or objects and keep them so that they are available when needed.

[0015] "Notification" refers to the act of presenting specific information and calling attention to it, and is a means of informing the user of important information.

[0016] "Means" refers to the methods or techniques used to achieve a particular goal.

[0017] "User" refers to any individual or organization that uses this system.

[0018] A "server" refers to a computer system that provides services to other devices over a network.

[0019] "Terminal" refers to a device that is directly operated by a user, including smartphones and tablets.

[0020] "Image analysis" refers to the technology of extracting and understanding information from images taken by a camera.

[0021] "OCR technology" refers to Optical Character Recognition technology, which reads characters from an image and converts them into digital data.

[0022] A "database" refers to a system that organizes and stores large amounts of information and allows for efficient searching and updating.

[0023] "Preservation method" refers to the techniques and technologies used to keep food in an appropriate state.

[0024] A "recipe" is a detailed set of instructions on how to prepare a particular dish or combination of ingredients. [Brief explanation of the drawings]

[0025] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2]1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0026] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0027] First, the terms used in the following description will be explained.

[0028] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0029] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0030] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0031] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0032] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0033] [First embodiment]

[0034] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0035] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0036] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0037] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0038] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0039] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0040] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0041] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0042] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0043] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0044] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0045] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0046] The present invention relates to a system for managing expiration dates of food products and proposing appropriate storage methods and recipes. Specific embodiments of the system will be described below.

[0047] 1. Acquisition and storage of expiration date data

[0048] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[0049] Specific examples

[0050] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[0051] 2. Best before date tracking and notification

[0052] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[0053] Specific examples

[0054] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[0055] 3. Storage method suggestions

[0056] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[0057] Specific examples

[0058] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[0059] 4. Recipe suggestions

[0060] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[0061] Specific examples

[0062] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[0063] As described above, this system aims to efficiently manage food expiration dates and reduce food waste by suggesting storage methods and recipes, allowing users to maintain food quality while consuming it and leading an economical and environmentally conscious lifestyle.

[0064] The processing flow will be explained below.

[0065] Acquiring and storing expiration date data

[0066] Step 1:

[0067] The user takes a photo of the expiration date of the food they have purchased using their smartphone camera.

[0068] Step 2:

[0069] The terminal transmits the captured image to the server.

[0070] Step 3:

[0071] The server uses image analysis technology (OCR technology) to extract the expiration date information from the image.

[0072] Step 4:

[0073] The server stores the extracted expiration date information in a database.

[0074] Best before date tracking and notification

[0075] Step 1:

[0076] The server periodically checks the expiration date information in the database.

[0077] Step 2:

[0078] The server identifies food items that are nearing their expiration date (e.g., 3 days away).

[0079] Step 3:

[0080] The server generates notification messages corresponding to approaching expiration dates.

[0081] Step 4:

[0082] The terminal receives the notification message from the server and sends a push notification to the user.

[0083] Storage suggestions

[0084] Step 1:

[0085] The server retrieves from the database the storage method for food items that are nearing their expiration date.

[0086] Step 2:

[0087] The server adds information about the storage method to the notification message.

[0088] Step 3:

[0089] The terminal displays a notification message to the user, including instructions on how to save.

[0090] Recipe Suggestions

[0091] Step 1:

[0092] The server searches the database for recipe information based on foods that are close to their expiration date.

[0093] Step 2:

[0094] The server selects an appropriate recipe based on the recipe information.

[0095] Step 3:

[0096] The server generates a notification message regarding the selected recipe.

[0097] Step 4:

[0098] The terminal displays a notification message to the user that includes the selected recipe.

[0099] Example 1

[0100] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0101] In modern society, many consumers waste food. In particular, discarding food that has passed its expiration date causes economic losses and negative impacts on the environment. To solve this problem, a system is needed that can efficiently manage food expiration dates, notify consumers at the appropriate time, and even suggest storage methods and recipes.

[0102] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0103] In this invention, the server includes means for photographing expiration dates of food products with a device camera and sending the image to the server, means for the server to extract date information using image analysis technology and store it in a database, means for the server to periodically check the expiration date information in the database and, when an expiration date is approaching, generate a notification message and send a push notification to the user via the device, means for the server to suggest appropriate storage methods for food products approaching their expiration dates, and means for the server to provide recipes that combine food products approaching their expiration dates. This makes it easier for users to manage expiration dates, reduces food waste, and enables them to live economically and environmentally conscious lives.

[0104] The "device camera" refers to a photographing device installed in a portable electronic device such as a smartphone or tablet that a user uses to photograph the expiration date of food products.

[0105] A "server" is a computer device that uses image analysis technology to extract date information from captured image data, stores it in a database, and periodically checks the information in the database and sends notification messages to users.

[0106] "Image analysis technology" is a technology that extracts text information from image data, including food expiration dates, and generally refers to OCR (Optical Character Recognition) technology.

[0107] A "database" is a structured data storage system for storing information about expiration dates and food products, which is accessed and updated by a server.

[0108] A "notification message" is text information that is generated by the server to inform the user of food products that are approaching their expiration date and is sent to the user via the terminal.

[0109] "Push notification" is a technology for displaying notification messages on a user's device in real time, and is a means of displaying messages on the device's home screen or notification bar.

[0110] "Storage method" refers to the recommended storage method or procedure for extending the expiration date while maintaining the quality of food, and the server searches the database and suggests it to the user.

[0111] A "recipe" is information that indicates steps and combinations of ingredients for creating a dish that efficiently uses food that is approaching its expiration date.

[0112] "Food" refers to ingredients or cooked products that are purchased and consumed by users and have a best-before date.

[0113] The present invention relates to a system that manages the expiration dates of food products and suggests appropriate storage methods and recipes. This system takes a photo of the expiration date using a device camera, performs image analysis on a server, and provides notifications and suggestions based on the expiration date information stored in a database. Specific embodiments are described below.

[0114] 1. Acquisition and storage of expiration date data

[0115] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses OCR technology (e.g., Tesseract OCR) to analyze and extract the expiration date information from the image. This extracted date information is then stored in a database along with the food's information. This saves users the trouble of manually entering the expiration date.

[0116] Examples:

[0117] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[0118] 2. Best before date tracking and notification

[0119] The server periodically checks the expiration date information in the database, identifies foods that are approaching their expiration date, and if so, generates a notification message and sends a push notification to the user via their device, allowing the user to consume the food at the appropriate time without forgetting the expiration date.

[0120] Examples:

[0121] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[0122] 3. Storage method suggestions

[0123] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[0124] Examples:

[0125] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[0126] 4. Recipe suggestions

[0127] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[0128] Examples:

[0129] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[0130] The specific processing flow of this system can be explained by inputting the following prompt sentence into the generative AI model.

[0131] "Please tell me the specific process flow for a system that manages food expiration dates and suggests appropriate storage methods and recipes. Please provide a detailed explanation of each process step, including specific operations."

[0132] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0133] Step 1:

[0134] The user takes a photo of the expiration date of the food product they purchased with their smartphone camera. For example, the user launches the smartphone's camera app and takes a photo of the milk they purchased so that the expiration date "2023-10-15" is displayed on the screen.

[0135] Input: An image of the expiration date taken with a smartphone camera

[0136] Output: Image data of expiration dates stored on a smartphone

[0137] Step 2:

[0138] The device sends the captured images to a server. Specifically, the device uses an Internet connection to upload image data to the server via the HTTPS protocol.

[0139] Input: Image data of expiration date stored on smartphone

[0140] Output: Image data sent to the server

[0141] Step 3:

[0142] The server analyzes the received image using OCR technology (e.g., Tesseract OCR) to extract the expiration date information. The server then runs its internal algorithms to analyze the image and obtain the text data.

[0143] Input: Image data sent to the server

[0144] Output: Extracted best-before date information (e.g. "2023-10-15")

[0145] Step 4:

[0146] The server stores the extracted date information along with the food information in a database. The server inserts the food information (e.g., "Milk", "2023-10-15", "2023-10-01") into the appropriate fields in the database.

[0147] Input: Extracted best-before date information and food information

[0148] Output: Best before date information stored in the database

[0149] Step 5:

[0150] The server periodically checks the expiration date information in the database, for example, as a scheduled job once a day.

[0151] Input: Best before date information in the database

[0152] Output: Checked expiration date information

[0153] Step 6:

[0154] The server identifies foods that are nearing their expiration date and generates a notification message. For example, if the server determines that there are only three days until 2023-10-15 as of 2023-10-12, it generates a message saying, "Milk's expiration date is in three days."

[0155] Input: Checked expiration date information and current date

[0156] Output: The generated notification message

[0157] Step 7:

[0158] The server sends the generated notification message to the device. The device receives the push notification and displays it to the user. For example, a notification saying "The milk's expiration date is in three days" will be displayed on the smartphone screen.

[0159] Input: The generated notification message

[0160] Output: Push notification displayed on device

[0161] Step 8:

[0162] The server retrieves the optimal storage method for food that is nearing its expiration date from the database, searches for the storage method data for the relevant food, and generates a suggestion message.

[0163] Input: Information about food that is close to its expiration date

[0164] Output: Generated save method suggestion message

[0165] Step 9:

[0166] The server sends the generated storage suggestion message to the device, which then displays it to the user via push notification. For example, a notification such as "The milk's expiration date is in three days. Please place it at the back of the refrigerator to keep it cool" may be displayed.

[0167] Input: Generated save method suggestion message

[0168] Output: Push notification with suggested saving methods displayed on the device

[0169] Step 10:

[0170] The server searches the database for recipes that combine food items whose expiration dates are approaching, retrieves the corresponding recipe data, and generates a recipe suggestion message.

[0171] Input: Information about food that is close to its expiration date

[0172] Output: Generated recipe suggestion message

[0173] Step 11:

[0174] The server sends the generated recipe suggestion message to the device, which then displays it to the user via a push notification. For example, a notification such as "Click here for a recipe for cream stew using milk" is displayed.

[0175] Input: Generated recipe suggestion message

[0176] Output: Push notification with recipe suggestions displayed on the device

[0177] (Application example 1)

[0178] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0179] The goal is to provide a system that manages food expiration dates and suggests appropriate storage methods and recipes, allowing users to easily manage expiration dates, suggest storage methods, and search for recipes. In particular, there is a need for a means to automate information input and send notifications at appropriate times. There is also a desire to reduce food waste by suggesting recipes and notifying users of storage methods to efficiently use food that is approaching its expiration date.

[0180] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0181] In this invention, the server includes a means for automatically extracting expiration date information for food products using image analysis technology and managing the data, a means for sending notifications to users using push notification technology when the expiration date approaches, and a means for displaying notifications on a user interface with storage methods and recipes. This not only enables users to efficiently manage expiration dates, but also automatically receives suggestions for appropriate storage methods and recipes that combine foods with approaching expiration dates. Furthermore, the use of notification technology allows users to obtain important information at the appropriate time, thereby reducing food waste.

[0182] "Means for photographing food expiration dates and saving the data" is a function that allows users to use a device such as a smartphone to photograph the expiration date displayed on food and save the data.

[0183] The "means for automatically notifying when the expiration date is approaching" is a function that automatically sends a warning or alert to the user when the expiration date is approaching based on stored expiration date data.

[0184] The "means for suggesting food preservation methods" is a function for suggesting to the user how to properly preserve food that is approaching its expiration date.

[0185] The "means for providing recipes that combine food items with close expiration dates" is a function for providing users with recipes that use multiple food items with close expiration dates.

[0186] "Means for automatically extracting expiration date information using image analysis technology and managing the data" refers to a function that converts photographed expiration date information into text data using image analysis technology such as OCR technology and manages the data.

[0187] "Means for sending notifications to users using push notification technology" refers to a function that uses cloud services, etc. to notify users' devices in real time that the expiration date is approaching.

[0188] The "means for displaying notifications on a user interface" is a function for displaying storage methods and recipe suggestions on the interface on the user's device.

[0189] The present invention relates to a system for managing expiration dates of food products and proposing appropriate storage methods and recipes. Specific embodiments of the system will be described below.

[0190] First, when a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[0191] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[0192] Furthermore, for food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[0193] The server also searches the database for recipe information to suggest recipes that combine foods that are approaching their expiration dates. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[0194] Specifically, the hardware used includes smartphones and servers, and the software used includes Tesseract OCR, which implements OCR technology, and Firebase Cloud Messaging, which provides push notification technology.

[0195] For example, if a user purchases milk and takes a photo of its expiration date, "2023-10-15," with their smartphone, the device sends this image to a server. The server uses OCR technology to extract the expiration date, "2023-10-15," from the image and stores it in a database. If the server checks the milk's expiration date, "2023-10-15," in the database and determines that the current date is "2023-10-12," the server generates a notification message stating, "The milk's expiration date is three days from now," and sends this push notification to the user via their device. It also generates a notification message suggesting a storage method, "Put it at the back of the refrigerator to cool," and a recipe, "Cream stew."

[0196] An example of a prompt sentence for the generative AI model is written as follows:

[0197] Extract the expiration date from the image below:

[0198] Image path: path_to_image.jpg

[0199] If the expiration date is "2023-10-15", generate a notification message suggesting the best way to store the food, "Put it at the back of the refrigerator to cool it down", and also suggest the recipe "Cream Stew".

[0200] As described above, this system aims to efficiently manage food expiration dates and reduce food waste by suggesting storage methods and recipes, allowing users to maintain food quality while consuming it and leading an economical and environmentally conscious lifestyle.

[0201] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0202] Step 1:

[0203] A user takes a photo of a food product's expiration date with a smartphone camera. The input is an image of the expiration date printed on the food's packaging. The output is the captured image data.

[0204] Step 2:

[0205] The device sends the captured image to the server. The input is the image data captured by the smartphone. The output is the image data transferred to the server.

[0206] Step 3:

[0207] The server uses OCR technology to extract the expiration date information from the image. The input is the received image data. The data processing is image analysis using Tesseract OCR. The output is the extracted expiration date string.

[0208] Step 4:

[0209] The extracted date information is saved in a database. The input is the date string obtained by OCR analysis. The data processing is writing the date string to the database. The output is a database entry with the expiration date saved.

[0210] Step 5:

[0211] The server periodically checks the expiration date information in the database. The input is the expiration date information of all food items stored in the database. The data calculation is a comparison with the current date. The output is a list of food items that are approaching their expiration date.

[0212] Step 6:

[0213] Identify foods approaching their expiration date and generate a notification message. The input is a list of foods approaching their expiration date. The data processing is the generation of the notification message. The output is the generated notification message.

[0214] Step 7:

[0215] The server sends the generated notification message to the device via Firebase Cloud Messaging. The input is the generated notification message. The output is the notification sent to the user's smartphone.

[0216] Step 8:

[0217] The device notifies the user via a push notification. The input is the notification message received from the server. The output is the push notification displayed to the user.

[0218] Step 9:

[0219] For foods approaching their expiration date, the server retrieves the optimal storage method from the database. The input is a list of foods approaching their expiration date. The data operation is a query from the storage method database. The output is the retrieved storage method.

[0220] Step 10:

[0221] A notification message is generated based on the storage method and notified to the user via the terminal. The input is the storage method. Data processing is the generation of the storage method notification message. The output is the storage method notification message.

[0222] Step 11:

[0223] To suggest recipes that combine foods with close expiration dates, the server searches for recipe information in a database. The input is a list of foods with close expiration dates. The data operation is a query from the recipe database. The output is a suitable recipe.

[0224] Step 12:

[0225] Select the appropriate recipe and generate a notification message. The input is the retrieved recipe information. The data processing is the generation of the recipe notification message. The output is the generated recipe notification message.

[0226] Step 13:

[0227] The terminal displays the generated recipe notification message to the user. The input is the recipe notification message from the server. The output is the recipe suggestion notification displayed to the user.

[0228] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0229] The present invention combines an emotion engine with a system that manages food expiration dates and suggests appropriate storage methods and recipes, providing a service based on the user's emotions. Specific embodiments of this system are described below.

[0230] 1. Acquisition and storage of expiration date data

[0231] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[0232] Specific examples

[0233] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[0234] 2. Best before date tracking and notification

[0235] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[0236] Specific examples

[0237] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[0238] 3. Storage method suggestions

[0239] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[0240] Specific examples

[0241] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[0242] 4. Recipe suggestions

[0243] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[0244] Specific examples

[0245] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[0246] 5. Providing services using emotion engines

[0247] 5.1 Emotion Recognition and Recipe Adjustment

[0248] The server uses an emotion engine to recognize the user's emotional state and adjusts the content of recipes and notifications depending on the user's emotions. For example, if the user is tired, it will suggest easy-to-make recipes.

[0249] Specific examples

[0250] The server uses an emotion engine to recognize that the user is tired, and then suggests a recipe for "stir-fried vegetables" as an "easy recipe recommended for when you're tired" and notifies the user via their device.

[0251] 5.2 Recipe suggestions based on tastes and preferences

[0252] The emotion engine determines the user's tastes and preferences and suggests recipes based on them, making it easier for users to cook their favorite dishes.

[0253] Specific examples

[0254] The emotion engine analyzes the user's preferences based on past data, and the server suggests a recipe for "Japanese-style pasta" as "a dish with the user's favorite seasoning." The device notifies the user of this information.

[0255] 5.3 Food suggestions for stress reduction

[0256] The emotion engine assesses the user's stress level and suggests foods and recipes that have a stress-reducing effect.

[0257] Specific examples

[0258] The emotion engine recognizes that the user is feeling stressed, and the server suggests a "relaxing herbal tea recipe." The device notifies the user of this information.

[0259] As described above, this system aims not only to efficiently manage food expiration dates and reduce food waste by suggesting storage methods and recipes, but also to provide services that respond to the user's emotional state using an emotion engine, thereby enabling users to live more comfortable and efficient lives.

[0260] The processing flow will be explained below.

[0261] Acquiring and storing expiration date data

[0262] Step 1:

[0263] The user takes a photo of the expiration date of the food they have purchased using their smartphone camera.

[0264] Step 2:

[0265] The terminal transmits the captured image to the server.

[0266] Step 3:

[0267] The server uses image analysis technology (OCR technology) to extract the expiration date information from the image.

[0268] Step 4:

[0269] The server stores the extracted expiration date information in a database.

[0270] Best before date tracking and notification

[0271] Step 1:

[0272] The server periodically checks the expiration date information in the database.

[0273] Step 2:

[0274] The server identifies food items that are nearing their expiration date (e.g., 3 days away).

[0275] Step 3:

[0276] The server generates notification messages corresponding to approaching expiration dates.

[0277] Step 4:

[0278] The terminal receives the notification message from the server and sends a push notification to the user.

[0279] Storage suggestions

[0280] Step 1:

[0281] The server retrieves from the database the storage method for food that is nearing its expiration date.

[0282] Step 2:

[0283] The server adds information about the storage method to the notification message.

[0284] Step 3:

[0285] The terminal displays a notification message to the user including the saving method.

[0286] Recipe Suggestions

[0287] Step 1:

[0288] The server searches the database for recipe information based on foods with an approaching expiration date.

[0289] Step 2:

[0290] The server selects an appropriate recipe based on the recipe information.

[0291] Step 3:

[0292] The server generates a notification message about the selected recipe.

[0293] Step 4:

[0294] The terminal displays a notification message to the user containing the selected recipe.

[0295] Providing services using emotion engines

[0296] Emotion recognition and recipe adjustment

[0297] Step 1:

[0298] The server uses an emotion engine to recognize the user's emotional state.

[0299] Step 2:

[0300] The server selects recipes and notification messages according to the user's emotions.

[0301] Step 3:

[0302] The server generates the selected recipe and notification message.

[0303] Step 4:

[0304] The device displays recipes and notification messages to the user based on their emotions.

[0305] Recipe suggestions based on your tastes and preferences

[0306] Step 1:

[0307] The server uses an emotion engine to determine the user's preferences and tastes.

[0308] Step 2:

[0309] The server retrieves recipes based on the user's preferences.

[0310] Step 3:

[0311] The server selects recipes based on preferences and generates notification messages.

[0312] Step 4:

[0313] The device displays a notification message to the user containing a recipe based on the preferences.

[0314] Food suggestions for stress reduction

[0315] Step 1:

[0316] The server uses an emotion engine to determine whether the user is experiencing stress.

[0317] Step 2:

[0318] The server searches for foods and recipes that have stress-reducing effects.

[0319] Step 3:

[0320] The server selects foods and recipes for stress reduction and generates a notification message.

[0321] Step 4:

[0322] The device will display a notification message to the user containing stress-reducing foods and recipes.

[0323] Example 2

[0324] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0325] Current food management systems require users to manually input expiration dates, making expiration date management cumbersome. Furthermore, storage methods and recipe suggestions are uniform and are not based on the user's emotional state or preferences, resulting in low user convenience. Furthermore, if expiration date notifications are not properly provided, there is a high possibility of food waste.

[0326] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for photographing the expiration date of food and saving it as data, a means for automatically notifying when the expiration date is approaching, a means for suggesting a food preservation method, a means for providing a recipe that combines foods with close expiration dates, and a means for recognizing the user's emotional state and adjusting the notification and suggestion content based on the emotional state. This enables efficient management of expiration dates, suggestion of appropriate preservation methods, and suggestion of recipes that correspond to the user's emotions.

[0327] "Means for photographing expiration dates and saving them as data" refers to a means by which a user photographs the expiration date of a food product using a smartphone camera, sends the image to a server, and uses image analysis technology (OCR technology) to extract the date information and save it in a database.

[0328] "Means for automatically notifying when the expiration date is approaching" refers to a means in which the server periodically checks the expiration date information in the database, generates a notification message when the expiration date is approaching, and pushes the message to the user via the terminal.

[0329] The "means for suggesting food preservation methods" refers to a means by which the server retrieves the optimal preservation method from a database for food that is approaching its expiration date, generates a notification message based on that information, and notifies the user via the terminal.

[0330] The "means for providing recipes that combine foods with close expiration dates" refers to a means by which the server searches a database for recipe information based on foods with close expiration dates, selects an appropriate recipe, generates a notification message, and displays it to the user via the terminal.

[0331] "Means for recognizing the user's emotional state and adjusting notifications and suggestions based on that" refers to a means by which the server recognizes the user's emotional state using an emotion engine and adjusts notifications and suggestions based on the user's emotions. Specifically, this includes adjustments such as suggesting easy-to-make recipes when the user is tired.

[0332] The present invention combines an emotion engine with a system that manages food expiration dates and suggests appropriate storage methods and recipes, providing a service based on the user's emotions. Specific embodiments of this system are described below.

[0333] Acquiring and storing expiration date data

[0334] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[0335] Specific examples

[0336] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[0337] Best before date tracking and notification

[0338] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[0339] Specific examples

[0340] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[0341] Storage suggestions

[0342] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[0343] Specific examples

[0344] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[0345] Recipe Suggestions

[0346] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[0347] Specific examples

[0348] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[0349] Providing services using emotion engines

[0350] Emotion recognition and recipe adjustment

[0351] The server uses an emotion engine to recognize the user's emotional state and adjusts the content of recipes and notifications depending on the user's emotions. For example, if the user is tired, it will suggest easy-to-make recipes.

[0352] Specific examples

[0353] The server uses an emotion engine to recognize that the user is tired, and then suggests a recipe for "stir-fried vegetables" as an "easy recipe recommended for when you're tired" and notifies the user via their device.

[0354] Recipe suggestions based on your tastes and preferences

[0355] The emotion engine determines the user's tastes and preferences and suggests recipes based on them, making it easier for users to cook their favorite dishes.

[0356] Specific examples

[0357] The emotion engine analyzes the user's preferences based on past data, and the server suggests a recipe for "Japanese-style pasta" as "a dish with the user's favorite seasoning." The device notifies the user of this information.

[0358] Food suggestions for stress reduction

[0359] The emotion engine assesses the user's stress level and suggests foods and recipes that have a stress-reducing effect.

[0360] Specific examples

[0361] The emotion engine recognizes that the user is feeling stressed, and the server suggests a "relaxing herbal tea recipe." The device notifies the user of this information.

[0362] Prompt Sentence Examples

[0363] "Please suggest recipes that efficiently use food in the refrigerator that is nearing its expiration date."

[0364] "Please suggest easy dinner recipes for when the user is tired."

[0365] "Provide new recipes based on your users' favorite flavors."

[0366] With this system, users can efficiently manage food expiration dates and receive suggestions for storage methods and recipes. Furthermore, by using an emotion engine, it is possible to provide services that correspond to the user's emotional state, improving the quality of daily life.

[0367] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0368] Step 1:

[0369] A user takes a photo of the expiration date on a food item

[0370] explanation:

[0371] The user uses the smartphone camera to take a photo of the expiration date on food. To do this, they launch the camera app, frame the food label, and take a photo. The input is the image data taken with the smartphone. The output is the image data taken.

[0372] Step 2:

[0373] The device sends the image to the server

[0374] explanation:

[0375] The device sends the image data captured in step 1 to the server. This transmission is performed via an HTTP request. The input is the captured image data. The output is the image data sent to the server.

[0376] Step 3:

[0377] The server extracts the date information from the image.

[0378] explanation:

[0379] The server processes the received image data and extracts the expiration date information as text using OCR technology (e.g., Tesseract OCR). The server passes the image data to the OCR engine to obtain the date information. The input is the image data sent to the server. The output is the extracted date information in text format.

[0380] Step 4:

[0381] The server stores the extracted information in a database

[0382] explanation:

[0383] The server stores the date information extracted in step 3 in a database, possibly along with additional information such as the name of the food item and the date of purchase. The input is the extracted date information and other relevant information. The output is the information stored in the database.

[0384] Step 5:

[0385] The server periodically checks the database

[0386] explanation:

[0387] The server periodically (e.g., every night at midnight) executes a batch process that checks expiration date information in the database. The input is the expiration date information in the database. The output is the checked expiration date information.

[0388] Step 6:

[0389] Identify approaching expiration dates on your servers

[0390] explanation:

[0391] The server checks each record in the database and calculates the difference between the current date and the expiration date, thereby identifying foods that are nearing their expiration date. The input is the expiration date information being checked and the current date. The output is the approaching expiration date information.

[0392] Step 7:

[0393] The server generates a notification message

[0394] explanation:

[0395] The server generates a notification message for the food item identified in step 6 that is nearing its expiration date. For example, it creates a message that reads "Milk will expire in 3 days." The input is the upcoming expiration date information. The output is the generated notification message.

[0396] Step 8:

[0397] The device sends a push notification to the user

[0398] explanation:

[0399] The server sends the notification message generated in step 7 to the device. The device displays this message to the user as a push notification. The input is the generated notification message. The output is the notification message displayed to the user.

[0400] Step 9:

[0401] The server retrieves the save method.

[0402] explanation:

[0403] The server retrieves the optimal storage method from the database for food items that are nearing their expiration date. For example, for milk, it retrieves information such as "keep it at the back of the refrigerator to keep it cool." The input is information about food items that are nearing their expiration date. The output is the retrieved storage method information.

[0404] Step 10:

[0405] The server generates a message informing you of the save method.

[0406] explanation:

[0407] The server generates a notification message based on the storage method information obtained in step 9. For example, it creates a message saying, "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator and keep it cool." The input is the obtained storage method information. The output is the generated notification message.

[0408] Step 11:

[0409] The device sends a notification to the user

[0410] explanation:

[0411] The server sends the notification message generated in step 10 to the terminal, and the terminal notifies the user of the information. The input is the generated notification message. The output is the notification message displayed to the user.

[0412] Step 12:

[0413] The server searches the database for the recipe

[0414] explanation:

[0415] The server searches the database for recipes that use foods that are close to their expiration date. For example, it finds a recipe for "cream stew" that uses milk. The input is information about foods that are close to their expiration date. The output is the searched recipe information.

[0416] Step 13:

[0417] The server generates a notification message with the appropriate recipe

[0418] explanation:

[0419] The server generates a notification message based on the recipe information retrieved in step 12. For example, it creates a message that reads, "Click here for a recipe for cream stew using milk." The input is the retrieved recipe information. The output is the generated notification message.

[0420] Step 14:

[0421] The device sends a notification to the user

[0422] explanation:

[0423] The server sends the notification message generated in step 13 to the terminal, and the terminal notifies the user of the information. The input is the generated notification message. The output is the notification message displayed to the user.

[0424] Step 15:

[0425] The server uses an emotion engine to recognize the user's emotional state.

[0426] explanation:

[0427] The server uses an emotion engine to recognize the user's emotional state. The input is the user's past data and current interaction information. The output is the recognized user's emotional state.

[0428] Step 16:

[0429] The server tailors notifications and suggestions based on emotions

[0430] explanation:

[0431] The server adjusts notifications and suggestions based on the emotional state recognized in step 15. For example, if it recognizes that the user is tired, it may suggest easy-to-make recipes. The input is the recognized emotional state of the user and the desired information. The output is the adjusted notifications and suggestions.

[0432] Step 17:

[0433] The server generates emotion-based recipe notification messages.

[0434] explanation:

[0435] The server generates a notification message based on the suggestions adjusted in step 16. For example, it creates a message such as "Easy recipes for when you're tired: Stir-fried vegetables." The input is the adjusted notification and suggestions. The output is the generated notification message.

[0436] Step 18:

[0437] The device notifies the user with emotion-based suggestions

[0438] explanation:

[0439] The server sends the notification message generated in step 17 to the terminal, and the terminal notifies the user of the information. The input is the generated notification message. The output is the notification message displayed to the user.

[0440] This allows users to efficiently manage expiration dates, receive suggestions for appropriate storage methods, and receive notifications and recipe suggestions based on the user's emotions.

[0441] (Application example 2)

[0442] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0443] In recent years, managing food expiration dates has become an important issue in homes and brick-and-mortar stores. However, efficiently managing expiration dates and reducing waste requires a lot of effort and cost. Furthermore, when users and staff are in an emotional state, it is necessary to further improve convenience and satisfaction by suggesting appropriate storage methods and recipes. Current technology does not provide such a comprehensive system, so a new approach is needed to improve users' quality of life.

[0444] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0445] In this invention, the server includes means for photographing expiration dates of food products and saving the photographed data, means for automatically notifying users when the expiration date is approaching, means for suggesting food preservation methods, means for providing recipes that combine foods with approaching expiration dates, means for photographing expiration dates using a smart device, and means for adjusting notification content and recipes based on the user's emotional state using an emotion engine. This makes it possible to efficiently manage expiration dates and make appropriate suggestions according to the user's emotions.

[0446] "Means for photographing food expiration dates and saving them as data" refers to a means for a user to photograph the expiration date of food purchased by the user using the camera on their smart device, analyze the image, and save the date information in a database.

[0447] The "means for automatically notifying when the expiration date is approaching" is a means for the server to periodically analyze the stored expiration date data and notify the user about food products whose expiration date is approaching.

[0448] The "means for suggesting a food preservation method" is a means for retrieving the optimum preservation method for food that is approaching its expiration date from a database and notifying the user of the method.

[0449] The "means for providing recipes that combine foods with close expiration dates" refers to a means for the server to search a database for recipes that combine foods with close expiration dates based on the expiration date information of the foods stored on the server, and suggest them to the user.

[0450] "Means for photographing expiration dates using a smart device" refers to means for photographing expiration dates on food products using a smart device such as a smartphone, smart glasses, or a head-mounted display.

[0451] "Means for adjusting notification content and recipes based on the user's emotional state using an emotion engine" refers to means for analyzing the user's emotional state using emotion recognition technology and adjusting the notification and recipe content based on the results.

[0452] As an embodiment of the present invention, a system for managing expiration dates of food products and suggesting appropriate storage methods and recipes includes the following processing steps.

[0453] The overall system configuration uses smart devices (smartphones, smart glasses, head-mounted displays, etc.) to manage food expiration dates. Specifically, when a user purchases food, they use the smart device's camera to take a photo of the expiration date, and the device sends the image to a server. The server then uses image analysis technology (OCR technology) to extract the expiration date information and saves it in a database.

[0454] Examples of devices used include smartphones and smart glasses. Examples of smart glasses include commonly used commercial products (e.g., Google Glass). The software uses Python, OpenCV for image processing, and pytesseract for OCR analysis. The emotion engine uses machine learning models and natural language processing techniques to analyze the user's emotional state.

[0455] The server periodically checks the expiration date information in the database, generates a notification message for food items whose expiration date is approaching, and sends it to the device. The device then notifies the user of this notification message via push notification. This allows the user to consume the food at the appropriate time without forgetting the expiration date.

[0456] Furthermore, the server retrieves the optimal storage method for food items approaching their expiration date from the database, generates a notification message suggesting that storage method, and notifies the user via the terminal. This allows the user to store the food properly and maintain its quality. The server also searches for recipe information in the database to suggest recipes that combine foods approaching their expiration date, selects appropriate recipes, generates a notification message, and displays it to the user via the terminal.

[0457] The emotion engine is used to recognize the user's emotional state. For example, when a user takes a picture while wearing the smart glasses, the emotion engine analyzes the user's facial expressions and evaluates the user's emotional state. Based on this, the server will adjust the content of recipes and notifications according to the user's emotions.

[0458] As a concrete example, if a user photographs the expiration date of milk with smart glasses, the system automatically analyzes the expiration date "2023-10-15" and stores it in a database. When the expiration date approaches, the server generates a notification stating "The milk's expiration date is in three days" and sends it as a push notification to the user. It also suggests storing the milk at the back of the refrigerator to keep it cool. If the emotion engine recognizes that the user is tired, it suggests a recipe for "stir-fried vegetables" as an "easy recipe recommended for when you're tired."

[0459] An example of a prompt for the generative AI model is as follows:

[0460] Example prompt sentence:

[0461] "Create a Python program that analyzes images of food taken with smart glasses and automatically extracts the expiration date. If the expiration date is approaching, it generates a message suggesting storage methods or recipes, and uses an emotion engine to respond according to the user's emotions. However, it should have conditions that allow the emotion engine to recognize fatigue, stress, or other specific emotional states. Please explain with a concrete example using smart glasses and Python."

[0462] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0463] Step 1:

[0464] A user takes a photo of the expiration date of a food product using a smart device (smartphone, smart glasses, etc.).

[0465] (Input) The image of the food to be photographed.

[0466] (Output) Food image data including expiration date.

[0467] (Specific Operation) For example, a user puts on smart glasses and takes a picture of the food label with the expiration date written on it. At this time, the camera function of the smart device is used.

[0468] Step 2:

[0469] The device sends the photographed image of the food to the server.

[0470] (Input) Food image data including expiration dates.

[0471] (Output) Image data sent to the server.

[0472] (Specific operation) Using the communication function built into the smart device, the image data of the food taken by the user is transferred to a server on the cloud, using Wi-Fi or mobile data communication technology.

[0473] Step 3:

[0474] The server uses image analysis technology (OCR technology) to extract the expiration date information from the transmitted image.

[0475] (Input) Image data of the food sent to the server.

[0476] (Output) The extracted expiration date information.

[0477] (Specific operation) The server analyzes the image using Python and OpenCV, and extracts the expiration date information (e.g., "2023-10-15") from the image by performing OCR processing using pytesseract.

[0478] Step 4:

[0479] The server stores the extracted expiration date information in a database.

[0480] (Input) The extracted expiration date information.

[0481] (Output) Best before date data stored in the database.

[0482] (Specific operation) The server saves the extracted date information in a database (e.g., MySQL or SQLite) using an SQL query.

[0483] Step 5:

[0484] The server periodically checks the expiration date information in the database.

[0485] (Input) Best before date data in the database.

[0486] (Output) A list of foods that are nearing their expiration date.

[0487] (Specific Operation) The server periodically queries all expiration date data in the database using a scheduler to identify items whose expiration dates will arrive within a certain period from the current date.

[0488] Step 6:

[0489] The server generates a notification message about food products that are approaching their expiration date and sends it to the terminal.

[0490] (Input) A list of food items that are nearing their expiration date.

[0491] (Output) Informational message.

[0492] (Specific operation) The server generates the content of the push notification (for example, "The expiration date of the milk is three days later") and sends it to the terminal via an API for sending notification messages to smart devices.

[0493] Step 7:

[0494] The terminal notifies the user of the notification message by push notification.

[0495] (Input) The notification message.

[0496] (Output) The notification that is displayed to the user.

[0497] (Specific operation) The notification message received by the terminal is displayed to the user using the notification function of the smart device.

[0498] Step 8:

[0499] The server retrieves the optimum storage method for food items whose expiration date is approaching from a database, generates a notification message, and transmits it to the terminal.

[0500] (Input) A list of foods that are close to expiry.

[0501] (Output) A message informing you of the save method.

[0502] (Specific operation) For example, the server obtains information from the database about how to store milk, such as "place it at the back of the refrigerator to cool it down," and based on this information generates a notification message and sends it to the terminal.

[0503] Step 9:

[0504] The server searches a database for recipes that combine foods whose expiration dates are approaching, selects them, generates a notification message, and transmits it to the terminal.

[0505] (Input) A list of foods that are close to expiry.

[0506] (Output) The recipe notification message.

[0507] (Specific operation) The server searches the database for "recipes that combine foods with close expiration dates" (for example, "cream stew using milk"), generates a notification message based on this, and sends it to the terminal.

[0508] Step 10:

[0509] It uses an emotion engine to recognize the user's emotional state and adjust recipes and notifications accordingly.

[0510] (Input) The user's emotional state.

[0511] (Output) Emotion-based recipes and notification content.

[0512] (Specific operation) The emotion engine analyzes the user's image, voice, and input content to recognize their emotional state, such as fatigue or stress. For example, if the server determines that the user is tired, it generates a notification message suggesting a "simple vegetable stir-fry recipe" and sends it to the device.

[0513] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0514] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0515] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0516] [Second embodiment]

[0517] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0518] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0519] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0520] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0521] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0522] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0523] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0524] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0525] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0526] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0527] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0528] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0529] The present invention relates to a system for managing expiration dates of food products and proposing appropriate storage methods and recipes. Specific embodiments of the system will be described below.

[0530] 1. Acquisition and storage of expiration date data

[0531] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[0532] Specific examples

[0533] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[0534] 2. Best before date tracking and notification

[0535] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[0536] Specific examples

[0537] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[0538] 3. Storage method suggestions

[0539] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[0540] Specific examples

[0541] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[0542] 4. Recipe suggestions

[0543] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[0544] Specific examples

[0545] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[0546] As described above, this system aims to efficiently manage food expiration dates and reduce food waste by suggesting storage methods and recipes, allowing users to maintain food quality while consuming it and leading an economical and environmentally conscious lifestyle.

[0547] The processing flow will be explained below.

[0548] Acquiring and storing expiration date data

[0549] Step 1:

[0550] The user takes a photo of the expiration date of the food they have purchased using their smartphone camera.

[0551] Step 2:

[0552] The terminal transmits the captured image to the server.

[0553] Step 3:

[0554] The server uses image analysis technology (OCR technology) to extract the expiration date information from the image.

[0555] Step 4:

[0556] The server stores the extracted expiration date information in a database.

[0557] Best before date tracking and notification

[0558] Step 1:

[0559] The server periodically checks the expiration date information in the database.

[0560] Step 2:

[0561] The server identifies food items that are nearing their expiration date (e.g., 3 days away).

[0562] Step 3:

[0563] The server generates notification messages corresponding to approaching expiration dates.

[0564] Step 4:

[0565] The terminal receives the notification message from the server and sends a push notification to the user.

[0566] Storage suggestions

[0567] Step 1:

[0568] The server retrieves from the database the storage method for food items that are nearing their expiration date.

[0569] Step 2:

[0570] The server adds information about the storage method to the notification message.

[0571] Step 3:

[0572] The terminal displays a notification message to the user, including instructions on how to save.

[0573] Recipe Suggestions

[0574] Step 1:

[0575] The server searches the database for recipe information based on foods that are close to their expiration date.

[0576] Step 2:

[0577] The server selects an appropriate recipe based on the recipe information.

[0578] Step 3:

[0579] The server generates a notification message regarding the selected recipe.

[0580] Step 4:

[0581] The terminal displays a notification message to the user that includes the selected recipe.

[0582] Example 1

[0583] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0584] In modern society, many consumers waste food. In particular, discarding food that has passed its expiration date causes economic losses and negative impacts on the environment. To solve this problem, a system is needed that can efficiently manage food expiration dates, notify consumers at the appropriate time, and even suggest storage methods and recipes.

[0585] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0586] In this invention, the server includes means for photographing expiration dates of food products with a device camera and sending the image to the server, means for the server to extract date information using image analysis technology and store it in a database, means for the server to periodically check the expiration date information in the database and, when an expiration date is approaching, generate a notification message and send a push notification to the user via the device, means for the server to suggest appropriate storage methods for food products approaching their expiration dates, and means for the server to provide recipes that combine food products approaching their expiration dates. This makes it easier for users to manage expiration dates, reduces food waste, and enables them to live economically and environmentally conscious lives.

[0587] The "device camera" refers to a photographing device installed in a portable electronic device such as a smartphone or tablet that a user uses to photograph the expiration date of food products.

[0588] A "server" is a computer device that uses image analysis technology to extract date information from captured image data, stores it in a database, and periodically checks the information in the database and sends notification messages to users.

[0589] "Image analysis technology" is a technology that extracts text information from image data, including food expiration dates, and generally refers to OCR (Optical Character Recognition) technology.

[0590] A "database" is a structured data storage system for storing information about expiration dates and food products, which is accessed and updated by a server.

[0591] A "notification message" is text information that is generated by the server to inform the user of food products that are approaching their expiration date and is sent to the user via the terminal.

[0592] "Push notification" is a technology for displaying notification messages on a user's device in real time, and is a means of displaying messages on the device's home screen or notification bar.

[0593] "Storage method" refers to the recommended storage method or procedure for extending the expiration date while maintaining the quality of food, and the server searches the database and suggests it to the user.

[0594] A "recipe" is information that indicates steps and combinations of ingredients for creating a dish that efficiently uses food that is approaching its expiration date.

[0595] "Food" refers to ingredients or cooked products that are purchased and consumed by users and have a best-before date.

[0596] The present invention relates to a system that manages the expiration dates of food products and suggests appropriate storage methods and recipes. This system takes a photo of the expiration date using a device camera, performs image analysis on a server, and provides notifications and suggestions based on the expiration date information stored in a database. Specific embodiments are described below.

[0597] 1. Acquisition and storage of expiration date data

[0598] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses OCR technology (e.g., Tesseract OCR) to analyze and extract the expiration date information from the image. This extracted date information is then stored in a database along with the food's information. This saves users the trouble of manually entering the expiration date.

[0599] Examples:

[0600] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[0601] 2. Best before date tracking and notification

[0602] The server periodically checks the expiration date information in the database, identifies foods that are approaching their expiration date, and if so, generates a notification message and sends a push notification to the user via their device, allowing the user to consume the food at the appropriate time without forgetting the expiration date.

[0603] Examples:

[0604] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[0605] 3. Storage method suggestions

[0606] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[0607] Examples:

[0608] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[0609] 4. Recipe suggestions

[0610] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[0611] Examples:

[0612] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[0613] The specific processing flow of this system can be explained by inputting the following prompt sentence into the generative AI model.

[0614] "Please tell me the specific process flow for a system that manages food expiration dates and suggests appropriate storage methods and recipes. Please provide a detailed explanation of each process step, including specific operations."

[0615] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0616] Step 1:

[0617] The user takes a photo of the expiration date of the food product they purchased with their smartphone camera. For example, the user launches the smartphone's camera app and takes a photo of the milk they purchased so that the expiration date "2023-10-15" is displayed on the screen.

[0618] Input: An image of the expiration date taken with a smartphone camera

[0619] Output: Image data of expiration dates stored on a smartphone

[0620] Step 2:

[0621] The device sends the captured images to a server. Specifically, the device uses an Internet connection to upload image data to the server via the HTTPS protocol.

[0622] Input: Image data of expiration date stored on smartphone

[0623] Output: Image data sent to the server

[0624] Step 3:

[0625] The server analyzes the received image using OCR technology (e.g., Tesseract OCR) to extract the expiration date information. The server then runs its internal algorithms to analyze the image and obtain the text data.

[0626] Input: Image data sent to the server

[0627] Output: Extracted best-before date information (e.g. "2023-10-15")

[0628] Step 4:

[0629] The server stores the extracted date information along with the food information in a database. The server inserts the food information (e.g., "Milk", "2023-10-15", "2023-10-01") into the appropriate fields in the database.

[0630] Input: Extracted best-before date information and food information

[0631] Output: Best before date information stored in the database

[0632] Step 5:

[0633] The server periodically checks the expiration date information in the database, for example, as a scheduled job once a day.

[0634] Input: Best before date information in the database

[0635] Output: Checked expiration date information

[0636] Step 6:

[0637] The server identifies foods that are nearing their expiration date and generates a notification message. For example, if the server determines that there are only three days until 2023-10-15 as of 2023-10-12, it generates a message saying, "Milk's expiration date is in three days."

[0638] Input: Checked expiration date information and current date

[0639] Output: The generated notification message

[0640] Step 7:

[0641] The server sends the generated notification message to the device. The device receives the push notification and displays it to the user. For example, a notification saying "The milk's expiration date is in three days" will be displayed on the smartphone screen.

[0642] Input: The generated notification message

[0643] Output: Push notification displayed on device

[0644] Step 8:

[0645] The server retrieves the optimal storage method for food that is nearing its expiration date from the database, searches for the storage method data for the relevant food, and generates a suggestion message.

[0646] Input: Information about food that is close to its expiration date

[0647] Output: Generated save method suggestion message

[0648] Step 9:

[0649] The server sends the generated storage suggestion message to the device, which then displays it to the user via push notification. For example, a notification such as "The milk's expiration date is in three days. Please place it at the back of the refrigerator to keep it cool" may be displayed.

[0650] Input: Generated save method suggestion message

[0651] Output: Push notification with suggested saving methods displayed on the device

[0652] Step 10:

[0653] The server searches the database for recipes that combine food items whose expiration dates are approaching, retrieves the corresponding recipe data, and generates a recipe suggestion message.

[0654] Input: Information about food that is close to its expiration date

[0655] Output: Generated recipe suggestion message

[0656] Step 11:

[0657] The server sends the generated recipe suggestion message to the device, which then displays it to the user via a push notification. For example, a notification such as "Click here for a recipe for cream stew using milk" is displayed.

[0658] Input: Generated recipe suggestion message

[0659] Output: Push notification with recipe suggestions displayed on the device

[0660] (Application example 1)

[0661] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0662] The goal is to provide a system that manages food expiration dates and suggests appropriate storage methods and recipes, allowing users to easily manage expiration dates, suggest storage methods, and search for recipes. In particular, there is a need for a means to automate information input and send notifications at appropriate times. There is also a desire to reduce food waste by suggesting recipes and notifying users of storage methods to efficiently use food that is approaching its expiration date.

[0663] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0664] In this invention, the server includes a means for automatically extracting expiration date information for food products using image analysis technology and managing the data, a means for sending notifications to users using push notification technology when the expiration date approaches, and a means for displaying notifications on a user interface with storage methods and recipes. This not only enables users to efficiently manage expiration dates, but also automatically receives suggestions for appropriate storage methods and recipes that combine foods with approaching expiration dates. Furthermore, the use of notification technology allows users to obtain important information at the appropriate time, thereby reducing food waste.

[0665] "Means for photographing food expiration dates and saving the data" is a function that allows users to use a device such as a smartphone to photograph the expiration date displayed on food and save the data.

[0666] The "means for automatically notifying when the expiration date is approaching" is a function that automatically sends a warning or alert to the user when the expiration date is approaching based on stored expiration date data.

[0667] The "means for suggesting food preservation methods" is a function for suggesting to the user how to properly preserve food that is approaching its expiration date.

[0668] The "means for providing recipes that combine food items with close expiration dates" is a function for providing users with recipes that use multiple food items with close expiration dates.

[0669] "Means for automatically extracting expiration date information using image analysis technology and managing the data" refers to a function that converts photographed expiration date information into text data using image analysis technology such as OCR technology and manages the data.

[0670] "Means for sending notifications to users using push notification technology" refers to a function that uses cloud services, etc. to notify users' devices in real time that the expiration date is approaching.

[0671] The "means for displaying notifications on a user interface" is a function for displaying storage methods and recipe suggestions on the interface on the user's device.

[0672] The present invention relates to a system for managing expiration dates of food products and proposing appropriate storage methods and recipes. Specific embodiments of the system will be described below.

[0673] First, when a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[0674] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[0675] Furthermore, for food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[0676] The server also searches the database for recipe information to suggest recipes that combine foods that are approaching their expiration dates. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[0677] Specifically, the hardware used includes smartphones and servers, and the software used includes Tesseract OCR, which implements OCR technology, and Firebase Cloud Messaging, which provides push notification technology.

[0678] For example, if a user purchases milk and takes a photo of its expiration date, "2023-10-15," with their smartphone, the device sends this image to a server. The server uses OCR technology to extract the expiration date, "2023-10-15," from the image and stores it in a database. If the server checks the milk's expiration date, "2023-10-15," in the database and determines that the current date is "2023-10-12," the server generates a notification message stating, "The milk's expiration date is three days from now," and sends this push notification to the user via their device. It also generates a notification message suggesting a storage method, "Put it at the back of the refrigerator to cool," and a recipe, "Cream stew."

[0679] An example of a prompt sentence for the generative AI model is written as follows:

[0680] Extract the expiration date from the image below:

[0681] Image path: path_to_image.jpg

[0682] If the expiration date is "2023-10-15", generate a notification message suggesting the best way to store the food, "Put it at the back of the refrigerator to cool it down", and also suggest the recipe "Cream Stew".

[0683] As described above, this system aims to efficiently manage food expiration dates and reduce food waste by suggesting storage methods and recipes, allowing users to maintain food quality while consuming it and leading an economical and environmentally conscious lifestyle.

[0684] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0685] Step 1:

[0686] A user takes a photo of a food product's expiration date with a smartphone camera. The input is an image of the expiration date printed on the food's packaging. The output is the captured image data.

[0687] Step 2:

[0688] The device sends the captured image to the server. The input is the image data captured by the smartphone. The output is the image data transferred to the server.

[0689] Step 3:

[0690] The server uses OCR technology to extract the expiration date information from the image. The input is the received image data. The data processing is image analysis using Tesseract OCR. The output is the extracted expiration date string.

[0691] Step 4:

[0692] The extracted date information is saved in a database. The input is the date string obtained by OCR analysis. The data processing is writing the date string to the database. The output is a database entry with the expiration date saved.

[0693] Step 5:

[0694] The server periodically checks the expiration date information in the database. The input is the expiration date information of all food items stored in the database. The data calculation is a comparison with the current date. The output is a list of food items that are approaching their expiration date.

[0695] Step 6:

[0696] Identify foods approaching their expiration date and generate a notification message. The input is a list of foods approaching their expiration date. The data processing is the generation of the notification message. The output is the generated notification message.

[0697] Step 7:

[0698] The server sends the generated notification message to the device via Firebase Cloud Messaging. The input is the generated notification message. The output is the notification sent to the user's smartphone.

[0699] Step 8:

[0700] The device notifies the user via a push notification. The input is the notification message received from the server. The output is the push notification displayed to the user.

[0701] Step 9:

[0702] For foods approaching their expiration date, the server retrieves the optimal storage method from the database. The input is a list of foods approaching their expiration date. The data operation is a query from the storage method database. The output is the retrieved storage method.

[0703] Step 10:

[0704] A notification message is generated based on the storage method and notified to the user via the terminal. The input is the storage method. Data processing is the generation of the storage method notification message. The output is the storage method notification message.

[0705] Step 11:

[0706] To suggest recipes that combine foods with close expiration dates, the server searches for recipe information in a database. The input is a list of foods with close expiration dates. The data operation is a query from the recipe database. The output is a suitable recipe.

[0707] Step 12:

[0708] Select the appropriate recipe and generate a notification message. The input is the retrieved recipe information. The data processing is the generation of the recipe notification message. The output is the generated recipe notification message.

[0709] Step 13:

[0710] The terminal displays the generated recipe notification message to the user. The input is the recipe notification message from the server. The output is the recipe suggestion notification displayed to the user.

[0711] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0712] The present invention combines an emotion engine with a system that manages food expiration dates and suggests appropriate storage methods and recipes, providing a service based on the user's emotions. Specific embodiments of this system are described below.

[0713] 1. Acquisition and storage of expiration date data

[0714] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[0715] Specific examples

[0716] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[0717] 2. Best before date tracking and notification

[0718] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[0719] Specific examples

[0720] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[0721] 3. Storage method suggestions

[0722] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[0723] Specific examples

[0724] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[0725] 4. Recipe suggestions

[0726] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[0727] Specific examples

[0728] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[0729] 5. Providing services using emotion engines

[0730] 5.1 Emotion Recognition and Recipe Adjustment

[0731] The server uses an emotion engine to recognize the user's emotional state and adjusts the content of recipes and notifications depending on the user's emotions. For example, if the user is tired, it will suggest easy-to-make recipes.

[0732] Specific examples

[0733] The server uses an emotion engine to recognize that the user is tired, and then suggests a recipe for "stir-fried vegetables" as an "easy recipe recommended for when you're tired" and notifies the user via their device.

[0734] 5.2 Recipe suggestions based on tastes and preferences

[0735] The emotion engine determines the user's tastes and preferences and suggests recipes based on them, making it easier for users to cook their favorite dishes.

[0736] Specific examples

[0737] The emotion engine analyzes the user's preferences based on past data, and the server suggests a recipe for "Japanese-style pasta" as "a dish with the user's favorite seasoning." The device notifies the user of this information.

[0738] 5.3 Food suggestions for stress reduction

[0739] The emotion engine assesses the user's stress level and suggests foods and recipes that have a stress-reducing effect.

[0740] Specific examples

[0741] The emotion engine recognizes that the user is feeling stressed, and the server suggests a "relaxing herbal tea recipe." The device notifies the user of this information.

[0742] As described above, this system aims not only to efficiently manage food expiration dates and reduce food waste by suggesting storage methods and recipes, but also to provide services that respond to the user's emotional state using an emotion engine, thereby enabling users to live more comfortable and efficient lives.

[0743] The processing flow will be explained below.

[0744] Acquiring and storing expiration date data

[0745] Step 1:

[0746] The user takes a photo of the expiration date of the food they have purchased using their smartphone camera.

[0747] Step 2:

[0748] The terminal transmits the captured image to the server.

[0749] Step 3:

[0750] The server uses image analysis technology (OCR technology) to extract the expiration date information from the image.

[0751] Step 4:

[0752] The server stores the extracted expiration date information in a database.

[0753] Best before date tracking and notification

[0754] Step 1:

[0755] The server periodically checks the expiration date information in the database.

[0756] Step 2:

[0757] The server identifies food items that are nearing their expiration date (e.g., 3 days away).

[0758] Step 3:

[0759] The server generates notification messages corresponding to approaching expiration dates.

[0760] Step 4:

[0761] The terminal receives the notification message from the server and sends a push notification to the user.

[0762] Storage suggestions

[0763] Step 1:

[0764] The server retrieves from the database the storage method for food that is nearing its expiration date.

[0765] Step 2:

[0766] The server adds information about the storage method to the notification message.

[0767] Step 3:

[0768] The terminal displays a notification message to the user including the saving method.

[0769] Recipe Suggestions

[0770] Step 1:

[0771] The server searches the database for recipe information based on foods with an approaching expiration date.

[0772] Step 2:

[0773] The server selects an appropriate recipe based on the recipe information.

[0774] Step 3:

[0775] The server generates a notification message about the selected recipe.

[0776] Step 4:

[0777] The terminal displays a notification message to the user containing the selected recipe.

[0778] Providing services using emotion engines

[0779] Emotion recognition and recipe adjustment

[0780] Step 1:

[0781] The server uses an emotion engine to recognize the user's emotional state.

[0782] Step 2:

[0783] The server selects recipes and notification messages according to the user's emotions.

[0784] Step 3:

[0785] The server generates the selected recipe and notification message.

[0786] Step 4:

[0787] The device displays recipes and notification messages to the user based on their emotions.

[0788] Recipe suggestions based on your tastes and preferences

[0789] Step 1:

[0790] The server uses an emotion engine to determine the user's preferences and tastes.

[0791] Step 2:

[0792] The server retrieves recipes based on the user's preferences.

[0793] Step 3:

[0794] The server selects recipes based on preferences and generates notification messages.

[0795] Step 4:

[0796] The device displays a notification message to the user containing a recipe based on the preferences.

[0797] Food suggestions for stress reduction

[0798] Step 1:

[0799] The server uses an emotion engine to determine whether the user is experiencing stress.

[0800] Step 2:

[0801] The server searches for foods and recipes that have stress-reducing effects.

[0802] Step 3:

[0803] The server selects foods and recipes for stress reduction and generates a notification message.

[0804] Step 4:

[0805] The device will display a notification message to the user containing stress-reducing foods and recipes.

[0806] Example 2

[0807] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0808] Current food management systems require users to manually input expiration dates, making expiration date management cumbersome. Furthermore, storage methods and recipe suggestions are uniform and are not based on the user's emotional state or preferences, resulting in low user convenience. Furthermore, if expiration date notifications are not properly provided, there is a high possibility of food waste.

[0809] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for photographing the expiration date of food and saving it as data, a means for automatically notifying when the expiration date is approaching, a means for suggesting a food preservation method, a means for providing a recipe that combines foods with close expiration dates, and a means for recognizing the user's emotional state and adjusting the notification and suggestion content based on the emotional state. This enables efficient management of expiration dates, suggestion of appropriate preservation methods, and suggestion of recipes that correspond to the user's emotions.

[0810] "Means for photographing expiration dates and saving them as data" refers to a means by which a user photographs the expiration date of a food product using a smartphone camera, sends the image to a server, and uses image analysis technology (OCR technology) to extract the date information and save it in a database.

[0811] "Means for automatically notifying when the expiration date is approaching" refers to a means in which the server periodically checks the expiration date information in the database, generates a notification message when the expiration date is approaching, and pushes the message to the user via the terminal.

[0812] The "means for suggesting food preservation methods" refers to a means by which the server retrieves the optimal preservation method from a database for food that is approaching its expiration date, generates a notification message based on that information, and notifies the user via the terminal.

[0813] The "means for providing recipes that combine foods with close expiration dates" refers to a means by which the server searches a database for recipe information based on foods with close expiration dates, selects an appropriate recipe, generates a notification message, and displays it to the user via the terminal.

[0814] "Means for recognizing the user's emotional state and adjusting notifications and suggestions based on that" refers to a means by which the server recognizes the user's emotional state using an emotion engine and adjusts notifications and suggestions based on the user's emotions. Specifically, this includes adjustments such as suggesting easy-to-make recipes when the user is tired.

[0815] The present invention combines an emotion engine with a system that manages food expiration dates and suggests appropriate storage methods and recipes, providing a service based on the user's emotions. Specific embodiments of this system are described below.

[0816] Acquiring and storing expiration date data

[0817] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[0818] Specific examples

[0819] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[0820] Best before date tracking and notification

[0821] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[0822] Specific examples

[0823] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[0824] Storage suggestions

[0825] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[0826] Specific examples

[0827] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[0828] Recipe Suggestions

[0829] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[0830] Specific examples

[0831] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[0832] Providing services using emotion engines

[0833] Emotion recognition and recipe adjustment

[0834] The server uses an emotion engine to recognize the user's emotional state and adjusts the content of recipes and notifications depending on the user's emotions. For example, if the user is tired, it will suggest easy-to-make recipes.

[0835] Specific examples

[0836] The server uses an emotion engine to recognize that the user is tired, and then suggests a recipe for "stir-fried vegetables" as an "easy recipe recommended for when you're tired" and notifies the user via their device.

[0837] Recipe suggestions based on your tastes and preferences

[0838] The emotion engine determines the user's tastes and preferences and suggests recipes based on them, making it easier for users to cook their favorite dishes.

[0839] Specific examples

[0840] The emotion engine analyzes the user's preferences based on past data, and the server suggests a recipe for "Japanese-style pasta" as "a dish with the user's favorite seasoning." The device notifies the user of this information.

[0841] Food suggestions for stress reduction

[0842] The emotion engine assesses the user's stress level and suggests foods and recipes that have a stress-reducing effect.

[0843] Specific examples

[0844] The emotion engine recognizes that the user is feeling stressed, and the server suggests a "relaxing herbal tea recipe." The device notifies the user of this information.

[0845] Prompt Sentence Examples

[0846] "Please suggest recipes that efficiently use food in the refrigerator that is nearing its expiration date."

[0847] "Please suggest easy dinner recipes for when the user is tired."

[0848] "Provide new recipes based on your users' favorite flavors."

[0849] With this system, users can efficiently manage food expiration dates and receive suggestions for storage methods and recipes. Furthermore, by using an emotion engine, it is possible to provide services that correspond to the user's emotional state, improving the quality of daily life.

[0850] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0851] Step 1:

[0852] A user takes a photo of the expiration date on a food item

[0853] explanation:

[0854] The user uses the smartphone camera to take a photo of the expiration date on food. To do this, they launch the camera app, frame the food label, and take a photo. The input is the image data taken with the smartphone. The output is the image data taken.

[0855] Step 2:

[0856] The device sends the image to the server

[0857] explanation:

[0858] The device sends the image data captured in step 1 to the server. This transmission is performed via an HTTP request. The input is the captured image data. The output is the image data sent to the server.

[0859] Step 3:

[0860] The server extracts the date information from the image.

[0861] explanation:

[0862] The server processes the received image data and extracts the expiration date information as text using OCR technology (e.g., Tesseract OCR). The server passes the image data to the OCR engine to obtain the date information. The input is the image data sent to the server. The output is the extracted date information in text format.

[0863] Step 4:

[0864] The server stores the extracted information in a database

[0865] explanation:

[0866] The server stores the date information extracted in step 3 in a database, possibly along with additional information such as the name of the food item and the date of purchase. The input is the extracted date information and other relevant information. The output is the information stored in the database.

[0867] Step 5:

[0868] The server periodically checks the database

[0869] explanation:

[0870] The server periodically (e.g., every night at midnight) executes a batch process that checks expiration date information in the database. The input is the expiration date information in the database. The output is the checked expiration date information.

[0871] Step 6:

[0872] Identify approaching expiration dates on your servers

[0873] explanation:

[0874] The server checks each record in the database and calculates the difference between the current date and the expiration date, thereby identifying foods that are nearing their expiration date. The input is the expiration date information being checked and the current date. The output is the approaching expiration date information.

[0875] Step 7:

[0876] The server generates a notification message

[0877] explanation:

[0878] The server generates a notification message for the food item identified in step 6 that is nearing its expiration date. For example, it creates a message that reads "Milk will expire in 3 days." The input is the upcoming expiration date information. The output is the generated notification message.

[0879] Step 8:

[0880] The device sends a push notification to the user

[0881] explanation:

[0882] The server sends the notification message generated in step 7 to the device. The device displays this message to the user as a push notification. The input is the generated notification message. The output is the notification message displayed to the user.

[0883] Step 9:

[0884] The server retrieves the save method.

[0885] explanation:

[0886] The server retrieves the optimal storage method from the database for food items that are nearing their expiration date. For example, for milk, it retrieves information such as "keep it at the back of the refrigerator to keep it cool." The input is information about food items that are nearing their expiration date. The output is the retrieved storage method information.

[0887] Step 10:

[0888] The server generates a message informing you of the save method.

[0889] explanation:

[0890] The server generates a notification message based on the storage method information obtained in step 9. For example, it creates a message saying, "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator and keep it cool." The input is the obtained storage method information. The output is the generated notification message.

[0891] Step 11:

[0892] The device sends a notification to the user

[0893] explanation:

[0894] The server sends the notification message generated in step 10 to the terminal, and the terminal notifies the user of the information. The input is the generated notification message. The output is the notification message displayed to the user.

[0895] Step 12:

[0896] The server searches the database for the recipe

[0897] explanation:

[0898] The server searches the database for recipes that use foods that are close to their expiration date. For example, it finds a recipe for "cream stew" that uses milk. The input is information about foods that are close to their expiration date. The output is the searched recipe information.

[0899] Step 13:

[0900] The server generates a notification message with the appropriate recipe

[0901] explanation:

[0902] The server generates a notification message based on the recipe information retrieved in step 12. For example, it creates a message that reads, "Click here for a recipe for cream stew using milk." The input is the retrieved recipe information. The output is the generated notification message.

[0903] Step 14:

[0904] The device sends a notification to the user

[0905] explanation:

[0906] The server sends the notification message generated in step 13 to the terminal, and the terminal notifies the user of the information. The input is the generated notification message. The output is the notification message displayed to the user.

[0907] Step 15:

[0908] The server uses an emotion engine to recognize the user's emotional state.

[0909] explanation:

[0910] The server uses an emotion engine to recognize the user's emotional state. The input is the user's past data and current interaction information. The output is the recognized user's emotional state.

[0911] Step 16:

[0912] The server tailors notifications and suggestions based on emotions

[0913] explanation:

[0914] The server adjusts notifications and suggestions based on the emotional state recognized in step 15. For example, if it recognizes that the user is tired, it may suggest easy-to-make recipes. The input is the recognized emotional state of the user and the desired information. The output is the adjusted notifications and suggestions.

[0915] Step 17:

[0916] The server generates emotion-based recipe notification messages.

[0917] explanation:

[0918] The server generates a notification message based on the suggestions adjusted in step 16. For example, it creates a message such as "Easy recipes for when you're tired: Stir-fried vegetables." The input is the adjusted notification and suggestions. The output is the generated notification message.

[0919] Step 18:

[0920] The device notifies the user with emotion-based suggestions

[0921] explanation:

[0922] The server sends the notification message generated in step 17 to the terminal, and the terminal notifies the user of the information. The input is the generated notification message. The output is the notification message displayed to the user.

[0923] This allows users to efficiently manage expiration dates, receive suggestions for appropriate storage methods, and receive notifications and recipe suggestions based on the user's emotions.

[0924] (Application example 2)

[0925] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0926] In recent years, managing food expiration dates has become an important issue in homes and brick-and-mortar stores. However, efficiently managing expiration dates and reducing waste requires a lot of effort and cost. Furthermore, when users and staff are in an emotional state, it is necessary to further improve convenience and satisfaction by suggesting appropriate storage methods and recipes. Current technology does not provide such a comprehensive system, so a new approach is needed to improve users' quality of life.

[0927] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0928] In this invention, the server includes means for photographing expiration dates of food products and saving the photographed data, means for automatically notifying users when the expiration date is approaching, means for suggesting food preservation methods, means for providing recipes that combine foods with approaching expiration dates, means for photographing expiration dates using a smart device, and means for adjusting notification content and recipes based on the user's emotional state using an emotion engine. This makes it possible to efficiently manage expiration dates and make appropriate suggestions according to the user's emotions.

[0929] "Means for photographing food expiration dates and saving them as data" refers to a means for a user to photograph the expiration date of food purchased by the user using the camera on their smart device, analyze the image, and save the date information in a database.

[0930] The "means for automatically notifying when the expiration date is approaching" is a means for the server to periodically analyze the stored expiration date data and notify the user about food products whose expiration date is approaching.

[0931] The "means for suggesting a food preservation method" is a means for retrieving the optimum preservation method for food that is approaching its expiration date from a database and notifying the user of the method.

[0932] The "means for providing recipes that combine foods with close expiration dates" refers to a means for the server to search a database for recipes that combine foods with close expiration dates based on the expiration date information of the foods stored on the server, and suggest them to the user.

[0933] "Means for photographing expiration dates using a smart device" refers to means for photographing expiration dates on food products using a smart device such as a smartphone, smart glasses, or a head-mounted display.

[0934] "Means for adjusting notification content and recipes based on the user's emotional state using an emotion engine" refers to means for analyzing the user's emotional state using emotion recognition technology and adjusting the notification and recipe content based on the results.

[0935] As an embodiment of the present invention, a system for managing expiration dates of food products and suggesting appropriate storage methods and recipes includes the following processing steps.

[0936] The overall system configuration uses smart devices (smartphones, smart glasses, head-mounted displays, etc.) to manage food expiration dates. Specifically, when a user purchases food, they use the smart device's camera to take a photo of the expiration date, and the device sends the image to a server. The server then uses image analysis technology (OCR technology) to extract the expiration date information and saves it in a database.

[0937] Examples of devices used include smartphones and smart glasses. Examples of smart glasses include commonly used commercial products (e.g., Google Glass). The software uses Python, OpenCV for image processing, and pytesseract for OCR analysis. The emotion engine uses machine learning models and natural language processing techniques to analyze the user's emotional state.

[0938] The server periodically checks the expiration date information in the database, generates a notification message for food items whose expiration date is approaching, and sends it to the device. The device then notifies the user of this notification message via push notification. This allows the user to consume the food at the appropriate time without forgetting the expiration date.

[0939] Furthermore, the server retrieves the optimal storage method for food items approaching their expiration date from the database, generates a notification message suggesting that storage method, and notifies the user via the terminal. This allows the user to store the food properly and maintain its quality. The server also searches for recipe information in the database to suggest recipes that combine foods approaching their expiration date, selects appropriate recipes, generates a notification message, and displays it to the user via the terminal.

[0940] The emotion engine is used to recognize the user's emotional state. For example, when a user takes a picture while wearing the smart glasses, the emotion engine analyzes the user's facial expressions and evaluates the user's emotional state. Based on this, the server will adjust the content of recipes and notifications according to the user's emotions.

[0941] As a concrete example, if a user photographs the expiration date of milk with smart glasses, the system automatically analyzes the expiration date "2023-10-15" and stores it in a database. When the expiration date approaches, the server generates a notification stating "The milk's expiration date is in three days" and sends it as a push notification to the user. It also suggests storing the milk at the back of the refrigerator to keep it cool. If the emotion engine recognizes that the user is tired, it suggests a recipe for "stir-fried vegetables" as an "easy recipe recommended for when you're tired."

[0942] An example of a prompt for the generative AI model is as follows:

[0943] Example prompt sentence:

[0944] "Create a Python program that analyzes images of food taken with smart glasses and automatically extracts the expiration date. If the expiration date is approaching, it generates a message suggesting storage methods or recipes, and uses an emotion engine to respond according to the user's emotions. However, it should have conditions that allow the emotion engine to recognize fatigue, stress, or other specific emotional states. Please explain with a concrete example using smart glasses and Python."

[0945] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0946] Step 1:

[0947] A user takes a photo of the expiration date of a food product using a smart device (smartphone, smart glasses, etc.).

[0948] (Input) The image of the food to be photographed.

[0949] (Output) Food image data including expiration date.

[0950] (Specific Operation) For example, a user puts on smart glasses and takes a picture of the food label with the expiration date written on it. At this time, the camera function of the smart device is used.

[0951] Step 2:

[0952] The device sends the photographed image of the food to the server.

[0953] (Input) Food image data including expiration dates.

[0954] (Output) Image data sent to the server.

[0955] (Specific operation) Using the communication function built into the smart device, the image data of the food taken by the user is transferred to a server on the cloud, using Wi-Fi or mobile data communication technology.

[0956] Step 3:

[0957] The server uses image analysis technology (OCR technology) to extract the expiration date information from the transmitted image.

[0958] (Input) Image data of the food sent to the server.

[0959] (Output) The extracted expiration date information.

[0960] (Specific operation) The server analyzes the image using Python and OpenCV, and extracts the expiration date information (e.g., "2023-10-15") from the image by performing OCR processing using pytesseract.

[0961] Step 4:

[0962] The server stores the extracted expiration date information in a database.

[0963] (Input) The extracted expiration date information.

[0964] (Output) Best before date data stored in the database.

[0965] (Specific operation) The server saves the extracted date information in a database (e.g., MySQL or SQLite) using an SQL query.

[0966] Step 5:

[0967] The server periodically checks the expiration date information in the database.

[0968] (Input) Best before date data in the database.

[0969] (Output) A list of foods that are nearing their expiration date.

[0970] (Specific Operation) The server periodically queries all expiration date data in the database using a scheduler to identify items whose expiration dates will arrive within a certain period from the current date.

[0971] Step 6:

[0972] The server generates a notification message about food products that are approaching their expiration date and sends it to the terminal.

[0973] (Input) A list of food items that are nearing their expiration date.

[0974] (Output) Informational message.

[0975] (Specific operation) The server generates the content of the push notification (for example, "The expiration date of the milk is three days later") and sends it to the terminal via an API for sending notification messages to smart devices.

[0976] Step 7:

[0977] The terminal notifies the user of the notification message by push notification.

[0978] (Input) The notification message.

[0979] (Output) The notification that is displayed to the user.

[0980] (Specific operation) The notification message received by the terminal is displayed to the user using the notification function of the smart device.

[0981] Step 8:

[0982] The server retrieves the optimum storage method for food items whose expiration date is approaching from a database, generates a notification message, and transmits it to the terminal.

[0983] (Input) A list of foods that are close to expiry.

[0984] (Output) A message informing you of the save method.

[0985] (Specific operation) For example, the server obtains information from the database about how to store milk, such as "place it at the back of the refrigerator to cool it down," and based on this information generates a notification message and sends it to the terminal.

[0986] Step 9:

[0987] The server searches a database for recipes that combine foods whose expiration dates are approaching, selects them, generates a notification message, and transmits it to the terminal.

[0988] (Input) A list of foods that are close to expiry.

[0989] (Output) The recipe notification message.

[0990] (Specific operation) The server searches the database for "recipes that combine foods with close expiration dates" (for example, "cream stew using milk"), generates a notification message based on this, and sends it to the terminal.

[0991] Step 10:

[0992] It uses an emotion engine to recognize the user's emotional state and adjust recipes and notifications accordingly.

[0993] (Input) The user's emotional state.

[0994] (Output) Emotion-based recipes and notification content.

[0995] (Specific operation) The emotion engine analyzes the user's image, voice, and input content to recognize their emotional state, such as fatigue or stress. For example, if the server determines that the user is tired, it generates a notification message suggesting a "simple vegetable stir-fry recipe" and sends it to the device.

[0996] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0997] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0998] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0999] [Third embodiment]

[1000] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1001] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1002] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1003] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1004] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1005] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1006] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1007] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1008] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1009] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1010] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1011] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1012] The present invention relates to a system for managing expiration dates of food products and proposing appropriate storage methods and recipes. Specific embodiments of the system will be described below.

[1013] 1. Acquisition and storage of expiration date data

[1014] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[1015] Specific examples

[1016] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[1017] 2. Best before date tracking and notification

[1018] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[1019] Specific examples

[1020] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[1021] 3. Storage method suggestions

[1022] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[1023] Specific examples

[1024] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[1025] 4. Recipe suggestions

[1026] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[1027] Specific examples

[1028] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[1029] As described above, this system aims to efficiently manage food expiration dates and reduce food waste by suggesting storage methods and recipes, allowing users to maintain food quality while consuming it and leading an economical and environmentally conscious lifestyle.

[1030] The processing flow will be explained below.

[1031] Acquiring and storing expiration date data

[1032] Step 1:

[1033] The user takes a photo of the expiration date of the food they have purchased using their smartphone camera.

[1034] Step 2:

[1035] The terminal transmits the captured image to the server.

[1036] Step 3:

[1037] The server uses image analysis technology (OCR technology) to extract the expiration date information from the image.

[1038] Step 4:

[1039] The server stores the extracted expiration date information in a database.

[1040] Best before date tracking and notification

[1041] Step 1:

[1042] The server periodically checks the expiration date information in the database.

[1043] Step 2:

[1044] The server identifies food items that are nearing their expiration date (e.g., 3 days away).

[1045] Step 3:

[1046] The server generates notification messages corresponding to approaching expiration dates.

[1047] Step 4:

[1048] The terminal receives the notification message from the server and sends a push notification to the user.

[1049] Storage suggestions

[1050] Step 1:

[1051] The server retrieves from the database the storage method for food items that are nearing their expiration date.

[1052] Step 2:

[1053] The server adds information about the storage method to the notification message.

[1054] Step 3:

[1055] The terminal displays a notification message to the user, including instructions on how to save.

[1056] Recipe Suggestions

[1057] Step 1:

[1058] The server searches the database for recipe information based on foods that are close to their expiration date.

[1059] Step 2:

[1060] The server selects an appropriate recipe based on the recipe information.

[1061] Step 3:

[1062] The server generates a notification message regarding the selected recipe.

[1063] Step 4:

[1064] The terminal displays a notification message to the user that includes the selected recipe.

[1065] Example 1

[1066] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1067] In modern society, many consumers waste food. In particular, discarding food that has passed its expiration date causes economic losses and negative impacts on the environment. To solve this problem, a system is needed that can efficiently manage food expiration dates, notify consumers at the appropriate time, and even suggest storage methods and recipes.

[1068] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1069] In this invention, the server includes means for photographing expiration dates of food products with a device camera and sending the image to the server, means for the server to extract date information using image analysis technology and store it in a database, means for the server to periodically check the expiration date information in the database and, when an expiration date is approaching, generate a notification message and send a push notification to the user via the device, means for the server to suggest appropriate storage methods for food products approaching their expiration dates, and means for the server to provide recipes that combine food products approaching their expiration dates. This makes it easier for users to manage expiration dates, reduces food waste, and enables them to live economically and environmentally conscious lives.

[1070] The "device camera" refers to a photographing device installed in a portable electronic device such as a smartphone or tablet that a user uses to photograph the expiration date of food products.

[1071] A "server" is a computer device that uses image analysis technology to extract date information from captured image data, stores it in a database, and periodically checks the information in the database and sends notification messages to users.

[1072] "Image analysis technology" is a technology that extracts text information from image data, including food expiration dates, and generally refers to OCR (Optical Character Recognition) technology.

[1073] A "database" is a structured data storage system for storing information about expiration dates and food products, which is accessed and updated by a server.

[1074] A "notification message" is text information that is generated by the server to inform the user of food products that are approaching their expiration date and is sent to the user via the terminal.

[1075] "Push notification" is a technology for displaying notification messages on a user's device in real time, and is a means of displaying messages on the device's home screen or notification bar.

[1076] "Storage method" refers to the recommended storage method or procedure for extending the expiration date while maintaining the quality of food, and the server searches the database and suggests it to the user.

[1077] A "recipe" is information that indicates steps and combinations of ingredients for creating a dish that efficiently uses food that is approaching its expiration date.

[1078] "Food" refers to ingredients or cooked products that are purchased and consumed by users and have a best-before date.

[1079] The present invention relates to a system that manages the expiration dates of food products and suggests appropriate storage methods and recipes. This system takes a photo of the expiration date using a device camera, performs image analysis on a server, and provides notifications and suggestions based on the expiration date information stored in a database. Specific embodiments are described below.

[1080] 1. Acquisition and storage of expiration date data

[1081] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses OCR technology (e.g., Tesseract OCR) to analyze and extract the expiration date information from the image. This extracted date information is then stored in a database along with the food's information. This saves users the trouble of manually entering the expiration date.

[1082] Examples:

[1083] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[1084] 2. Best before date tracking and notification

[1085] The server periodically checks the expiration date information in the database, identifies foods that are approaching their expiration date, and if so, generates a notification message and sends a push notification to the user via their device, allowing the user to consume the food at the appropriate time without forgetting the expiration date.

[1086] Examples:

[1087] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[1088] 3. Storage method suggestions

[1089] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[1090] Examples:

[1091] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[1092] 4. Recipe suggestions

[1093] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[1094] Examples:

[1095] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[1096] The specific processing flow of this system can be explained by inputting the following prompt sentence into the generative AI model.

[1097] "Please tell me the specific process flow for a system that manages food expiration dates and suggests appropriate storage methods and recipes. Please provide a detailed explanation of each process step, including specific operations."

[1098] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1099] Step 1:

[1100] The user takes a photo of the expiration date of the food product they purchased with their smartphone camera. For example, the user launches the smartphone's camera app and takes a photo of the milk they purchased so that the expiration date "2023-10-15" is displayed on the screen.

[1101] Input: An image of the expiration date taken with a smartphone camera

[1102] Output: Image data of expiration dates stored on a smartphone

[1103] Step 2:

[1104] The device sends the captured images to a server. Specifically, the device uses an Internet connection to upload image data to the server via the HTTPS protocol.

[1105] Input: Image data of expiration date stored on smartphone

[1106] Output: Image data sent to the server

[1107] Step 3:

[1108] The server analyzes the received image using OCR technology (e.g., Tesseract OCR) to extract the expiration date information. The server then runs its internal algorithms to analyze the image and obtain the text data.

[1109] Input: Image data sent to the server

[1110] Output: Extracted best-before date information (e.g. "2023-10-15")

[1111] Step 4:

[1112] The server stores the extracted date information along with the food information in a database. The server inserts the food information (e.g., "Milk", "2023-10-15", "2023-10-01") into the appropriate fields in the database.

[1113] Input: Extracted best-before date information and food information

[1114] Output: Best before date information stored in the database

[1115] Step 5:

[1116] The server periodically checks the expiration date information in the database, for example, as a scheduled job once a day.

[1117] Input: Best before date information in the database

[1118] Output: Checked expiration date information

[1119] Step 6:

[1120] The server identifies foods that are nearing their expiration date and generates a notification message. For example, if the server determines that there are only three days until 2023-10-15 as of 2023-10-12, it generates a message saying, "Milk's expiration date is in three days."

[1121] Input: Checked expiration date information and current date

[1122] Output: The generated notification message

[1123] Step 7:

[1124] The server sends the generated notification message to the device. The device receives the push notification and displays it to the user. For example, a notification saying "The milk's expiration date is in three days" will be displayed on the smartphone screen.

[1125] Input: The generated notification message

[1126] Output: Push notification displayed on device

[1127] Step 8:

[1128] The server retrieves the optimal storage method for food that is nearing its expiration date from the database, searches for the storage method data for the relevant food, and generates a suggestion message.

[1129] Input: Information about food that is close to its expiration date

[1130] Output: Generated save method suggestion message

[1131] Step 9:

[1132] The server sends the generated storage suggestion message to the device, which then displays it to the user via push notification. For example, a notification such as "The milk's expiration date is in three days. Please place it at the back of the refrigerator to keep it cool" may be displayed.

[1133] Input: Generated save method suggestion message

[1134] Output: Push notification with suggested saving methods displayed on the device

[1135] Step 10:

[1136] The server searches the database for recipes that combine food items whose expiration dates are approaching, retrieves the corresponding recipe data, and generates a recipe suggestion message.

[1137] Input: Information about food that is close to its expiration date

[1138] Output: Generated recipe suggestion message

[1139] Step 11:

[1140] The server sends the generated recipe suggestion message to the device, which then displays it to the user via a push notification. For example, a notification such as "Click here for a recipe for cream stew using milk" is displayed.

[1141] Input: Generated recipe suggestion message

[1142] Output: Push notification with recipe suggestions displayed on the device

[1143] (Application example 1)

[1144] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1145] The goal is to provide a system that manages food expiration dates and suggests appropriate storage methods and recipes, allowing users to easily manage expiration dates, suggest storage methods, and search for recipes. In particular, there is a need for a means to automate information input and send notifications at appropriate times. There is also a desire to reduce food waste by suggesting recipes and notifying users of storage methods to efficiently use food that is approaching its expiration date.

[1146] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1147] In this invention, the server includes a means for automatically extracting expiration date information for food products using image analysis technology and managing the data, a means for sending notifications to users using push notification technology when the expiration date approaches, and a means for displaying notifications on a user interface with storage methods and recipes. This not only enables users to efficiently manage expiration dates, but also automatically receives suggestions for appropriate storage methods and recipes that combine foods with approaching expiration dates. Furthermore, the use of notification technology allows users to obtain important information at the appropriate time, thereby reducing food waste.

[1148] "Means for photographing food expiration dates and saving the data" is a function that allows users to use a device such as a smartphone to photograph the expiration date displayed on food and save the data.

[1149] The "means for automatically notifying when the expiration date is approaching" is a function that automatically sends a warning or alert to the user when the expiration date is approaching based on stored expiration date data.

[1150] The "means for suggesting food preservation methods" is a function for suggesting to the user how to properly preserve food that is approaching its expiration date.

[1151] The "means for providing recipes that combine food items with close expiration dates" is a function for providing users with recipes that use multiple food items with close expiration dates.

[1152] "Means for automatically extracting expiration date information using image analysis technology and managing the data" refers to a function that converts photographed expiration date information into text data using image analysis technology such as OCR technology and manages the data.

[1153] "Means for sending notifications to users using push notification technology" refers to a function that uses cloud services, etc. to notify users' devices in real time that the expiration date is approaching.

[1154] The "means for displaying notifications on a user interface" is a function for displaying storage methods and recipe suggestions on the interface on the user's device.

[1155] The present invention relates to a system for managing expiration dates of food products and proposing appropriate storage methods and recipes. Specific embodiments of the system will be described below.

[1156] First, when a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[1157] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[1158] Furthermore, for food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[1159] The server also searches the database for recipe information to suggest recipes that combine foods that are approaching their expiration dates. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[1160] Specifically, the hardware used includes smartphones and servers, and the software used includes Tesseract OCR, which implements OCR technology, and Firebase Cloud Messaging, which provides push notification technology.

[1161] For example, if a user purchases milk and takes a photo of its expiration date, "2023-10-15," with their smartphone, the device sends this image to a server. The server uses OCR technology to extract the expiration date, "2023-10-15," from the image and stores it in a database. If the server checks the milk's expiration date, "2023-10-15," in the database and determines that the current date is "2023-10-12," the server generates a notification message stating, "The milk's expiration date is three days from now," and sends this push notification to the user via their device. It also generates a notification message suggesting a storage method, "Put it at the back of the refrigerator to cool," and a recipe, "Cream stew."

[1162] An example of a prompt sentence for the generative AI model is written as follows:

[1163] Extract the expiration date from the image below:

[1164] Image path: path_to_image.jpg

[1165] If the expiration date is "2023-10-15", generate a notification message suggesting the best way to store the food, "Put it at the back of the refrigerator to cool it down", and also suggest the recipe "Cream Stew".

[1166] As described above, this system aims to efficiently manage food expiration dates and reduce food waste by suggesting storage methods and recipes, allowing users to maintain food quality while consuming it and leading an economical and environmentally conscious lifestyle.

[1167] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1168] Step 1:

[1169] A user takes a photo of a food product's expiration date with a smartphone camera. The input is an image of the expiration date printed on the food's packaging. The output is the captured image data.

[1170] Step 2:

[1171] The device sends the captured image to the server. The input is the image data captured by the smartphone. The output is the image data transferred to the server.

[1172] Step 3:

[1173] The server uses OCR technology to extract the expiration date information from the image. The input is the received image data. The data processing is image analysis using Tesseract OCR. The output is the extracted expiration date string.

[1174] Step 4:

[1175] The extracted date information is saved in a database. The input is the date string obtained by OCR analysis. The data processing is writing the date string to the database. The output is a database entry with the expiration date saved.

[1176] Step 5:

[1177] The server periodically checks the expiration date information in the database. The input is the expiration date information of all food items stored in the database. The data calculation is a comparison with the current date. The output is a list of food items that are approaching their expiration date.

[1178] Step 6:

[1179] Identify foods approaching their expiration date and generate a notification message. The input is a list of foods approaching their expiration date. The data processing is the generation of the notification message. The output is the generated notification message.

[1180] Step 7:

[1181] The server sends the generated notification message to the device via Firebase Cloud Messaging. The input is the generated notification message. The output is the notification sent to the user's smartphone.

[1182] Step 8:

[1183] The device notifies the user via a push notification. The input is the notification message received from the server. The output is the push notification displayed to the user.

[1184] Step 9:

[1185] For foods approaching their expiration date, the server retrieves the optimal storage method from the database. The input is a list of foods approaching their expiration date. The data operation is a query from the storage method database. The output is the retrieved storage method.

[1186] Step 10:

[1187] A notification message is generated based on the storage method and notified to the user via the terminal. The input is the storage method. Data processing is the generation of the storage method notification message. The output is the storage method notification message.

[1188] Step 11:

[1189] To suggest recipes that combine foods with close expiration dates, the server searches for recipe information in a database. The input is a list of foods with close expiration dates. The data operation is a query from the recipe database. The output is a suitable recipe.

[1190] Step 12:

[1191] Select the appropriate recipe and generate a notification message. The input is the retrieved recipe information. The data processing is the generation of the recipe notification message. The output is the generated recipe notification message.

[1192] Step 13:

[1193] The terminal displays the generated recipe notification message to the user. The input is the recipe notification message from the server. The output is the recipe suggestion notification displayed to the user.

[1194] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1195] The present invention combines an emotion engine with a system that manages food expiration dates and suggests appropriate storage methods and recipes, providing a service based on the user's emotions. Specific embodiments of this system are described below.

[1196] 1. Acquisition and storage of expiration date data

[1197] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[1198] Specific examples

[1199] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[1200] 2. Best before date tracking and notification

[1201] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[1202] Specific examples

[1203] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[1204] 3. Storage method suggestions

[1205] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[1206] Specific examples

[1207] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[1208] 4. Recipe suggestions

[1209] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[1210] Specific examples

[1211] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[1212] 5. Providing services using emotion engines

[1213] 5.1 Emotion Recognition and Recipe Adjustment

[1214] The server uses an emotion engine to recognize the user's emotional state and adjusts the content of recipes and notifications depending on the user's emotions. For example, if the user is tired, it will suggest easy-to-make recipes.

[1215] Specific examples

[1216] The server uses an emotion engine to recognize that the user is tired, and then suggests a recipe for "stir-fried vegetables" as an "easy recipe recommended for when you're tired" and notifies the user via their device.

[1217] 5.2 Recipe suggestions based on tastes and preferences

[1218] The emotion engine determines the user's tastes and preferences and suggests recipes based on them, making it easier for users to cook their favorite dishes.

[1219] Specific examples

[1220] The emotion engine analyzes the user's preferences based on past data, and the server suggests a recipe for "Japanese-style pasta" as "a dish with the user's favorite seasoning." The device notifies the user of this information.

[1221] 5.3 Food suggestions for stress reduction

[1222] The emotion engine assesses the user's stress level and suggests foods and recipes that have a stress-reducing effect.

[1223] Specific examples

[1224] The emotion engine recognizes that the user is feeling stressed, and the server suggests a "relaxing herbal tea recipe." The device notifies the user of this information.

[1225] As described above, this system aims not only to efficiently manage food expiration dates and reduce food waste by suggesting storage methods and recipes, but also to provide services that respond to the user's emotional state using an emotion engine, thereby enabling users to live more comfortable and efficient lives.

[1226] The processing flow will be explained below.

[1227] Acquiring and storing expiration date data

[1228] Step 1:

[1229] The user takes a photo of the expiration date of the food they have purchased using their smartphone camera.

[1230] Step 2:

[1231] The terminal transmits the captured image to the server.

[1232] Step 3:

[1233] The server uses image analysis technology (OCR technology) to extract the expiration date information from the image.

[1234] Step 4:

[1235] The server stores the extracted expiration date information in a database.

[1236] Best before date tracking and notification

[1237] Step 1:

[1238] The server periodically checks the expiration date information in the database.

[1239] Step 2:

[1240] The server identifies food items that are nearing their expiration date (e.g., 3 days away).

[1241] Step 3:

[1242] The server generates notification messages corresponding to approaching expiration dates.

[1243] Step 4:

[1244] The terminal receives the notification message from the server and sends a push notification to the user.

[1245] Storage suggestions

[1246] Step 1:

[1247] The server retrieves from the database the storage method for food that is nearing its expiration date.

[1248] Step 2:

[1249] The server adds information about the storage method to the notification message.

[1250] Step 3:

[1251] The terminal displays a notification message to the user including the saving method.

[1252] Recipe Suggestions

[1253] Step 1:

[1254] The server searches the database for recipe information based on foods with an approaching expiration date.

[1255] Step 2:

[1256] The server selects an appropriate recipe based on the recipe information.

[1257] Step 3:

[1258] The server generates a notification message about the selected recipe.

[1259] Step 4:

[1260] The terminal displays a notification message to the user containing the selected recipe.

[1261] Providing services using emotion engines

[1262] Emotion recognition and recipe adjustment

[1263] Step 1:

[1264] The server uses an emotion engine to recognize the user's emotional state.

[1265] Step 2:

[1266] The server selects recipes and notification messages according to the user's emotions.

[1267] Step 3:

[1268] The server generates the selected recipe and notification message.

[1269] Step 4:

[1270] The device displays recipes and notification messages to the user based on their emotions.

[1271] Recipe suggestions based on your tastes and preferences

[1272] Step 1:

[1273] The server uses an emotion engine to determine the user's preferences and tastes.

[1274] Step 2:

[1275] The server retrieves recipes based on the user's preferences.

[1276] Step 3:

[1277] The server selects recipes based on preferences and generates notification messages.

[1278] Step 4:

[1279] The device displays a notification message to the user containing a recipe based on the preferences.

[1280] Food suggestions for stress reduction

[1281] Step 1:

[1282] The server uses an emotion engine to determine whether the user is experiencing stress.

[1283] Step 2:

[1284] The server searches for foods and recipes that have stress-reducing effects.

[1285] Step 3:

[1286] The server selects foods and recipes for stress reduction and generates a notification message.

[1287] Step 4:

[1288] The device will display a notification message to the user containing stress-reducing foods and recipes.

[1289] Example 2

[1290] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1291] Current food management systems require users to manually input expiration dates, making expiration date management cumbersome. Furthermore, storage methods and recipe suggestions are uniform and are not based on the user's emotional state or preferences, resulting in low user convenience. Furthermore, if expiration date notifications are not properly provided, there is a high possibility of food waste.

[1292] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for photographing the expiration date of food and saving it as data, a means for automatically notifying when the expiration date is approaching, a means for suggesting a food preservation method, a means for providing a recipe that combines foods with close expiration dates, and a means for recognizing the user's emotional state and adjusting the notification and suggestion content based on the emotional state. This enables efficient management of expiration dates, suggestion of appropriate preservation methods, and suggestion of recipes that correspond to the user's emotions.

[1293] "Means for photographing expiration dates and saving them as data" refers to a means by which a user photographs the expiration date of a food product using a smartphone camera, sends the image to a server, and uses image analysis technology (OCR technology) to extract the date information and save it in a database.

[1294] "Means for automatically notifying when the expiration date is approaching" refers to a means in which the server periodically checks the expiration date information in the database, generates a notification message when the expiration date is approaching, and pushes the message to the user via the terminal.

[1295] The "means for suggesting food preservation methods" refers to a means by which the server retrieves the optimal preservation method from a database for food that is approaching its expiration date, generates a notification message based on that information, and notifies the user via the terminal.

[1296] The "means for providing recipes that combine foods with close expiration dates" refers to a means by which the server searches a database for recipe information based on foods with close expiration dates, selects an appropriate recipe, generates a notification message, and displays it to the user via the terminal.

[1297] "Means for recognizing the user's emotional state and adjusting notifications and suggestions based on that" refers to a means by which the server recognizes the user's emotional state using an emotion engine and adjusts notifications and suggestions based on the user's emotions. Specifically, this includes adjustments such as suggesting easy-to-make recipes when the user is tired.

[1298] The present invention combines an emotion engine with a system that manages food expiration dates and suggests appropriate storage methods and recipes, providing a service based on the user's emotions. Specific embodiments of this system are described below.

[1299] Acquiring and storing expiration date data

[1300] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[1301] Specific examples

[1302] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[1303] Best before date tracking and notification

[1304] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[1305] Specific examples

[1306] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[1307] Storage suggestions

[1308] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[1309] Specific examples

[1310] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[1311] Recipe Suggestions

[1312] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[1313] Specific examples

[1314] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[1315] Providing services using emotion engines

[1316] Emotion recognition and recipe adjustment

[1317] The server uses an emotion engine to recognize the user's emotional state and adjusts the content of recipes and notifications depending on the user's emotions. For example, if the user is tired, it will suggest easy-to-make recipes.

[1318] Specific examples

[1319] The server uses an emotion engine to recognize that the user is tired, and then suggests a recipe for "stir-fried vegetables" as an "easy recipe recommended for when you're tired" and notifies the user via their device.

[1320] Recipe suggestions based on your tastes and preferences

[1321] The emotion engine determines the user's tastes and preferences and suggests recipes based on them, making it easier for users to cook their favorite dishes.

[1322] Specific examples

[1323] The emotion engine analyzes the user's preferences based on past data, and the server suggests a recipe for "Japanese-style pasta" as "a dish with the user's favorite seasoning." The device notifies the user of this information.

[1324] Food suggestions for stress reduction

[1325] The emotion engine assesses the user's stress level and suggests foods and recipes that have a stress-reducing effect.

[1326] Specific examples

[1327] The emotion engine recognizes that the user is feeling stressed, and the server suggests a "relaxing herbal tea recipe." The device notifies the user of this information.

[1328] Prompt Sentence Examples

[1329] "Please suggest recipes that efficiently use food in the refrigerator that is nearing its expiration date."

[1330] "Please suggest easy dinner recipes for when the user is tired."

[1331] "Provide new recipes based on your users' favorite flavors."

[1332] With this system, users can efficiently manage food expiration dates and receive suggestions for storage methods and recipes. Furthermore, by using an emotion engine, it is possible to provide services that correspond to the user's emotional state, improving the quality of daily life.

[1333] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1334] Step 1:

[1335] A user takes a photo of the expiration date on a food item

[1336] explanation:

[1337] The user uses the smartphone camera to take a photo of the expiration date on food. To do this, they launch the camera app, frame the food label, and take a photo. The input is the image data taken with the smartphone. The output is the image data taken.

[1338] Step 2:

[1339] The device sends the image to the server

[1340] explanation:

[1341] The device sends the image data captured in step 1 to the server. This transmission is performed via an HTTP request. The input is the captured image data. The output is the image data sent to the server.

[1342] Step 3:

[1343] The server extracts the date information from the image.

[1344] explanation:

[1345] The server processes the received image data and extracts the expiration date information as text using OCR technology (e.g., Tesseract OCR). The server passes the image data to the OCR engine to obtain the date information. The input is the image data sent to the server. The output is the extracted date information in text format.

[1346] Step 4:

[1347] The server stores the extracted information in a database

[1348] explanation:

[1349] The server stores the date information extracted in step 3 in a database, possibly along with additional information such as the name of the food item and the date of purchase. The input is the extracted date information and other relevant information. The output is the information stored in the database.

[1350] Step 5:

[1351] The server periodically checks the database

[1352] explanation:

[1353] The server periodically (e.g., every night at midnight) executes a batch process that checks expiration date information in the database. The input is the expiration date information in the database. The output is the checked expiration date information.

[1354] Step 6:

[1355] Identify approaching expiration dates on your servers

[1356] explanation:

[1357] The server checks each record in the database and calculates the difference between the current date and the expiration date, thereby identifying foods that are nearing their expiration date. The input is the expiration date information being checked and the current date. The output is the approaching expiration date information.

[1358] Step 7:

[1359] The server generates a notification message

[1360] explanation:

[1361] The server generates a notification message for the food item identified in step 6 that is nearing its expiration date. For example, it creates a message that reads "Milk will expire in 3 days." The input is the upcoming expiration date information. The output is the generated notification message.

[1362] Step 8:

[1363] The device sends a push notification to the user

[1364] explanation:

[1365] The server sends the notification message generated in step 7 to the device. The device displays this message to the user as a push notification. The input is the generated notification message. The output is the notification message displayed to the user.

[1366] Step 9:

[1367] The server retrieves the save method.

[1368] explanation:

[1369] The server retrieves the optimal storage method from the database for food items that are nearing their expiration date. For example, for milk, it retrieves information such as "keep it at the back of the refrigerator to keep it cool." The input is information about food items that are nearing their expiration date. The output is the retrieved storage method information.

[1370] Step 10:

[1371] The server generates a message informing you of the save method.

[1372] explanation:

[1373] The server generates a notification message based on the storage method information obtained in step 9. For example, it creates a message saying, "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator and keep it cool." The input is the obtained storage method information. The output is the generated notification message.

[1374] Step 11:

[1375] The device sends a notification to the user

[1376] explanation:

[1377] The server sends the notification message generated in step 10 to the terminal, and the terminal notifies the user of the information. The input is the generated notification message. The output is the notification message displayed to the user.

[1378] Step 12:

[1379] The server searches the database for the recipe

[1380] explanation:

[1381] The server searches the database for recipes that use foods that are close to their expiration date. For example, it finds a recipe for "cream stew" that uses milk. The input is information about foods that are close to their expiration date. The output is the searched recipe information.

[1382] Step 13:

[1383] The server generates a notification message with the appropriate recipe

[1384] explanation:

[1385] The server generates a notification message based on the recipe information retrieved in step 12. For example, it creates a message that reads, "Click here for a recipe for cream stew using milk." The input is the retrieved recipe information. The output is the generated notification message.

[1386] Step 14:

[1387] The device sends a notification to the user

[1388] explanation:

[1389] The server sends the notification message generated in step 13 to the terminal, and the terminal notifies the user of the information. The input is the generated notification message. The output is the notification message displayed to the user.

[1390] Step 15:

[1391] The server uses an emotion engine to recognize the user's emotional state.

[1392] explanation:

[1393] The server uses an emotion engine to recognize the user's emotional state. The input is the user's past data and current interaction information. The output is the recognized user's emotional state.

[1394] Step 16:

[1395] The server tailors notifications and suggestions based on emotions

[1396] explanation:

[1397] The server adjusts notifications and suggestions based on the emotional state recognized in step 15. For example, if it recognizes that the user is tired, it may suggest easy-to-make recipes. The input is the recognized emotional state of the user and the desired information. The output is the adjusted notifications and suggestions.

[1398] Step 17:

[1399] The server generates emotion-based recipe notification messages.

[1400] explanation:

[1401] The server generates a notification message based on the suggestions adjusted in step 16. For example, it creates a message such as "Easy recipes for when you're tired: Stir-fried vegetables." The input is the adjusted notification and suggestions. The output is the generated notification message.

[1402] Step 18:

[1403] The device notifies the user with emotion-based suggestions

[1404] explanation:

[1405] The server sends the notification message generated in step 17 to the terminal, and the terminal notifies the user of the information. The input is the generated notification message. The output is the notification message displayed to the user.

[1406] This allows users to efficiently manage expiration dates, receive suggestions for appropriate storage methods, and receive notifications and recipe suggestions based on the user's emotions.

[1407] (Application example 2)

[1408] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1409] In recent years, managing food expiration dates has become an important issue in homes and brick-and-mortar stores. However, efficiently managing expiration dates and reducing waste requires a lot of effort and cost. Furthermore, when users and staff are in an emotional state, it is necessary to further improve convenience and satisfaction by suggesting appropriate storage methods and recipes. Current technology does not provide such a comprehensive system, so a new approach is needed to improve users' quality of life.

[1410] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1411] In this invention, the server includes means for photographing expiration dates of food products and saving the photographed data, means for automatically notifying users when the expiration date is approaching, means for suggesting food preservation methods, means for providing recipes that combine foods with approaching expiration dates, means for photographing expiration dates using a smart device, and means for adjusting notification content and recipes based on the user's emotional state using an emotion engine. This makes it possible to efficiently manage expiration dates and make appropriate suggestions according to the user's emotions.

[1412] "Means for photographing food expiration dates and saving them as data" refers to a means for a user to photograph the expiration date of food purchased by the user using the camera on their smart device, analyze the image, and save the date information in a database.

[1413] The "means for automatically notifying when the expiration date is approaching" is a means for the server to periodically analyze the stored expiration date data and notify the user about food products whose expiration date is approaching.

[1414] The "means for suggesting a food preservation method" is a means for retrieving the optimum preservation method for food that is approaching its expiration date from a database and notifying the user of the method.

[1415] The "means for providing recipes that combine foods with close expiration dates" refers to a means for the server to search a database for recipes that combine foods with close expiration dates based on the expiration date information of the foods stored on the server, and suggest them to the user.

[1416] "Means for photographing expiration dates using a smart device" refers to means for photographing expiration dates on food products using a smart device such as a smartphone, smart glasses, or a head-mounted display.

[1417] "Means for adjusting notification content and recipes based on the user's emotional state using an emotion engine" refers to means for analyzing the user's emotional state using emotion recognition technology and adjusting the notification and recipe content based on the results.

[1418] As an embodiment of the present invention, a system for managing expiration dates of food products and suggesting appropriate storage methods and recipes includes the following processing steps.

[1419] The overall system configuration uses smart devices (smartphones, smart glasses, head-mounted displays, etc.) to manage food expiration dates. Specifically, when a user purchases food, they use the smart device's camera to take a photo of the expiration date, and the device sends the image to a server. The server then uses image analysis technology (OCR technology) to extract the expiration date information and saves it in a database.

[1420] Examples of devices used include smartphones and smart glasses. Examples of smart glasses include commonly used commercial products (e.g., Google Glass). The software uses Python, OpenCV for image processing, and pytesseract for OCR analysis. The emotion engine uses machine learning models and natural language processing techniques to analyze the user's emotional state.

[1421] The server periodically checks the expiration date information in the database, generates a notification message for food items whose expiration date is approaching, and sends it to the device. The device then notifies the user of this notification message via push notification. This allows the user to consume the food at the appropriate time without forgetting the expiration date.

[1422] Furthermore, the server retrieves the optimal storage method for food items approaching their expiration date from the database, generates a notification message suggesting that storage method, and notifies the user via the terminal. This allows the user to store the food properly and maintain its quality. The server also searches for recipe information in the database to suggest recipes that combine foods approaching their expiration date, selects appropriate recipes, generates a notification message, and displays it to the user via the terminal.

[1423] The emotion engine is used to recognize the user's emotional state. For example, when a user takes a picture while wearing the smart glasses, the emotion engine analyzes the user's facial expressions and evaluates the user's emotional state. Based on this, the server will adjust the content of recipes and notifications according to the user's emotions.

[1424] As a concrete example, if a user photographs the expiration date of milk with smart glasses, the system automatically analyzes the expiration date "2023-10-15" and stores it in a database. When the expiration date approaches, the server generates a notification stating "The milk's expiration date is in three days" and sends it as a push notification to the user. It also suggests storing the milk at the back of the refrigerator to keep it cool. If the emotion engine recognizes that the user is tired, it suggests a recipe for "stir-fried vegetables" as an "easy recipe recommended for when you're tired."

[1425] An example of a prompt for the generative AI model is as follows:

[1426] Example prompt sentence:

[1427] "Create a Python program that analyzes images of food taken with smart glasses and automatically extracts the expiration date. If the expiration date is approaching, it generates a message suggesting storage methods or recipes, and uses an emotion engine to respond according to the user's emotions. However, it should have conditions that allow the emotion engine to recognize fatigue, stress, or other specific emotional states. Please explain with a concrete example using smart glasses and Python."

[1428] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1429] Step 1:

[1430] A user takes a photo of the expiration date of a food product using a smart device (smartphone, smart glasses, etc.).

[1431] (Input) The image of the food to be photographed.

[1432] (Output) Food image data including expiration date.

[1433] (Specific Operation) For example, a user puts on smart glasses and takes a picture of the food label with the expiration date written on it. At this time, the camera function of the smart device is used.

[1434] Step 2:

[1435] The device sends the photographed image of the food to the server.

[1436] (Input) Food image data including expiration dates.

[1437] (Output) Image data sent to the server.

[1438] (Specific operation) Using the communication function built into the smart device, the image data of the food taken by the user is transferred to a server on the cloud, using Wi-Fi or mobile data communication technology.

[1439] Step 3:

[1440] The server uses image analysis technology (OCR technology) to extract the expiration date information from the transmitted image.

[1441] (Input) Image data of the food sent to the server.

[1442] (Output) The extracted expiration date information.

[1443] (Specific operation) The server analyzes the image using Python and OpenCV, and extracts the expiration date information (e.g., "2023-10-15") from the image by performing OCR processing using pytesseract.

[1444] Step 4:

[1445] The server stores the extracted expiration date information in a database.

[1446] (Input) The extracted expiration date information.

[1447] (Output) Best before date data stored in the database.

[1448] (Specific operation) The server saves the extracted date information in a database (e.g., MySQL or SQLite) using an SQL query.

[1449] Step 5:

[1450] The server periodically checks the expiration date information in the database.

[1451] (Input) Best before date data in the database.

[1452] (Output) A list of foods that are nearing their expiration date.

[1453] (Specific Operation) The server periodically queries all expiration date data in the database using a scheduler to identify items whose expiration dates will arrive within a certain period from the current date.

[1454] Step 6:

[1455] The server generates a notification message about food products that are approaching their expiration date and sends it to the terminal.

[1456] (Input) A list of food items that are nearing their expiration date.

[1457] (Output) Informational message.

[1458] (Specific operation) The server generates the content of the push notification (for example, "The expiration date of the milk is three days later") and sends it to the terminal via an API for sending notification messages to smart devices.

[1459] Step 7:

[1460] The terminal notifies the user of the notification message by push notification.

[1461] (Input) The notification message.

[1462] (Output) The notification that is displayed to the user.

[1463] (Specific operation) The notification message received by the terminal is displayed to the user using the notification function of the smart device.

[1464] Step 8:

[1465] The server retrieves the optimum storage method for food items whose expiration date is approaching from a database, generates a notification message, and transmits it to the terminal.

[1466] (Input) A list of foods that are close to expiry.

[1467] (Output) A message informing you of the save method.

[1468] (Specific operation) For example, the server obtains information from the database about how to store milk, such as "place it at the back of the refrigerator to cool it down," and based on this information generates a notification message and sends it to the terminal.

[1469] Step 9:

[1470] The server searches a database for recipes that combine foods whose expiration dates are approaching, selects them, generates a notification message, and transmits it to the terminal.

[1471] (Input) A list of foods that are close to expiry.

[1472] (Output) The recipe notification message.

[1473] (Specific operation) The server searches the database for "recipes that combine foods with close expiration dates" (for example, "cream stew using milk"), generates a notification message based on this, and sends it to the terminal.

[1474] Step 10:

[1475] It uses an emotion engine to recognize the user's emotional state and adjust recipes and notifications accordingly.

[1476] (Input) The user's emotional state.

[1477] (Output) Emotion-based recipes and notification content.

[1478] (Specific operation) The emotion engine analyzes the user's image, voice, and input content to recognize their emotional state, such as fatigue or stress. For example, if the server determines that the user is tired, it generates a notification message suggesting a "simple vegetable stir-fry recipe" and sends it to the device.

[1479] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1480] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1481] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1482] [Fourth embodiment]

[1483] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1484] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1485] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1486] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1487] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1488] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1489] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1490] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1491] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1492] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1493] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1494] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1495] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1496] The present invention relates to a system for managing expiration dates of food products and proposing appropriate storage methods and recipes. Specific embodiments of the system will be described below.

[1497] 1. Acquisition and storage of expiration date data

[1498] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[1499] Specific examples

[1500] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[1501] 2. Best before date tracking and notification

[1502] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[1503] Specific examples

[1504] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[1505] 3. Storage method suggestions

[1506] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[1507] Specific examples

[1508] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[1509] 4. Recipe suggestions

[1510] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[1511] Specific examples

[1512] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[1513] As described above, this system aims to efficiently manage food expiration dates and reduce food waste by suggesting storage methods and recipes, allowing users to maintain food quality while consuming it and leading an economical and environmentally conscious lifestyle.

[1514] The processing flow will be explained below.

[1515] Acquiring and storing expiration date data

[1516] Step 1:

[1517] The user takes a photo of the expiration date of the food they have purchased using their smartphone camera.

[1518] Step 2:

[1519] The terminal transmits the captured image to the server.

[1520] Step 3:

[1521] The server uses image analysis technology (OCR technology) to extract the expiration date information from the image.

[1522] Step 4:

[1523] The server stores the extracted expiration date information in a database.

[1524] Best before date tracking and notification

[1525] Step 1:

[1526] The server periodically checks the expiration date information in the database.

[1527] Step 2:

[1528] The server identifies food items that are nearing their expiration date (e.g., 3 days away).

[1529] Step 3:

[1530] The server generates notification messages corresponding to approaching expiration dates.

[1531] Step 4:

[1532] The terminal receives the notification message from the server and sends a push notification to the user.

[1533] Storage suggestions

[1534] Step 1:

[1535] The server retrieves from the database the storage method for food items that are nearing their expiration date.

[1536] Step 2:

[1537] The server adds information about the storage method to the notification message.

[1538] Step 3:

[1539] The terminal displays a notification message to the user, including instructions on how to save.

[1540] Recipe Suggestions

[1541] Step 1:

[1542] The server searches the database for recipe information based on foods that are close to their expiration date.

[1543] Step 2:

[1544] The server selects an appropriate recipe based on the recipe information.

[1545] Step 3:

[1546] The server generates a notification message regarding the selected recipe.

[1547] Step 4:

[1548] The terminal displays a notification message to the user that includes the selected recipe.

[1549] Example 1

[1550] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1551] In modern society, many consumers waste food. In particular, discarding food that has passed its expiration date causes economic losses and negative impacts on the environment. To solve this problem, a system is needed that can efficiently manage food expiration dates, notify consumers at the appropriate time, and even suggest storage methods and recipes.

[1552] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1553] In this invention, the server includes means for photographing expiration dates of food products with a device camera and sending the image to the server, means for the server to extract date information using image analysis technology and store it in a database, means for the server to periodically check the expiration date information in the database and, when an expiration date is approaching, generate a notification message and send a push notification to the user via the device, means for the server to suggest appropriate storage methods for food products approaching their expiration dates, and means for the server to provide recipes that combine food products approaching their expiration dates. This makes it easier for users to manage expiration dates, reduces food waste, and enables them to live economically and environmentally conscious lives.

[1554] The "device camera" refers to a photographing device installed in a portable electronic device such as a smartphone or tablet that a user uses to photograph the expiration date of food products.

[1555] A "server" is a computer device that uses image analysis technology to extract date information from captured image data, stores it in a database, and periodically checks the information in the database and sends notification messages to users.

[1556] "Image analysis technology" is a technology that extracts text information from image data, including food expiration dates, and generally refers to OCR (Optical Character Recognition) technology.

[1557] A "database" is a structured data storage system for storing information about expiration dates and food products, which is accessed and updated by a server.

[1558] A "notification message" is text information that is generated by the server to inform the user of food products that are approaching their expiration date and is sent to the user via the terminal.

[1559] "Push notification" is a technology for displaying notification messages on a user's device in real time, and is a means of displaying messages on the device's home screen or notification bar.

[1560] "Storage method" refers to the recommended storage method or procedure for extending the expiration date while maintaining the quality of food, and the server searches the database and suggests it to the user.

[1561] A "recipe" is information that indicates steps and combinations of ingredients for creating a dish that efficiently uses food that is approaching its expiration date.

[1562] "Food" refers to ingredients or cooked products that are purchased and consumed by users and have a best-before date.

[1563] The present invention relates to a system that manages the expiration dates of food products and suggests appropriate storage methods and recipes. This system takes a photo of the expiration date using a device camera, performs image analysis on a server, and provides notifications and suggestions based on the expiration date information stored in a database. Specific embodiments are described below.

[1564] 1. Acquisition and storage of expiration date data

[1565] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses OCR technology (e.g., Tesseract OCR) to analyze and extract the expiration date information from the image. This extracted date information is then stored in a database along with the food's information. This saves users the trouble of manually entering the expiration date.

[1566] Examples:

[1567] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[1568] 2. Best before date tracking and notification

[1569] The server periodically checks the expiration date information in the database, identifies foods that are approaching their expiration date, and if so, generates a notification message and sends a push notification to the user via their device, allowing the user to consume the food at the appropriate time without forgetting the expiration date.

[1570] Examples:

[1571] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[1572] 3. Storage method suggestions

[1573] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[1574] Examples:

[1575] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[1576] 4. Recipe suggestions

[1577] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[1578] Examples:

[1579] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[1580] The specific processing flow of this system can be explained by inputting the following prompt sentence into the generative AI model.

[1581] "Please tell me the specific process flow for a system that manages food expiration dates and suggests appropriate storage methods and recipes. Please provide a detailed explanation of each process step, including specific operations."

[1582] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1583] Step 1:

[1584] The user takes a photo of the expiration date of the food product they purchased with their smartphone camera. For example, the user launches the smartphone's camera app and takes a photo of the milk they purchased so that the expiration date "2023-10-15" is displayed on the screen.

[1585] Input: An image of the expiration date taken with a smartphone camera

[1586] Output: Image data of expiration dates stored on a smartphone

[1587] Step 2:

[1588] The device sends the captured images to a server. Specifically, the device uses an Internet connection to upload image data to the server via the HTTPS protocol.

[1589] Input: Image data of expiration date stored on smartphone

[1590] Output: Image data sent to the server

[1591] Step 3:

[1592] The server analyzes the received image using OCR technology (e.g., Tesseract OCR) to extract the expiration date information. The server then runs its internal algorithms to analyze the image and obtain the text data.

[1593] Input: Image data sent to the server

[1594] Output: Extracted best-before date information (e.g. "2023-10-15")

[1595] Step 4:

[1596] The server stores the extracted date information along with the food information in a database. The server inserts the food information (e.g., "Milk", "2023-10-15", "2023-10-01") into the appropriate fields in the database.

[1597] Input: Extracted best-before date information and food information

[1598] Output: Best before date information stored in the database

[1599] Step 5:

[1600] The server periodically checks the expiration date information in the database, for example, as a scheduled job once a day.

[1601] Input: Best before date information in the database

[1602] Output: Checked expiration date information

[1603] Step 6:

[1604] The server identifies foods that are nearing their expiration date and generates a notification message. For example, if the server determines that there are only three days until 2023-10-15 as of 2023-10-12, it generates a message saying, "Milk's expiration date is in three days."

[1605] Input: Checked expiration date information and current date

[1606] Output: The generated notification message

[1607] Step 7:

[1608] The server sends the generated notification message to the device. The device receives the push notification and displays it to the user. For example, a notification saying "The milk's expiration date is in three days" will be displayed on the smartphone screen.

[1609] Input: The generated notification message

[1610] Output: Push notification displayed on device

[1611] Step 8:

[1612] The server retrieves the optimal storage method for food that is nearing its expiration date from the database, searches for the storage method data for the relevant food, and generates a suggestion message.

[1613] Input: Information about food that is close to its expiration date

[1614] Output: Generated save method suggestion message

[1615] Step 9:

[1616] The server sends the generated storage suggestion message to the device, which then displays it to the user via push notification. For example, a notification such as "The milk's expiration date is in three days. Please place it at the back of the refrigerator to keep it cool" may be displayed.

[1617] Input: Generated save method suggestion message

[1618] Output: Push notification with suggested saving methods displayed on the device

[1619] Step 10:

[1620] The server searches the database for recipes that combine food items whose expiration dates are approaching, retrieves the corresponding recipe data, and generates a recipe suggestion message.

[1621] Input: Information about food that is close to its expiration date

[1622] Output: Generated recipe suggestion message

[1623] Step 11:

[1624] The server sends the generated recipe suggestion message to the device, which then displays it to the user via a push notification. For example, a notification such as "Click here for a recipe for cream stew using milk" is displayed.

[1625] Input: Generated recipe suggestion message

[1626] Output: Push notification with recipe suggestions displayed on the device

[1627] (Application example 1)

[1628] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1629] The goal is to provide a system that manages food expiration dates and suggests appropriate storage methods and recipes, allowing users to easily manage expiration dates, suggest storage methods, and search for recipes. In particular, there is a need for a means to automate information input and send notifications at appropriate times. There is also a desire to reduce food waste by suggesting recipes and notifying users of storage methods to efficiently use food that is approaching its expiration date.

[1630] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1631] In this invention, the server includes a means for automatically extracting expiration date information for food products using image analysis technology and managing the data, a means for sending notifications to users using push notification technology when the expiration date approaches, and a means for displaying notifications on a user interface with storage methods and recipes. This not only enables users to efficiently manage expiration dates, but also automatically receives suggestions for appropriate storage methods and recipes that combine foods with approaching expiration dates. Furthermore, the use of notification technology allows users to obtain important information at the appropriate time, thereby reducing food waste.

[1632] "Means for photographing food expiration dates and saving the data" is a function that allows users to use a device such as a smartphone to photograph the expiration date displayed on food and save the data.

[1633] The "means for automatically notifying when the expiration date is approaching" is a function that automatically sends a warning or alert to the user when the expiration date is approaching based on stored expiration date data.

[1634] The "means for suggesting food preservation methods" is a function for suggesting to the user how to properly preserve food that is approaching its expiration date.

[1635] The "means for providing recipes that combine food items with close expiration dates" is a function for providing users with recipes that use multiple food items with close expiration dates.

[1636] "Means for automatically extracting expiration date information using image analysis technology and managing the data" refers to a function that converts photographed expiration date information into text data using image analysis technology such as OCR technology and manages the data.

[1637] "Means for sending notifications to users using push notification technology" refers to a function that uses cloud services, etc. to notify users' devices in real time that the expiration date is approaching.

[1638] The "means for displaying notifications on a user interface" is a function for displaying storage methods and recipe suggestions on the interface on the user's device.

[1639] The present invention relates to a system for managing expiration dates of food products and proposing appropriate storage methods and recipes. Specific embodiments of the system will be described below.

[1640] First, when a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[1641] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[1642] Furthermore, for food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[1643] The server also searches the database for recipe information to suggest recipes that combine foods that are approaching their expiration dates. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[1644] Specifically, the hardware used includes smartphones and servers, and the software used includes Tesseract OCR, which implements OCR technology, and Firebase Cloud Messaging, which provides push notification technology.

[1645] For example, if a user purchases milk and takes a photo of its expiration date, "2023-10-15," with their smartphone, the device sends this image to a server. The server uses OCR technology to extract the expiration date, "2023-10-15," from the image and stores it in a database. If the server checks the milk's expiration date, "2023-10-15," in the database and determines that the current date is "2023-10-12," the server generates a notification message stating, "The milk's expiration date is three days from now," and sends this push notification to the user via their device. It also generates a notification message suggesting a storage method, "Put it at the back of the refrigerator to cool," and a recipe, "Cream stew."

[1646] An example of a prompt sentence for the generative AI model is written as follows:

[1647] Extract the expiration date from the image below:

[1648] Image path: path_to_image.jpg

[1649] If the expiration date is "2023-10-15", generate a notification message suggesting the best way to store the food, "Put it at the back of the refrigerator to cool it down", and also suggest the recipe "Cream Stew".

[1650] As described above, this system aims to efficiently manage food expiration dates and reduce food waste by suggesting storage methods and recipes, allowing users to maintain food quality while consuming it and leading an economical and environmentally conscious lifestyle.

[1651] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1652] Step 1:

[1653] A user takes a photo of a food product's expiration date with a smartphone camera. The input is an image of the expiration date printed on the food's packaging. The output is the captured image data.

[1654] Step 2:

[1655] The device sends the captured image to the server. The input is the image data captured by the smartphone. The output is the image data transferred to the server.

[1656] Step 3:

[1657] The server uses OCR technology to extract the expiration date information from the image. The input is the received image data. The data processing is image analysis using Tesseract OCR. The output is the extracted expiration date string.

[1658] Step 4:

[1659] The extracted date information is saved in a database. The input is the date string obtained by OCR analysis. The data processing is writing the date string to the database. The output is a database entry with the expiration date saved.

[1660] Step 5:

[1661] The server periodically checks the expiration date information in the database. The input is the expiration date information of all food items stored in the database. The data calculation is a comparison with the current date. The output is a list of food items that are approaching their expiration date.

[1662] Step 6:

[1663] Identify foods approaching their expiration date and generate a notification message. The input is a list of foods approaching their expiration date. The data processing is the generation of the notification message. The output is the generated notification message.

[1664] Step 7:

[1665] The server sends the generated notification message to the device via Firebase Cloud Messaging. The input is the generated notification message. The output is the notification sent to the user's smartphone.

[1666] Step 8:

[1667] The device notifies the user via a push notification. The input is the notification message received from the server. The output is the push notification displayed to the user.

[1668] Step 9:

[1669] For foods approaching their expiration date, the server retrieves the optimal storage method from the database. The input is a list of foods approaching their expiration date. The data operation is a query from the storage method database. The output is the retrieved storage method.

[1670] Step 10:

[1671] A notification message is generated based on the storage method and notified to the user via the terminal. The input is the storage method. Data processing is the generation of the storage method notification message. The output is the storage method notification message.

[1672] Step 11:

[1673] To suggest recipes that combine foods with close expiration dates, the server searches for recipe information in a database. The input is a list of foods with close expiration dates. The data operation is a query from the recipe database. The output is a suitable recipe.

[1674] Step 12:

[1675] Select the appropriate recipe and generate a notification message. The input is the retrieved recipe information. The data processing is the generation of the recipe notification message. The output is the generated recipe notification message.

[1676] Step 13:

[1677] The terminal displays the generated recipe notification message to the user. The input is the recipe notification message from the server. The output is the recipe suggestion notification displayed to the user.

[1678] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1679] The present invention combines an emotion engine with a system that manages food expiration dates and suggests appropriate storage methods and recipes, providing a service based on the user's emotions. Specific embodiments of this system are described below.

[1680] 1. Acquisition and storage of expiration date data

[1681] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[1682] Specific examples

[1683] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[1684] 2. Best before date tracking and notification

[1685] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[1686] Specific examples

[1687] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[1688] 3. Storage method suggestions

[1689] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[1690] Specific examples

[1691] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[1692] 4. Recipe suggestions

[1693] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[1694] Specific examples

[1695] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[1696] 5. Providing services using emotion engines

[1697] 5.1 Emotion Recognition and Recipe Adjustment

[1698] The server uses an emotion engine to recognize the user's emotional state and adjusts the content of recipes and notifications depending on the user's emotions. For example, if the user is tired, it will suggest easy-to-make recipes.

[1699] Specific examples

[1700] The server uses an emotion engine to recognize that the user is tired, and then suggests a recipe for "stir-fried vegetables" as an "easy recipe recommended for when you're tired" and notifies the user via their device.

[1701] 5.2 Recipe suggestions based on tastes and preferences

[1702] The emotion engine determines the user's tastes and preferences and suggests recipes based on them, making it easier for users to cook their favorite dishes.

[1703] Specific examples

[1704] The emotion engine analyzes the user's preferences based on past data, and the server suggests a recipe for "Japanese-style pasta" as "a dish with the user's favorite seasoning." The device notifies the user of this information.

[1705] 5.3 Food suggestions for stress reduction

[1706] The emotion engine assesses the user's stress level and suggests foods and recipes that have a stress-reducing effect.

[1707] Specific examples

[1708] The emotion engine recognizes that the user is feeling stressed, and the server suggests a "relaxing herbal tea recipe." The device notifies the user of this information.

[1709] As described above, this system aims not only to efficiently manage food expiration dates and reduce food waste by suggesting storage methods and recipes, but also to provide services that respond to the user's emotional state using an emotion engine, thereby enabling users to live more comfortable and efficient lives.

[1710] The processing flow will be explained below.

[1711] Acquiring and storing expiration date data

[1712] Step 1:

[1713] The user takes a photo of the expiration date of the food they have purchased using their smartphone camera.

[1714] Step 2:

[1715] The terminal transmits the captured image to the server.

[1716] Step 3:

[1717] The server uses image analysis technology (OCR technology) to extract the expiration date information from the image.

[1718] Step 4:

[1719] The server stores the extracted expiration date information in a database.

[1720] Best before date tracking and notification

[1721] Step 1:

[1722] The server periodically checks the expiration date information in the database.

[1723] Step 2:

[1724] The server identifies food items that are nearing their expiration date (e.g., 3 days away).

[1725] Step 3:

[1726] The server generates notification messages corresponding to approaching expiration dates.

[1727] Step 4:

[1728] The terminal receives the notification message from the server and sends a push notification to the user.

[1729] Storage suggestions

[1730] Step 1:

[1731] The server retrieves from the database the storage method for food that is nearing its expiration date.

[1732] Step 2:

[1733] The server adds information about the storage method to the notification message.

[1734] Step 3:

[1735] The terminal displays a notification message to the user including the saving method.

[1736] Recipe Suggestions

[1737] Step 1:

[1738] The server searches the database for recipe information based on foods with an approaching expiration date.

[1739] Step 2:

[1740] The server selects an appropriate recipe based on the recipe information.

[1741] Step 3:

[1742] The server generates a notification message about the selected recipe.

[1743] Step 4:

[1744] The terminal displays a notification message to the user containing the selected recipe.

[1745] Providing services using emotion engines

[1746] Emotion recognition and recipe adjustment

[1747] Step 1:

[1748] The server uses an emotion engine to recognize the user's emotional state.

[1749] Step 2:

[1750] The server selects recipes and notification messages according to the user's emotions.

[1751] Step 3:

[1752] The server generates the selected recipe and notification message.

[1753] Step 4:

[1754] The device displays recipes and notification messages to the user based on their emotions.

[1755] Recipe suggestions based on your tastes and preferences

[1756] Step 1:

[1757] The server uses an emotion engine to determine the user's preferences and tastes.

[1758] Step 2:

[1759] The server retrieves recipes based on the user's preferences.

[1760] Step 3:

[1761] The server selects recipes based on preferences and generates notification messages.

[1762] Step 4:

[1763] The device displays a notification message to the user containing a recipe based on the preferences.

[1764] Food suggestions for stress reduction

[1765] Step 1:

[1766] The server uses an emotion engine to determine whether the user is experiencing stress.

[1767] Step 2:

[1768] The server searches for foods and recipes that have stress-reducing effects.

[1769] Step 3:

[1770] The server selects foods and recipes for stress reduction and generates a notification message.

[1771] Step 4:

[1772] The device will display a notification message to the user containing stress-reducing foods and recipes.

[1773] Example 2

[1774] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1775] Current food management systems require users to manually input expiration dates, making expiration date management cumbersome. Furthermore, storage methods and recipe suggestions are uniform and are not based on the user's emotional state or preferences, resulting in low user convenience. Furthermore, if expiration date notifications are not properly provided, there is a high possibility of food waste.

[1776] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for photographing the expiration date of food and saving it as data, a means for automatically notifying when the expiration date is approaching, a means for suggesting a food preservation method, a means for providing a recipe that combines foods with close expiration dates, and a means for recognizing the user's emotional state and adjusting the notification and suggestion content based on the emotional state. This enables efficient management of expiration dates, suggestion of appropriate preservation methods, and suggestion of recipes that correspond to the user's emotions.

[1777] "Means for photographing expiration dates and saving them as data" refers to a means by which a user photographs the expiration date of a food product using a smartphone camera, sends the image to a server, and uses image analysis technology (OCR technology) to extract the date information and save it in a database.

[1778] "Means for automatically notifying when the expiration date is approaching" refers to a means in which the server periodically checks the expiration date information in the database, generates a notification message when the expiration date is approaching, and pushes the message to the user via the terminal.

[1779] The "means for suggesting food preservation methods" refers to a means by which the server retrieves the optimal preservation method from a database for food that is approaching its expiration date, generates a notification message based on that information, and notifies the user via the terminal.

[1780] The "means for providing recipes that combine foods with close expiration dates" refers to a means by which the server searches a database for recipe information based on foods with close expiration dates, selects an appropriate recipe, generates a notification message, and displays it to the user via the terminal.

[1781] "Means for recognizing the user's emotional state and adjusting notifications and suggestions based on that" refers to a means by which the server recognizes the user's emotional state using an emotion engine and adjusts notifications and suggestions based on the user's emotions. Specifically, this includes adjustments such as suggesting easy-to-make recipes when the user is tired.

[1782] The present invention combines an emotion engine with a system that manages food expiration dates and suggests appropriate storage methods and recipes, providing a service based on the user's emotions. Specific embodiments of this system are described below.

[1783] Acquiring and storing expiration date data

[1784] When a user purchases food, they take a photo of the food's expiration date using their smartphone camera. The device then sends the image to a server, which then uses image analysis technology (OCR technology) to extract the expiration date information from the image. This extracted date information is then stored in a database. This saves users the trouble of manually entering expiration dates for food products.

[1785] Specific examples

[1786] A user purchases milk and takes a photo of the expiration date "2023-10-15" with their smartphone. The device sends this image to a server, which uses OCR technology to extract the expiration date "2023-10-15" from the image and stores it in a database.

[1787] Best before date tracking and notification

[1788] The server periodically checks the expiration date information in the database. It identifies foods that are approaching their expiration date and, if so, generates a notification message. This notification message is sent to the device, which then notifies the user via push notification. This allows the user to remember the expiration date and consume the food at the appropriate time.

[1789] Specific examples

[1790] The server checks the expiration date of the milk in the database, "2023-10-15," and determines that the current date is "2023-10-12." The server generates a notification message stating, "The milk's expiration date is in 3 days," and pushes it to the user via the device.

[1791] Storage suggestions

[1792] For food products approaching their expiration date, the server retrieves the optimal storage method from the database. Based on this information, it generates a notification message suggesting a storage method and notifies the user via their device. This allows the user to store the food properly and maintain its quality.

[1793] Specific examples

[1794] The server confirms that the milk's expiration date is approaching and retrieves the storage method "Keep it at the back of the refrigerator to cool" from the database. The server then generates a notification saying "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator to cool it" and notifies the user via the terminal.

[1795] Recipe Suggestions

[1796] To suggest recipes that combine foods that are approaching their expiration dates, the server searches for recipe information in the database. It selects an appropriate recipe and generates a notification message. The device displays this notification message to the user. This allows the user to efficiently use foods that are approaching their expiration dates when cooking.

[1797] Specific examples

[1798] The server searches the database for a recipe for "cream stew" that uses milk that is close to its expiration date. The server generates a notification message saying "Here is a recipe for cream stew that uses milk," and notifies the user via the terminal.

[1799] Providing services using emotion engines

[1800] Emotion recognition and recipe adjustment

[1801] The server uses an emotion engine to recognize the user's emotional state and adjusts the content of recipes and notifications depending on the user's emotions. For example, if the user is tired, it will suggest easy-to-make recipes.

[1802] Specific examples

[1803] The server uses an emotion engine to recognize that the user is tired, and then suggests a recipe for "stir-fried vegetables" as an "easy recipe recommended for when you're tired" and notifies the user via their device.

[1804] Recipe suggestions based on your tastes and preferences

[1805] The emotion engine determines the user's tastes and preferences and suggests recipes based on them, making it easier for users to cook their favorite dishes.

[1806] Specific examples

[1807] The emotion engine analyzes the user's preferences based on past data, and the server suggests a recipe for "Japanese-style pasta" as "a dish with the user's favorite seasoning." The device notifies the user of this information.

[1808] Food suggestions for stress reduction

[1809] The emotion engine assesses the user's stress level and suggests foods and recipes that have a stress-reducing effect.

[1810] Specific examples

[1811] The emotion engine recognizes that the user is feeling stressed, and the server suggests a "relaxing herbal tea recipe." The device notifies the user of this information.

[1812] Prompt Sentence Examples

[1813] "Please suggest recipes that efficiently use food in the refrigerator that is nearing its expiration date."

[1814] "Please suggest easy dinner recipes for when the user is tired."

[1815] "Provide new recipes based on your users' favorite flavors."

[1816] With this system, users can efficiently manage food expiration dates and receive suggestions for storage methods and recipes. Furthermore, by using an emotion engine, it is possible to provide services that correspond to the user's emotional state, improving the quality of daily life.

[1817] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1818] Step 1:

[1819] A user takes a photo of the expiration date on a food item

[1820] explanation:

[1821] The user uses the smartphone camera to take a photo of the expiration date on food. To do this, they launch the camera app, frame the food label, and take a photo. The input is the image data taken with the smartphone. The output is the image data taken.

[1822] Step 2:

[1823] The device sends the image to the server

[1824] explanation:

[1825] The device sends the image data captured in step 1 to the server. This transmission is performed via an HTTP request. The input is the captured image data. The output is the image data sent to the server.

[1826] Step 3:

[1827] The server extracts the date information from the image.

[1828] explanation:

[1829] The server processes the received image data and extracts the expiration date information as text using OCR technology (e.g., Tesseract OCR). The server passes the image data to the OCR engine to obtain the date information. The input is the image data sent to the server. The output is the extracted date information in text format.

[1830] Step 4:

[1831] The server stores the extracted information in a database

[1832] explanation:

[1833] The server stores the date information extracted in step 3 in a database, possibly along with additional information such as the name of the food item and the date of purchase. The input is the extracted date information and other relevant information. The output is the information stored in the database.

[1834] Step 5:

[1835] The server periodically checks the database

[1836] explanation:

[1837] The server periodically (e.g., every night at midnight) executes a batch process that checks expiration date information in the database. The input is the expiration date information in the database. The output is the checked expiration date information.

[1838] Step 6:

[1839] Identify approaching expiration dates on your servers

[1840] explanation:

[1841] The server checks each record in the database and calculates the difference between the current date and the expiration date, thereby identifying foods that are nearing their expiration date. The input is the expiration date information being checked and the current date. The output is the approaching expiration date information.

[1842] Step 7:

[1843] The server generates a notification message

[1844] explanation:

[1845] The server generates a notification message for the food item identified in step 6 that is nearing its expiration date. For example, it creates a message that reads "Milk will expire in 3 days." The input is the upcoming expiration date information. The output is the generated notification message.

[1846] Step 8:

[1847] The device sends a push notification to the user

[1848] explanation:

[1849] The server sends the notification message generated in step 7 to the device. The device displays this message to the user as a push notification. The input is the generated notification message. The output is the notification message displayed to the user.

[1850] Step 9:

[1851] The server retrieves the save method.

[1852] explanation:

[1853] The server retrieves the optimal storage method from the database for food items that are nearing their expiration date. For example, for milk, it retrieves information such as "keep it at the back of the refrigerator to keep it cool." The input is information about food items that are nearing their expiration date. The output is the retrieved storage method information.

[1854] Step 10:

[1855] The server generates a message informing you of the save method.

[1856] explanation:

[1857] The server generates a notification message based on the storage method information obtained in step 9. For example, it creates a message saying, "The milk's expiration date is in 3 days. Please keep it at the back of the refrigerator and keep it cool." The input is the obtained storage method information. The output is the generated notification message.

[1858] Step 11:

[1859] The device sends a notification to the user

[1860] explanation:

[1861] The server sends the notification message generated in step 10 to the terminal, and the terminal notifies the user of the information. The input is the generated notification message. The output is the notification message displayed to the user.

[1862] Step 12:

[1863] The server searches the database for the recipe

[1864] explanation:

[1865] The server searches the database for recipes that use foods that are close to their expiration date. For example, it finds a recipe for "cream stew" that uses milk. The input is information about foods that are close to their expiration date. The output is the searched recipe information.

[1866] Step 13:

[1867] The server generates a notification message with the appropriate recipe

[1868] explanation:

[1869] The server generates a notification message based on the recipe information retrieved in step 12. For example, it creates a message that reads, "Click here for a recipe for cream stew using milk." The input is the retrieved recipe information. The output is the generated notification message.

[1870] Step 14:

[1871] The device sends a notification to the user

[1872] explanation:

[1873] The server sends the notification message generated in step 13 to the terminal, and the terminal notifies the user of the information. The input is the generated notification message. The output is the notification message displayed to the user.

[1874] Step 15:

[1875] The server uses an emotion engine to recognize the user's emotional state.

[1876] explanation:

[1877] The server uses an emotion engine to recognize the user's emotional state. The input is the user's past data and current interaction information. The output is the recognized user's emotional state.

[1878] Step 16:

[1879] The server tailors notifications and suggestions based on emotions

[1880] explanation:

[1881] The server adjusts notifications and suggestions based on the emotional state recognized in step 15. For example, if it recognizes that the user is tired, it may suggest easy-to-make recipes. The input is the recognized emotional state of the user and the desired information. The output is the adjusted notifications and suggestions.

[1882] Step 17:

[1883] The server generates emotion-based recipe notification messages.

[1884] explanation:

[1885] The server generates a notification message based on the suggestions adjusted in step 16. For example, it creates a message such as "Easy recipes for when you're tired: Stir-fried vegetables." The input is the adjusted notification and suggestions. The output is the generated notification message.

[1886] Step 18:

[1887] The device notifies the user with emotion-based suggestions

[1888] explanation:

[1889] The server sends the notification message generated in step 17 to the terminal, and the terminal notifies the user of the information. The input is the generated notification message. The output is the notification message displayed to the user.

[1890] This allows users to efficiently manage expiration dates, receive suggestions for appropriate storage methods, and receive notifications and recipe suggestions based on the user's emotions.

[1891] (Application example 2)

[1892] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1893] In recent years, managing food expiration dates has become an important issue in homes and brick-and-mortar stores. However, efficiently managing expiration dates and reducing waste requires a lot of effort and cost. Furthermore, when users and staff are in an emotional state, it is necessary to further improve convenience and satisfaction by suggesting appropriate storage methods and recipes. Current technology does not provide such a comprehensive system, so a new approach is needed to improve users' quality of life.

[1894] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1895] In this invention, the server includes means for photographing expiration dates of food products and saving the photographed data, means for automatically notifying users when the expiration date is approaching, means for suggesting food preservation methods, means for providing recipes that combine foods with approaching expiration dates, means for photographing expiration dates using a smart device, and means for adjusting notification content and recipes based on the user's emotional state using an emotion engine. This makes it possible to efficiently manage expiration dates and make appropriate suggestions according to the user's emotions.

[1896] "Means for photographing food expiration dates and saving them as data" refers to a means for a user to photograph the expiration date of food purchased by the user using the camera on their smart device, analyze the image, and save the date information in a database.

[1897] The "means for automatically notifying when the expiration date is approaching" is a means for the server to periodically analyze the stored expiration date data and notify the user about food products whose expiration date is approaching.

[1898] The "means for suggesting a food preservation method" is a means for retrieving the optimum preservation method for food that is approaching its expiration date from a database and notifying the user of the method.

[1899] The "means for providing recipes that combine foods with close expiration dates" refers to a means for the server to search a database for recipes that combine foods with close expiration dates based on the expiration date information of the foods stored on the server, and suggest them to the user.

[1900] "Means for photographing expiration dates using a smart device" refers to means for photographing expiration dates on food products using a smart device such as a smartphone, smart glasses, or a head-mounted display.

[1901] "Means for adjusting notification content and recipes based on the user's emotional state using an emotion engine" refers to means for analyzing the user's emotional state using emotion recognition technology and adjusting the notification and recipe content based on the results.

[1902] As an embodiment of the present invention, a system for managing expiration dates of food products and suggesting appropriate storage methods and recipes includes the following processing steps.

[1903] The overall system configuration uses smart devices (smartphones, smart glasses, head-mounted displays, etc.) to manage food expiration dates. Specifically, when a user purchases food, they use the smart device's camera to take a photo of the expiration date, and the device sends the image to a server. The server then uses image analysis technology (OCR technology) to extract the expiration date information and saves it in a database.

[1904] Examples of devices used include smartphones and smart glasses. Examples of smart glasses include commonly used commercial products (e.g., Google Glass). The software uses Python, OpenCV for image processing, and pytesseract for OCR analysis. The emotion engine uses machine learning models and natural language processing techniques to analyze the user's emotional state.

[1905] The server periodically checks the expiration date information in the database, generates a notification message for food items whose expiration date is approaching, and sends it to the device. The device then notifies the user of this notification message via push notification. This allows the user to consume the food at the appropriate time without forgetting the expiration date.

[1906] Furthermore, the server retrieves the optimal storage method for food items approaching their expiration date from the database, generates a notification message suggesting that storage method, and notifies the user via the terminal. This allows the user to store the food properly and maintain its quality. The server also searches for recipe information in the database to suggest recipes that combine foods approaching their expiration date, selects appropriate recipes, generates a notification message, and displays it to the user via the terminal.

[1907] The emotion engine is used to recognize the user's emotional state. For example, when a user takes a picture while wearing the smart glasses, the emotion engine analyzes the user's facial expressions and evaluates the user's emotional state. Based on this, the server will adjust the content of recipes and notifications according to the user's emotions.

[1908] As a concrete example, if a user photographs the expiration date of milk with smart glasses, the system automatically analyzes the expiration date "2023-10-15" and stores it in a database. When the expiration date approaches, the server generates a notification stating "The milk's expiration date is in three days" and sends it as a push notification to the user. It also suggests storing the milk at the back of the refrigerator to keep it cool. If the emotion engine recognizes that the user is tired, it suggests a recipe for "stir-fried vegetables" as an "easy recipe recommended for when you're tired."

[1909] An example of a prompt for the generative AI model is as follows:

[1910] Example prompt sentence:

[1911] "Create a Python program that analyzes images of food taken with smart glasses and automatically extracts the expiration date. If the expiration date is approaching, it generates a message suggesting storage methods or recipes, and uses an emotion engine to respond according to the user's emotions. However, it should have conditions that allow the emotion engine to recognize fatigue, stress, or other specific emotional states. Please explain with a concrete example using smart glasses and Python."

[1912] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1913] Step 1:

[1914] A user takes a photo of the expiration date of a food product using a smart device (smartphone, smart glasses, etc.).

[1915] (Input) The image of the food to be photographed.

[1916] (Output) Food image data including expiration date.

[1917] (Specific Operation) For example, a user puts on smart glasses and takes a picture of the food label with the expiration date written on it. At this time, the camera function of the smart device is used.

[1918] Step 2:

[1919] The device sends the photographed image of the food to the server.

[1920] (Input) Food image data including expiration dates.

[1921] (Output) Image data sent to the server.

[1922] (Specific operation) Using the communication function built into the smart device, the image data of the food taken by the user is transferred to a server on the cloud, using Wi-Fi or mobile data communication technology.

[1923] Step 3:

[1924] The server uses image analysis technology (OCR technology) to extract the expiration date information from the transmitted image.

[1925] (Input) Image data of the food sent to the server.

[1926] (Output) The extracted expiration date information.

[1927] (Specific operation) The server analyzes the image using Python and OpenCV, and extracts the expiration date information (e.g., "2023-10-15") from the image by performing OCR processing using pytesseract.

[1928] Step 4:

[1929] The server stores the extracted expiration date information in a database.

[1930] (Input) The extracted expiration date information.

[1931] (Output) Best before date data stored in the database.

[1932] (Specific operation) The server saves the extracted date information in a database (e.g., MySQL or SQLite) using an SQL query.

[1933] Step 5:

[1934] The server periodically checks the expiration date information in the database.

[1935] (Input) Best before date data in the database.

[1936] (Output) A list of foods that are nearing their expiration date.

[1937] (Specific Operation) The server periodically queries all expiration date data in the database using a scheduler to identify items whose expiration dates will arrive within a certain period from the current date.

[1938] Step 6:

[1939] The server generates a notification message about food products that are approaching their expiration date and sends it to the terminal.

[1940] (Input) A list of food items that are nearing their expiration date.

[1941] (Output) Informational message.

[1942] (Specific operation) The server generates the content of the push notification (for example, "The expiration date of the milk is three days later") and sends it to the terminal via an API for sending notification messages to smart devices.

[1943] Step 7:

[1944] The terminal notifies the user of the notification message by push notification.

[1945] (Input) The notification message.

[1946] (Output) The notification that is displayed to the user.

[1947] (Specific operation) The notification message received by the terminal is displayed to the user using the notification function of the smart device.

[1948] Step 8:

[1949] The server retrieves the optimum storage method for food items whose expiration date is approaching from a database, generates a notification message, and transmits it to the terminal.

[1950] (Input) A list of foods that are close to expiry.

[1951] (Output) A message informing you of the save method.

[1952] (Specific operation) For example, the server obtains information from the database about how to store milk, such as "place it at the back of the refrigerator to cool it down," and based on this information generates a notification message and sends it to the terminal.

[1953] Step 9:

[1954] The server searches a database for recipes that combine foods whose expiration dates are approaching, selects them, generates a notification message, and transmits it to the terminal.

[1955] (Input) A list of foods that are close to expiry.

[1956] (Output) The recipe notification message.

[1957] (Specific operation) The server searches the database for "recipes that combine foods with close expiration dates" (for example, "cream stew using milk"), generates a notification message based on this, and sends it to the terminal.

[1958] Step 10:

[1959] It uses an emotion engine to recognize the user's emotional state and adjust recipes and notifications accordingly.

[1960] (Input) The user's emotional state.

[1961] (Output) Emotion-based recipes and notification content.

[1962] (Specific operation) The emotion engine analyzes the user's image, voice, and input content to recognize their emotional state, such as fatigue or stress. For example, if the server determines that the user is tired, it generates a notification message suggesting a "simple vegetable stir-fry recipe" and sends it to the device.

[1963] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1964] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1965] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1966] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1967] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, mu...

Claims

1. A method to photograph the expiration date of food and save it as data, A means to automatically notify you when the expiration date is approaching, A means of suggesting food preservation methods; A means to provide recipes that combine foods with close expiration dates, A system including:

2. A means for taking a photograph of the expiration date, extracting the date through image analysis, and storing the date in a database; The system of claim 1 .

3. means for periodically checking the expiration date and generating and sending a notification message to a user when the expiration date is determined to be approaching; The system of claim 1 .

4. A means for retrieving information about the storage method from the database and notifying the user of an appropriate storage method based on the information; The system of claim 1 .

5. A means for searching for recipes in a database based on information about food products with an approaching expiration date, selecting a recipe to suggest, and notifying the user; The system of claim 1 .

Images