System

Abstract

A system is provided.SOLUTION: This system is provided with a means for photographing a receipt and a commodity picture after shopping and analyzing commodity information from the photographed image, a means for automatically updating stock in a refrigerator based on the analyzed commodity information, and a means for receiving the information when a user performs voice input or image input of a food material used in cooking. This system includes a means for updating stock in a refrigerator, a means for acquiring flier information from an external Web site every day, analyzing it and registering merchandise information in a database, and a means for receiving a voice input for consulting a menu by a user, and generating and proposing a proper menu and shopping list based on the present stock and the acquired flier information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern households, managing food inventory, planning cooking menus, and creating necessary shopping plans are important tasks, but these processes are often done manually, resulting in labor-intensive and time-consuming issues. It is particularly difficult to grasp food consumption status in real time, making it difficult to make appropriate purchases at the right time. Furthermore, manually checking the daily changing sale information in supermarket flyers and updating the shopping list is a cumbersome task. There is a need for a system that can resolve these issues and efficiently manage food ingredients and plan shopping. [Means for solving the problem]

[0005] This invention provides a system that analyzes receipts and product photos taken by users and automatically updates the inventory in a refrigerator. Specifically, it includes a means for extracting product information using optical character recognition (OCR) technology and updating the inventory database based on that information. It also provides a means for receiving information and updating the inventory database when a user inputs ingredients used in cooking by voice or image. It also includes a means for acquiring images of supermarket flyers daily from an external website, analyzing them, and registering product information in the database. When a user consults about a menu by voice, it provides a means for generating and suggesting an appropriate menu and shopping list based on current inventory information and flyer information. In this way, a system is realized that enables efficient ingredient management, menu planning, and shopping planning.

[0006] "Receipts and product photos after shopping" refers to receipts that contain information about the products purchased by the user and photos of the purchased products.

[0007] "Means for analyzing product information" refers to technology that extracts text information from photographed receipts or product photos and identifies information such as product name, quantity, and price.

[0008] "Means for automatically updating inventory in the refrigerator" refers to a function that automatically updates inventory information recorded in the database within the system based on analyzed product information.

[0009] "Voice input or image input" refers to a method in which the user tells the system the ingredients used through voice, or provides the system with an image of the ingredients used.

[0010] "Means for obtaining daily flyer information from external websites" refers to the function of downloading images and information of daily flyers from websites of supermarkets and other businesses via the Internet.

[0011] "Means for registering flyer information in a database" refers to technology that analyzes acquired flyer information and stores information such as product name, price, and validity period in a database.

[0012] "Means for receiving voice input and generating an appropriate menu and shopping list based on current stock and acquired flyer information" refers to a function that analyzes the voice input when a user discusses a menu by voice, and creates an appropriate menu and a shopping list of ingredients that are in short supply based on the current stock situation and flyer information. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention is a system that manages refrigerator inventory by analyzing receipts and product photos taken by users after shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information obtained daily.

[0035] Program processing overview

[0036] Inventory management using receipts and product photos after shopping

[0037] 1. The user takes a photo of the receipt or product with their smartphone.

[0038] 2. Upload the photos taken by the device (smartphone) to the cloud server.

[0039] 3. The server analyzes the received image using optical character recognition (OCR) technology and extracts product information (product name, quantity, price, etc.).

[0040] 4. The server records and updates the analyzed product information in the refrigerator's inventory database.

[0041] Control delivery by voice or image input during cooking

[0042] 1. When the user prepares a dish, they tell the system the ingredients they will use using voice or images.

[0043] 2. The device converts the voice input into text and sends the information to the server. If the input is an image, the image is sent to the server.

[0044] 3. The server updates the refrigerator's inventory database based on the received information, decrementing the quantity of ingredients used and notifying the user if the ingredient is out of stock.

[0045] Daily flyer information acquisition and analysis

[0046] 1. The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[0047] 2. The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[0048] 3. The extracted product information is registered in the flyer information database.

[0049] Voice menu consultation and suggestions

[0050] 1. The user asks the system verbally, "What would you like for dinner tonight?"

[0051] 2. The device converts the voice input into text and sends the information to the server.

[0052] 3. Based on the information received by the server, the current refrigerator inventory database is compared with the flyer information database to generate an appropriate menu and a shopping list of missing ingredients.

[0053] 4. The device will output a suggestion to the user via voice, such as "To make chicken and broccoli stir-fry, you will need to buy broccoli from the supermarket."

[0054] Specific examples

[0055] Specific examples of inventory management

[0056] A user purchases milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which uses OCR technology to extract the product information (1 liter of milk) and updates the refrigerator's inventory database with the entry "Add 1 liter of milk."

[0057] Specific examples of inventory management

[0058] While cooking, the user says "I used milk" by voice input. The device converts the voice to text and sends it to the server as "I used 1 liter of milk." The server updates the inventory database and records it as "I reduced 1 liter of milk."

[0059] Examples of flyer information

[0060] At 9:00 AM, the server retrieves the flyer image from the supermarket's website, analyzes the product information "Broccoli 98 yen," and registers it in the flyer information database. It saves it as "Broccoli 98 yen, sale period: 3 days."

[0061] Specific examples of menu consultation

[0062] The user asks, "What should I have for dinner tonight?" The device converts this speech into text and sends it to the server. The server compares the current inventory database with the flyer information database, determines that "there is chicken in the fridge and broccoli on sale," and suggests "stir-fried chicken and broccoli."

[0063] This system provides a multifunctional mechanism that includes these steps, and centrally and automatically supports users in daily food management, menu planning, and shopping planning.

[0064] The processing flow will be explained below.

[0065] Manage refrigerator inventory using shopping photos and receipts

[0066] Step 1:

[0067] The user takes a photo of the receipt or product with their smartphone.

[0068] Open the camera app on your smartphone and take a photo of your receipt or purchased items.

[0069] Step 2:

[0070] The device uploads the photos it takes to the server.

[0071] Use the dedicated application and press the button to upload the photo to the cloud server.

[0072] Step 3:

[0073] The server applies OCR technology to analyze the received image.

[0074] The server invokes an image processing module to extract text information from the image.

[0075] Step 4:

[0076] The server extracts the parsed product information (product name, quantity, price, etc.).

[0077] Product name, quantity, and price are identified from the text information and converted into a database format.

[0078] Step 5:

[0079] The server records the extracted product information in the refrigerator's inventory database and updates it.

[0080] Adds the product information to the corresponding user's inventory database and updates existing inventory information.

[0081] Control delivery by voice or image input during cooking

[0082] Step 1:

[0083] When cooking, the user verbally tells the system what ingredients to use.

[0084] Use your smartphone's microphone to input voice instructions such as "I used chicken."

[0085] Step 2:

[0086] The device converts the voice input into text and sends the information to the server.

[0087] Uses voice recognition software to convert speech into text.

[0088] Step 3:

[0089] The server updates the refrigerator inventory database based on the received information.

[0090] The quantity of the specified ingredient is reduced and the inventory database is updated.

[0091] Step 4:

[0092] If the server is out of stock, the user is notified.

[0093] If the stock reaches 0, a notification message will be sent to the user.

[0094] Daily flyer information acquisition and analysis

[0095] Step 1:

[0096] The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[0097] Based on a fixed schedule, it automatically visits each supermarket's website and downloads flyer images.

[0098] Step 2:

[0099] The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[0100] Use the flyer image processing module to extract product information from the image.

[0101] Step 3:

[0102] The extracted product information is registered in a flyer information database.

[0103] The flyer information is stored in a database based on the product name, price, and validity period.

[0104] Voice menu consultation and suggestions

[0105] Step 1:

[0106] The user verbally asks the system, "What would you like for dinner tonight?"

[0107] Voice menu consultation is performed using the smartphone's microphone.

[0108] Step 2:

[0109] The device converts the voice input into text and sends the information to the server.

[0110] Use speech recognition software to convert speech to text.

[0111] Step 3:

[0112] Based on the information received by the server, the current refrigerator inventory database is compared with the flyer information database.

[0113] Cross-check the database to see what ingredients are available and what discounts are available.

[0114] Step 4:

[0115] Generate appropriate menus and shopping lists for missing ingredients.

[0116] Generate suggested recipes and add missing ingredients to your shopping list.

[0117] Step 5:

[0118] The terminal outputs the suggestions to the user by voice.

[0119] Using the smartphone's speaker, a voice suggestion is made: "To make chicken and broccoli stir-fry, you need to buy broccoli at the supermarket."

[0120] Through the above processing flow, this system efficiently supports the user in managing ingredients, planning menus, and making shopping plans.

[0121] Example 1

[0122] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0123] In modern life, efficient ingredient management and menu planning are important challenges for many households. However, manually managing ingredients and creating appropriate menus and shopping lists is time-consuming and labor-intensive, making it a burden for many people. In particular, manually collecting and using information from refrigerator inventory and supermarket flyers is cumbersome and inefficient. Additionally, it is difficult to update the inventory of ingredients used during cooking in real time, resulting in problems such as out-of-stock situations and food waste.

[0124] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0125] In this invention, the server includes means for taking pictures of receipts and products after a user has finished shopping and analyzing the product information from the captured images, means for automatically updating the inventory in the refrigerator based on the analyzed product information, means for receiving information about ingredients used in cooking when the user inputs the information by voice or image and updating the inventory in the refrigerator, means for acquiring flyer information from an external website every day, analyzing the information, and registering the product information in a database, means for receiving voice input from the user discussing menus and generating and suggesting appropriate menus and shopping lists based on the current inventory and the acquired flyer information, means for converting the user's voice input into text using a voice recognition system and sending the information to the server, and means for outputting suggestions by voice using a text-to-speech conversion system.This allows the user to efficiently manage their refrigerator inventory without any hassle and have appropriate menus and shopping lists automatically suggested.

[0126] A "user" is an entity that uses the system to manage ingredients, create menus, and plan shopping.

[0127] A "receipt" is a paper record of a purchase made at a store.

[0128] "Product photo" is an image of the purchased product.

[0129] "Means for analyzing product information from images" refers to a method for extracting information such as product name, quantity, and price from images using optical character recognition technology.

[0130] The "means for automatically updating inventory in the refrigerator" is a system that updates the database based on analyzed product information to reflect the new inventory status.

[0131] "Means for inputting information about ingredients used in cooking by voice or image" refers to a method in which the user communicates information about ingredients used to the system through voice or image.

[0132] A "voice recognition system" is a technology that converts voice input into text.

[0133] The "means for updating inventory" is a system that modifies the inventory database based on user input and maintains the latest inventory status.

[0134] "Means for obtaining daily flyer information from external websites" refers to a technology for downloading flyer images from websites of supermarkets and the like via the Internet.

[0135] The "means of analyzing and registering product information in a database" refers to a system that analyzes acquired flyer images using optical character recognition technology and stores the product information in a database.

[0136] The "means for receiving voice input for consulting a menu" is a method for receiving information in which a user requests menu suggestions by voice.

[0137] "Means for generating appropriate menus and shopping lists" refers to technology that creates optimal cooking menus and lists of ingredients that are in short supply for users based on current inventory information and flyer information.

[0138] "Means for converting a user's voice input into text using a voice recognition system" refers to a method that uses technology to convert voice data into text data.

[0139] The "means for outputting suggestions by voice using a text-to-voice conversion system" is a technology for converting generated text information into voice and conveying it to the user.

[0140] A "natural language processing model" is a technology that generates appropriate text and analyzes intent based on user input.

[0141] This invention is a system that manages refrigerator inventory by analyzing receipts and product photos taken by users after shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information obtained daily.

[0142] The system is implemented using the following hardware and software:

[0143] Hardware:

[0144] Smartphone (user device)

[0145] Cloud Server

[0146] Refrigerator (physical storage location)

[0147] software:

[0148] Optical character recognition technology (OCR): Google Cloud Vision API

[0149] Speech recognition system: Google Cloud Speech-to-Text, Amazon Transcribe

[0150] Text-to-speech system: Amazon Polly

[0151] Web scraping tools: Selenium, BeautifulSoup

[0152] Natural language processing model: GPT-3

[0153] Database management systems: MySQL, PostgreSQL

[0154] Voice UI Applications

[0155] After a user finishes shopping at the supermarket, they take a photo of the receipt or product with their smartphone. The device uploads the photo to a cloud server, which then analyzes the image using the Google Cloud Vision API. Using OCR technology, product information (product name, quantity, price, etc.) is extracted, and the server automatically registers and updates the analyzed information in the refrigerator's inventory database.

[0156] When cooking, the user can tell the system which ingredients they will use by voice or image input. The device converts the voice input into text using Google Cloud Speech-to-Text or Amazon Transcribe and sends the information to the server. When using image input, the device also sends the image to the server, which updates the inventory database based on the analysis results. The server reduces the quantity of ingredients used and notifies the user if they are out of stock.

[0157] The server also retrieves flyer images from each supermarket's website at a specified time every day using web scraping tools (Selenium, BeautifulSoup). The server then analyzes the flyer images using Tesseract OCR, extracts product information (product name, price, validity period, etc.), and registers it in the flyer information database.

[0158] When a user asks the system, "What would you like for dinner tonight?", the device converts the speech to text using Amazon Transcribe and sends it to the server. The server compares the current inventory database with the flyer information database and uses the GPT-3 natural language processing model to generate an appropriate menu and a shopping list of missing ingredients. The device then uses Amazon Polly to output the suggestions to the user.

[0159] Specific examples

[0160] For example, suppose a user buys milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which then uses the Google Cloud Vision API to extract the product information, such as "1 liter of milk." The information is then registered and updated in the refrigerator's inventory database as "1 liter of milk added."

[0161] If a user says "I used milk" while cooking, the device converts the speech into text using Google Cloud Speech-to-Text and sends it to the server as "I used 1 liter of milk." The server then updates the inventory database, recording "I reduced 1 liter of milk."

[0162] Every day at 9:00 a.m., the server uses Selenium to retrieve flyer images from the supermarket's website, analyzes the product information, such as "Broccoli 98 yen," using Tesseract OCR, and registers it in the flyer information database.

[0163] When a user asks, "What should I have for dinner tonight?", the device converts the speech into text using Amazon Transcribe and sends it to the server. The server compares the current inventory database with the flyer information database and uses the GPT-3 natural language processing model to suggest "chicken and broccoli stir-fry." The device then uses Amazon Polly to announce, "To make chicken and broccoli stir-fry, you need to buy broccoli at the supermarket."

[0164] Prompt Sentence Examples

[0165] Example prompt sentences when the user says "I used milk":

[0166] The user says "Milk used." Update the refrigerator inventory database. Decrease the quantity of milk by 1 liter and record the result.

[0167] Example prompt sentences when a user says "What should I have for dinner tonight?":

[0168] The user says, "What should I have for dinner tonight?" Check the current refrigerator inventory database against the flyer information database to suggest an appropriate dinner menu. If there are any ingredients missing, prepare a shopping list for them.

[0169] The above is a specific embodiment for carrying out the present invention.

[0170] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0171] Inventory management using receipts and product photos after shopping

[0172] Step 1:

[0173] After a user finishes shopping at a supermarket, they take a photo of the receipt or product with their smartphone. The user opens the smartphone's camera app and takes a photo of the receipt or product. This image becomes input data for the system.

[0174] Step 2:

[0175] The device (smartphone) takes a photo and uploads it to the cloud server. The upload program encodes the image data and sends it to the cloud server via a secure connection. The input is an image of the receipt or product, and the output is the storage of the image data on the cloud server.

[0176] Step 3:

[0177] The server analyzes the received photos using optical character recognition (OCR) technology. Specifically, it uses the Google Cloud Vision API to extract product information such as product name, quantity, and price from the image. The input is image data, and the output is analyzed product information (text data).

[0178] Step 4:

[0179] The server records and updates the parsed product information in the refrigerator's inventory database. The database management system (e.g., MySQL) adds the new product information and updates the existing inventory data. The input is the parsed product information, and the output is the updated inventory database.

[0180] Control delivery by voice or image input during cooking

[0181] Step 1:

[0182] When a user prepares a dish, they can tell the system which ingredients they will use by voice or image. Specifically, the user opens the smartphone app and either voice-inputs "I used milk" or takes a photo of the ingredients used. The input is voice data or image data.

[0183] Step 2:

[0184] The device converts voice input into text and sends that information to the server. "Google Cloud Speech-to-Text" is used as the voice recognition system. Voice is converted into text and the textual information is sent to the server. The input is voice data and the output is text data. In the case of image input, the image is sent directly to the server. The input is image data and the output is saving the image data to the server.

[0185] Step 3:

[0186] The server updates the refrigerator's inventory database based on the information it receives. Specifically, it analyzes the text data and reduces the quantity of ingredients used. If the item is out of stock, a program is activated to notify the user. The input is the text data, and the output is the updated inventory database and a notification.

[0187] Daily flyer information acquisition and analysis

[0188] Step 1:

[0189] The server retrieves flyer information as images from each supermarket's website at a specified time every day. Specifically, the web scraping tools "Selenium" and "BeautifulSoup" are used to download the flyer images. The input is the supermarket's website URL, and the output is the retrieved flyer image.

[0190] Step 2:

[0191] The server analyzes the flyer image it has acquired. Specifically, it uses "Tesseract OCR" to extract product information (product name, price, validity period, etc.). The input is the flyer image, and the output is the analyzed product information (text data).

[0192] Step 3:

[0193] The extracted product information is registered in a flyer information database. New flyer information is added and saved using a database management system (e.g., PostgreSQL). The input is the parsed product information, and the output is an updated flyer information database.

[0194] Voice menu consultation and suggestions

[0195] Step 1:

[0196] The user asks the system by voice, "What would you like for dinner tonight?" The user opens the app on their smartphone and enters voice input. The input is voice data.

[0197] Step 2:

[0198] The device converts voice input into text and sends that information to the server. Specifically, it uses "Amazon Transcribe." The voice is converted into text and the text information is sent to the server. The input is voice data and the output is text data.

[0199] Step 3:

[0200] Based on the information received by the server, the current refrigerator inventory database is compared with the flyer information database. The natural language processing model "GPT-3" is used to generate an appropriate menu and a shopping list of missing ingredients. The input is text data and database information, and the output is a menu and shopping list.

[0201] Step 4:

[0202] The device outputs the suggestions by voice. Using the text-to-speech conversion system "Amazon Polly," the generated text information is converted into voice and conveyed to the user. The input is the text data of the suggestions, and the output is a voice suggestion.

[0203] The above processing steps allow users to effortlessly manage refrigerator inventory, manage food delivery while cooking, and receive appropriate menu suggestions. Shopping lists are also automatically generated based on flyer information acquired daily.

[0204] (Application example 1)

[0205] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0206] Modern consumers have access to a wide variety of products and services, but lack efficient and effective means for daily food management and shopping planning. The present invention aims to automatically manage refrigerator inventory by analyzing receipts and product photos after a user's shopping trip, update inventory in real time based on voice or image input while cooking, and suggest appropriate menus and shopping lists using external flyer information. However, current technology does not offer a system that integrates these functions, resulting in significant labor-intensive tasks during shopping and actual cooking. In particular, product recognition and inventory updates during shopping are performed manually, resulting in inefficiencies and limitations in real-time inventory updates and menu suggestions.

[0207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0208] In this invention, the server includes means for taking pictures of receipts and products after shopping and analyzing product information from the captured images, means for automatically updating inventory in the refrigerator based on the analyzed product information, means for receiving information on ingredients used in cooking when the user inputs the ingredients by voice or image and updating the inventory in the refrigerator, means for obtaining flyer information from an external website every day, analyzing the information, and registering the product information in a database, means for receiving voice input from the user discussing menus and generating and suggesting appropriate menus and shopping lists based on the current inventory and the obtained flyer information, means for scanning product barcodes when shopping, means for automatically adding product information obtained from the scanned barcodes to the database, and means for creating and updating shopping lists based on the user's voice input. This allows the user to manage refrigerator inventory in real time while shopping or cooking and receive suggestions for appropriate menus and shopping lists.

[0209] A "receipt" is a paper medium that lists information such as the purchased item, the amount, and the date and time of purchase.

[0210] "Product Photos" refers to images of the products purchased by a User.

[0211] "Analysis" refers to extracting necessary data from captured images and input information.

[0212] "Inventory" refers to the food and products stored in the refrigerator.

[0213] "Voice input" is a method in which a user inputs information by voice using a microphone device.

[0214] "Image input" is a method in which a user inputs information by means of an image using a camera device.

[0215] "Flyer information" refers to advertising materials including sale information and product information issued by supermarkets and the like.

[0216] A "barcode" is a series of lines or graphics printed on a product to allow computer-readable information about the product.

[0217] A "database" is a collection of information that stores managed data and allows for rapid searching and updating.

[0218] "Menu" means a list of dishes or meals from which a User can choose.

[0219] A "shopping list" refers to a list of products or ingredients that a user plans to purchase.

[0220] "Scanning" refers to reading information from a barcode or the like.

[0221] "Suggestion" refers to presenting appropriate options or actions to the user.

[0222] "User" refers to a consumer who uses the system to shop and cook.

[0223] A "system" refers to a structure in which a series of devices and programs function in conjunction with one another.

[0224] This invention is a system that manages refrigerator inventory by analyzing receipts and product photos after users have finished shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information obtained daily.The system uses hardware and software such as a cloud server, smartphones, smart glasses, and OCR (optical character recognition) technology.

[0225] The main process is as follows:

[0226] Post-shopping inventory management:

[0227] 1. The user takes a photo of the receipt or product with their smartphone and uploads it to the cloud server. This uploads the photo to the cloud server using the smartphone's camera and internet connection functions.

[0228] 2. The cloud server analyzes the uploaded image using OCR technology to extract information such as product name and quantity. This OCR technology is used to quickly and accurately digitize product information.

[0229] 3. The extracted product information is recorded in an inventory database on the cloud server, and the refrigerator's inventory is automatically updated.

[0230] Cooking Inventory Update:

[0231] 1. When cooking, the user inputs voice or images. For example, the user inputs "I used milk" into the smartphone.

[0232] 2. The smartphone converts the voice input into text and sends it to a cloud server. The speech recognition module handles this process.

[0233] 3. The cloud server updates the inventory database based on the received information, decrements the quantity of ingredients used, and notifies the user if the ingredients are out of stock.

[0234] Daily flyer updates:

[0235] 1. The cloud server retrieves flyer information as images from supermarket websites at a specified time every day. A web scraper performs this process automatically.

[0236] 2. The server uses OCR technology to analyze the flyer image, extract product information, and obtain data such as product name, price, and validity period.

[0237] 3. The extracted product information is registered in the flyer information database and becomes accessible to users.

[0238] Menu suggestions and shopping list generation:

[0239] 1. A user speaks, "What would you like for dinner tonight?", for example, through smart glasses.

[0240] 2. The smart device converts the voice into text and sends the information to a cloud server.

[0241] 3. The server compares the current inventory database and flyer information database to generate the appropriate menu and shopping list. A generative AI model performs this analysis and recommendation.

[0242] 4. The smart device will give the user voice suggestions, for example, "To make a chicken and broccoli stir-fry, you need to buy some broccoli from the supermarket."

[0243] Inventory management when shopping:

[0244] 1. When a user shops in a physical store, they scan the product barcode with their smartphone. The barcode scanner app then sends the product information to a cloud server.

[0245] 2. The cloud server parses the product information from the scanned barcode and adds it to the inventory database.

[0246] Example of a concrete example and prompt for the generative AI model:

[0247] Examples:

[0248] When a user purchases a lettuce at a physical store and scans the barcode with their smartphone, the cloud server updates the inventory database and records "1 lettuce added."

[0249] When a user asks the smart glasses, "What's for dinner tonight?", the cloud server will suggest "hamburger steak and salad" based on the refrigerator's inventory and store special offers.

[0250] Example prompt sentence:

[0251] When a user says, "What ingredients do I need for a hot pot?", the following answers are generated based on the current refrigerator inventory and special offers:

[0252] The recipe is "Chicken Hot Pot"

[0253] refrigerator inventory

[0254] Special sale information

[0255] This allows users to enjoy an efficient and profitable shopping and cooking experience.

[0256] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0257] Step 1:

[0258] The user takes a photo of a receipt or product with their smartphone. The input is the image of the receipt or product. The output is an image file stored in the smartphone's storage. The device prepares the captured image to be uploaded to the cloud server.

[0259] Step 2:

[0260] The device uploads the captured image to the cloud server. The input is the image file stored on the smartphone. The output is the image data transferred to the cloud server. The device transmits the data via an internet connection.

[0261] Step 3:

[0262] The server analyzes the received image using OCR technology. The input is the image data transferred to the server, and the output is the extracted product information (product name, quantity, price, etc.). The server uses OCR technology to extract text data from the image.

[0263] Step 4:

[0264] The server records and updates the analyzed product information in the refrigerator's inventory database. The input is the product information extracted using OCR technology. The output is the updated inventory database. The server matches the product information with existing inventory data and updates the database.

[0265] Step 5:

[0266] When cooking, the user speaks the ingredients they will use into their smartphone. The input is the user's voice data. The output is the voice information converted into text. The device then uses a voice recognition module to convert the voice data into text.

[0267] Step 6:

[0268] The device converts voice input into text and sends the information to the server. The input is text data. The output is text information transferred to the cloud server. The device transmits the data via the Internet.

[0269] Step 7:

[0270] The server updates the inventory database based on the received information. The input is text information obtained by speech recognition. The output is the updated inventory database. The server processes the text information to reduce the amount of ingredients used.

[0271] Step 8:

[0272] The server uses a web scraper to retrieve supermarket flyer information at a specified time every day. The input is the website URL. The output is the retrieved flyer image. The server automatically retrieves flyer information using the web scraper.

[0273] Step 9:

[0274] The server uses OCR technology to analyze the acquired flyer image and extract product information. The input is the flyer image. The output is the analyzed product information (product name, price, validity period, etc.). The server uses OCR technology to extract text data from the flyer image.

[0275] Step 10:

[0276] The server registers the extracted product information in a flyer information database. The input is the product information extracted using OCR technology. The output is an updated flyer information database. The server stores the product information in the database.

[0277] Step 11:

[0278] The user speaks to the smart glasses, saying, "What would you like for dinner tonight?" The input is the user's voice data. The output is the voice information converted into text. The device uses a voice recognition module to convert the voice data into text.

[0279] Step 12:

[0280] The device converts voice input into text and sends the information to the server. The input is text data. The output is text information sent to the cloud server. The device sends the data over the Internet.

[0281] Step 13:

[0282] The server compares the current inventory database and flyer information database and generates the appropriate menu and shopping list. The input is the inventory database and flyer information database. The output is the generated menu and shopping list. The server generates the menu and list using a generative AI model.

[0283] Step 14:

[0284] The device makes audio suggestions to the user. The input is the generated menu and shopping list. The output is audio suggestions. The device uses a speaker to provide information to the user.

[0285] Step 15:

[0286] A user scans a product barcode with their smartphone in a physical store. The input is the product barcode data. The output is the barcode information stored on the smartphone. The device uses the camera function and a barcode scanner app.

[0287] Step 16:

[0288] The terminal sends the product information obtained by barcode scanning to the cloud server. The input is the barcode information. The output is the product information sent to the cloud server. The terminal sends the data via the Internet.

[0289] Step 17:

[0290] The server adds and updates the received product information to the inventory database. The input is the product information obtained by barcode scanning. The output is the updated inventory database. The server stores the product information in the database.

[0291] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0292] This system manages refrigerator inventory by analyzing receipts and product photos taken by the user after shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to the user based on supermarket flyer information acquired daily.The system also combines an emotion engine that recognizes the user's emotions using voice input and image analysis technology and adjusts the suggestions based on those emotions.

[0293] Program processing overview

[0294] Inventory management using shopping photos and receipts

[0295] 1. The user takes a photo of the receipt or product with their smartphone.

[0296] 2. Upload the photos taken by the device (smartphone) to the cloud server.

[0297] 3. The server analyzes the received image using optical character recognition (OCR) technology and extracts product information (product name, quantity, price, etc.).

[0298] 4. The server records and updates the analyzed product information in the refrigerator's inventory database.

[0299] Control delivery by voice or image input during cooking

[0300] 1. When the user prepares a dish, they tell the system the ingredients they will use using voice or images.

[0301] 2. The device converts the voice input into text and sends the information to the server. If the input is an image, the image is sent to the server.

[0302] 3. The server updates the refrigerator's inventory database based on the received information, decrementing the quantity of ingredients used and notifying the user if the ingredient is out of stock.

[0303] Daily flyer information acquisition and analysis

[0304] 1. The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[0305] 2. The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[0306] 3. The extracted product information is registered in the flyer information database.

[0307] Recognizing user emotions with an emotion engine

[0308] 1. Recognize emotions through voice input or camera when users consult the menu.

[0309] 2. The device converts the voice input into text and sends the information to the server. At the same time, the emotion engine analyzes the user's emotions from their tone of voice and facial expressions.

[0310] 3. The server adjusts the menu suggestions and shopping list contents based on the emotional information received.

[0311] Voice and emotion menu consultation and suggestions

[0312] 1. The user asks the system verbally, "What would you like for dinner tonight?"

[0313] 2. The device converts the voice input into text and sends the information to the server. At the same time, the emotion engine recognizes the user's emotions.

[0314] 3. The server compares the current refrigerator inventory database, flyer information database, and emotion information to generate an appropriate menu.

[0315] 4. The server creates a shopping list for ingredients that are in short supply and adjusts its suggestions based on feedback from the emotion engine.

[0316] 5. The device will then output a suggestion to the user, such as, "You seem a little tired today. How about a quick chicken and broccoli stir-fry?"

[0317] Specific examples

[0318] Specific examples of inventory management

[0319] A user purchases milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which uses OCR technology to extract the product information (1 liter of milk) and updates the refrigerator's inventory database with the entry "Add 1 liter of milk."

[0320] Specific examples of inventory management

[0321] While cooking, the user says "I used milk" by voice input. The device converts the voice to text and sends it to the server as "I used 1 liter of milk." The server updates the inventory database and records it as "I reduced 1 liter of milk."

[0322] Examples of flyer information

[0323] At 9:00 AM, the server retrieves the flyer image from the supermarket's website, analyzes the product information "Broccoli 98 yen," and registers it in the flyer information database. It saves it as "Broccoli 98 yen, sale period: 3 days."

[0324] Examples of emotion engines

[0325] The user asks, "What should we have for dinner tonight?" The emotion engine determines from the user's tone of voice and facial expression that they seem tired. The device converts this speech into text and sends it to the server. Based on the emotion engine's feedback, the server compares the refrigerator's inventory with flyer information and suggests, "You seem a little tired today, how about a simple chicken and broccoli stir-fry?"

[0326] This system provides a multifunctional mechanism that includes these steps and can support users in daily food management, menu planning, and shopping planning, including emotional information.

[0327] The processing flow will be explained below.

[0328] Manage refrigerator inventory using shopping photos and receipts

[0329] Step 1:

[0330] The user takes a photo of the receipt or product with their smartphone.

[0331] Open the camera app on your smartphone and take a photo of your receipt or purchased items.

[0332] Step 2:

[0333] The device uploads the photos it takes to the server.

[0334] Use the dedicated application and press the button to upload the photo to the cloud server.

[0335] Step 3:

[0336] The server applies OCR technology to analyze the received image.

[0337] The server invokes an image processing module to extract text information from the image.

[0338] Step 4:

[0339] The server extracts the parsed product information (product name, quantity, price, etc.).

[0340] Product name, quantity, and price are identified from the text information and converted into a database format.

[0341] Step 5:

[0342] The server records the extracted product information in the refrigerator's inventory database and updates it.

[0343] Adds the product information to the corresponding user's inventory database and updates existing inventory information.

[0344] Control delivery by voice or image input during cooking

[0345] Step 1:

[0346] When cooking, the user verbally tells the system what ingredients to use.

[0347] Use your smartphone's microphone to input voice instructions such as "I used chicken."

[0348] Step 2:

[0349] The device converts the voice input into text and sends the information to the server.

[0350] Uses voice recognition software to convert speech into text.

[0351] Step 3:

[0352] The server updates the refrigerator inventory database based on the received information.

[0353] The quantity of the specified ingredient is reduced and the inventory database is updated.

[0354] Step 4:

[0355] If the server is out of stock, the user is notified.

[0356] If the stock reaches 0, a notification message will be sent to the user.

[0357] Daily flyer information acquisition and analysis

[0358] Step 1:

[0359] The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[0360] Based on a fixed schedule, it automatically visits each supermarket's website and downloads flyer images.

[0361] Step 2:

[0362] The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[0363] Use the flyer image processing module to extract product information from the image.

[0364] Step 3:

[0365] The extracted product information is registered in a flyer information database.

[0366] The flyer information is stored in a database based on the product name, price, and validity period.

[0367] Recognizing user emotions with an emotion engine

[0368] Step 1:

[0369] When the user consults the menu, he or she uses voice input.

[0370] Using the smartphone's microphone, voice menu consultation is performed, asking, "What would you like for dinner tonight?"

[0371] Step 2:

[0372] The device converts the voice input into text and sends the information to the server.

[0373] Use speech recognition software to convert speech to text.

[0374] Step 3:

[0375] The emotion engine analyzes emotions from the user's tone of voice.

[0376] A voice tone analysis module is used to identify the user's emotions.

[0377] Step 4:

[0378] The server adjusts the suggested menu and shopping list based on the emotional information received.

[0379] Taking emotion information into consideration, simple dishes and dishes that the user likes are selected.

[0380] Voice and emotion menu consultation and suggestions

[0381] Step 1:

[0382] The user verbally asks the system, "What would you like for dinner tonight?"

[0383] Voice menu consultation is performed using the smartphone's microphone.

[0384] Step 2:

[0385] The device converts the voice input into text and sends the information to the server.

[0386] Use speech recognition software to convert speech to text.

[0387] Step 3:

[0388] The server compares the current refrigerator inventory database, flyer information database, and emotion information.

[0389] Cross-check the database to see what ingredients are available and what discounts are available.

[0390] Step 4:

[0391] Generate appropriate menus and shopping lists for missing ingredients.

[0392] Generate suggested recipes and add missing ingredients to your shopping list.

[0393] Step 5:

[0394] The terminal outputs the suggestions to the user by voice.

[0395] Using the smartphone's speaker, the system makes a voice suggestion such as, "You seem a little tired today, how about an easy-to-make chicken and broccoli stir-fry?"

[0396] Through the above processing flow, this system can efficiently support users in managing ingredients, planning menus, and making shopping plans, including emotional information.

[0397] Example 2

[0398] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0399] In today's busy lifestyles, food inventory management has become a complicated task for many households. Deciding on a menu and creating a shopping list for the necessary ingredients is also a time-consuming task. Furthermore, there is a demand for appropriate menu suggestions based on the user's emotions and physical condition, but current technology has difficulty meeting this demand.

[0400] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for taking pictures of receipts and products after shopping and analyzing product information from the captured images; means for automatically updating inventory in the refrigerator based on the analyzed product information; means for receiving information on ingredients used in cooking when the user inputs the ingredients by voice or image and updating the inventory in the refrigerator; means for acquiring flyer information from an external website every day, analyzing the information, and registering the product information in a database; means for receiving voice input from the user discussing menus, and generating and suggesting appropriate menus and shopping lists based on the current inventory and the acquired flyer information; means for recognizing emotions from the user's voice tone and facial expressions; and means for adjusting the suggestions based on the emotion recognition information. This enables efficient and accurate daily ingredient management, menu planning, and shopping planning, including emotional information.

[0401] "Receipts and product photos after shopping" refers to receipts showing purchase details that users receive when they purchase products at a store, and image data of purchased products.

[0402] "Product Information" means detailed data about the purchased item, such as product name, quantity, and price.

[0403] "Optical character recognition (OCR) technology" refers to the technology that scans images and handwritten characters and converts them into text data.

[0404] "Inventory database" refers to a database system for recording and managing inventory status and product information in a refrigerator.

[0405] "Voice input" means a means by which a user communicates information to a system using speech.

[0406] "Image input" means the means by which a user communicates information to a system using an image.

[0407] "Web scraping" refers to the technique of extracting data from websites using automated programs.

[0408] "Flyer information" refers to data including special sale items and price information provided by stores for promotional purposes.

[0409] "Emotion engine" refers to algorithms and technologies that analyze a user's tone of voice and facial expressions to recognize their emotional state.

[0410] "Suggestions" refers to information such as menus and shopping lists provided to users.

[0411] "Refrigerator inventory" refers to the ingredients and products stored in the user's refrigerator.

[0412] "Voice input for menu consultation" refers to voice input by the user to ask the system about the dish they would like to make.

[0413] This system analyzes receipts and product photos taken by users after shopping, manages refrigerator inventory, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information acquired daily.The system also combines an emotion engine that recognizes the user's emotions using voice input and image analysis technology and adjusts the suggestions based on those emotions.

[0414] Inventory management using shopping photos and receipts

[0415] Users take photos of receipts and products with their smartphones. The device uploads these images to a cloud server. The server then analyzes the images using optical character recognition (OCR) technology. Specifically, OCR software such as Tesseract OCR is used to extract product information such as product name, quantity, and price. The server then records this information in the refrigerator's inventory database and automatically updates it.

[0416] Examples:

[0417] A user buys milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which uses Tesseract OCR to extract the product information (1 liter of milk) and updates the refrigerator inventory with the entry "Add 1 liter of milk."

[0418] Control delivery by voice or image input during cooking

[0419] When a user prepares a dish, they tell the system which ingredients they will use using voice or images. The device converts the voice input into text and sends that information to the server. If an image is input, the image is sent to the server. The server updates the refrigerator's inventory database based on the received information and reduces the quantity of ingredients used. If the inventory falls below a certain level, the user is notified.

[0420] Examples:

[0421] While cooking, the user voice-inputs "Milk used." The device converts the voice data into text and sends it to the server as "1 liter of milk used." The server updates the inventory database and records "1 liter of milk decreased."

[0422] Daily flyer information acquisition and analysis

[0423] The server retrieves flyer information as image data from each supermarket's website at a specified time every day. Libraries such as Beautiful Soup and Selenium are used for web scraping. The server analyzes the flyer images retrieved using OCR technology and extracts product information (product name, price, validity period, etc.). This information is registered in the flyer information database.

[0424] Examples:

[0425] The server retrieves the flyer image from the supermarket's website at 9:00 AM, analyzes the product information "Broccoli 98 yen" using Beautiful Soup, and saves it in the flyer information database as "Broccoli 98 yen, sale period: 3 days."

[0426] Recognizing user emotions with an emotion engine

[0427] When a user consults about a menu, emotions are recognized through voice input or a camera. The device converts the voice input into text and sends the information to the server. At the same time, the device is equipped with an emotion analysis engine such as OpenCV or Face API, which analyzes emotions from the user's tone of voice and facial expressions. The server then adjusts the menu suggestions and shopping list contents based on this emotional information.

[0428] Examples:

[0429] The user asks, "What should we have for dinner tonight?" The device converts this speech into text, and at the same time, the emotion engine detects tired facial expressions and analyzes them. Based on the feedback from the emotion engine, the server adjusts the suggestions, such as "You seem a little tired today, so I'll recommend some easy-to-make dishes."

[0430] Voice and emotion menu consultation and suggestions

[0431] The user verbally asks the system, "What would you like for dinner tonight?" The device converts the voice input into text and sends the information to the server. At the same time, the emotion engine recognizes the user's emotions. The server compares the current refrigerator inventory database, flyer information database, and emotion information to generate an appropriate menu. The server creates a shopping list for missing ingredients and further adjusts this list based on feedback from the emotion engine. The device then outputs the suggestions to the user verbally.

[0432] Examples:

[0433] The user asks the system via voice, "What would you like for dinner tonight?" The device converts the text and sends the emotion data to the server, which then makes a suggestion to the user via voice, such as, "You seem a little tired today, how about a quick chicken and broccoli stir-fry?"

[0434] Example prompt sentence:

[0435] "Take a photo of your supermarket receipt and upload it"

[0436] "List the ingredients used while cooking by voice"

[0437] "I'd like some suggestions for dinner tonight."

[0438] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0439] Inventory management using shopping photos and receipts

[0440] Step 1:

[0441] After shopping, the user takes a photo of the receipt or product with their smartphone. The user launches the camera app and takes a photo, taking care to ensure that the receipt and product are clearly visible. This is the initial step in obtaining input data (receipt photo).

[0442] Step 2:

[0443] The device uploads the captured image to the cloud server. The device uses the HTTP protocol via an internet connection to send the image data to the server. The input is a photo of the receipt, and the output is the transmission of image data to the server.

[0444] Step 3:

[0445] The image received by the server is analyzed using OCR technology. Software such as Tesseract OCR is used to recognize characters in the image and extract text information such as product name, quantity, and price. The input is image data, and the output is analyzed product information.

[0446] Step 4:

[0447] The server records the extracted product information in the refrigerator's inventory database and updates it. The database update process keeps the inventory information up to date. The input is the analyzed product information, and the output is the updated inventory database.

[0448] Control delivery by voice or image input during cooking

[0449] Step 1:

[0450] When a user prepares a dish, they tell the system what ingredients they will use by voice or image. For example, they can say "I used milk." This is input data for recording the ingredients used.

[0451] Step 2:

[0452] The device converts the user's voice input into text data. It uses Google Cloud Speech-to-Text, a speech recognition technology, to convert speech to text. The input is voice data, and the output is text data.

[0453] Step 3:

[0454] The terminal sends the converted text data to the server via an Internet connection using the HTTP protocol. The input is text data, and the output is data sent to the server.

[0455] Step 4:

[0456] The server updates the refrigerator's inventory database based on the received data. It reduces the quantity of ingredients used and notifies the user when the stock falls below a certain level. The input is text data, and the output is the updated inventory database and a notification.

[0457] Daily flyer information acquisition and analysis

[0458] Step 1:

[0459] The server retrieves flyer images from each supermarket's website at a specified time every day using a web scraping tool (e.g., Beautiful Soup or Selenium). The input is the supermarket's website URL, and the output is the flyer image.

[0460] Step 2:

[0461] The server analyzes the flyer image acquired by the server using OCR technology and extracts product information. The input is the flyer image and the output is the analyzed product information.

[0462] Step 3:

[0463] The server registers the analyzed product information in a flyer information database. The database is updated and sale information is stored. The input is the analyzed product information, and the output is the updated flyer information database.

[0464] Recognizing user emotions with an emotion engine

[0465] Step 1:

[0466] Emotion recognition is performed through voice input or camera when the user is consulting a menu. For example, the user faces the camera and says, "What should I have for dinner tonight?" This is the initial step in obtaining input data (voice and image).

[0467] Step 2:

[0468] The device converts the user's voice input into text and sends that information to the server. At the same time, the emotion engine analyzes emotions from voice tone and facial expressions. OpenCV and Face API are used. The input is voice and image data, and the output is text data and emotion data.

[0469] Step 3:

[0470] The server adjusts menu suggestions based on emotional information. It compares the inventory database and flyer information to generate the optimal menu for the user. The input is emotional data and inventory / flyer information, and the output is the adjusted menu suggestions.

[0471] Voice and emotion menu consultation and suggestions

[0472] Step 1:

[0473] The user verbally asks the system what dish to make, for example, "What would you like for dinner tonight?" This is the input data for the suggestion.

[0474] Step 2:

[0475] The device converts voice input into text and sends it to the server. At the same time, the emotion engine recognizes the user's emotions. The input is voice data, and the output is text data and emotion data.

[0476] Step 3:

[0477] The server generates an appropriate menu based on the current refrigerator inventory database, flyer information database, and emotion information. The input is inventory and flyer information and emotion data, and the output is the generated menu.

[0478] Step 4:

[0479] The server creates a shopping list of missing ingredients and further refines the suggestions. The input is the generated menu, and the output is the shopping list and refined suggestions.

[0480] Step 5:

[0481] The device outputs a suggestion to the user by voice, for example, "You seem a little tired today, how about a quick chicken and broccoli stir-fry?" The input is the tailored suggestion, and the output is the voice prompt.

[0482] (Application example 2)

[0483] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0484] Conventional refrigerator inventory management systems have the problem that updating inventory is cumbersome and difficult for users to use. They also lack a way for users to easily grasp the ingredients and menu items they need, and are unable to make suggestions based on their emotions. Furthermore, while real-time inventory updates and emotion-based suggestions are required, conventional systems have been unable to comprehensively address these issues.

[0485] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0486] In this invention, the server includes means for taking pictures of receipts and items after shopping and analyzing item information from the captured images, means for automatically updating inventory in the refrigerator based on the analyzed item information, means for receiving information on ingredients used in cooking when the user inputs the information by voice or image and updating the inventory in the refrigerator, means for acquiring flyer information from an external website every day, analyzing the information, and registering the item information in a database, means for receiving voice input from the user discussing menus, and generating and suggesting appropriate menus and shopping lists based on the current inventory and the acquired flyer information, means for analyzing user emotional information, and means for adjusting the suggestions based on the analyzed emotional information. This makes it possible to efficiently manage inventory in the refrigerator and generate menu suggestions and shopping lists that reflect the user's emotions.

[0487] A "post-purchase receipt" is a paper statement issued after the purchase of an item, which contains information including the name, quantity, and price of the purchased item.

[0488] An "item photo" is an image of a purchased item taken by a user after shopping, and includes the appearance and label information of the item.

[0489] "Item information" refers to detailed information such as the name, quantity, and price of an item, and is data extracted by image analysis or character recognition.

[0490] "Inventory in refrigerator" is a database that shows the total amount of food and beverages stored in a refrigerator or other refrigeration device.

[0491] "Voice input" is a means by which the user verbally communicates to the system the materials to be used, the details of the proposal, etc., which is converted into text data using voice recognition technology.

[0492] "Image input" is a means by which a user sends an image of the material or item they are using to the system, and information is extracted using image analysis techniques.

[0493] "External websites" are web pages of commercial facilities or information providers that exist on the Internet, and on which flyer information and the like are made public.

[0494] "Flyer information" is image data or text data that includes special sale information and product price lists offered by commercial facilities.

[0495] A "database" is an electronic information aggregation system for systematically managing and storing analyzed product information, inventory information, and flyer information.

[0496] "User's emotional information" is data that indicates the emotional state of the user, which is analyzed from the user's tone of voice, facial expression, content of words, and the like.

[0497] "Suggestion content" refers to information such as menus and shopping lists that the system provides to users, and is generated based on inventory data, emotional information, and the like.

[0498] This invention begins when a user takes a photo of a receipt or item after shopping with their smartphone and uploads it to a cloud server. The server analyzes the image using optical character recognition (OCR) technology to extract item information (item name, quantity, price, etc.). This item information is recorded in an inventory database inside the refrigerator and automatically updated. When the user prepares a meal, they tell the system which ingredients they will be using via voice or image input. The device converts the voice to text and sends that information to the server. The server updates the inventory database based on the received information and reduces the quantity of ingredients used. In addition, flyer information is obtained daily from an external website, analyzed as an image, and item information (item name, price, expiration date, etc.) is extracted. This information is registered in the flyer information database.

[0499] The system's main hardware includes a smartphone (camera, microphone) and a cloud server. The main software used includes optical character recognition (OCR), voice input analysis, and image analysis technologies, which utilize cloud services such as Google Cloud Vision API and Google Cloud Speech-to-Text. When a user asks the system, "What would you like for dinner tonight?", the voice is recorded by the smartphone's microphone. The device converts the voice into text and sends it to the cloud server. At the same time, an emotion engine analyzes the user's emotions from their tone of voice and facial expressions. The server compares the current refrigerator inventory database, flyer information database, and emotion information to generate an appropriate menu. It creates a shopping list for any missing ingredients and adjusts the suggestions based on feedback from the emotion engine.

[0500] For example, a user purchases milk at a supermarket and takes a photo of the receipt with their smartphone. When this image is uploaded to the cloud server, OCR technology is used to extract the product information—"1 liter of milk." This information is recorded in the refrigerator's inventory database and updated as "1 liter of milk added." Furthermore, if the user voice-inputs "I used milk" while cooking, this voice is converted to text and sent to the cloud server. The server updates the inventory database based on this information, recording it as "1 liter of milk reduced." Furthermore, the server retrieves flyer images from each supermarket's website at a specified time each morning, analyzes the product information, such as "Broccoli 98 yen," and registers it in the flyer information database. This information is saved as "Broccoli 98 yen, special sale period: 3 days." The emotion engine determines that the user is tired based on their tone of voice and facial expression when they ask, "What should I have for dinner tonight?" The server then compares the refrigerator's inventory with the flyer information based on the emotion engine's feedback and suggests, "You seem tired today. How about a simple chicken and broccoli stir-fry?"

[0501] An example of a prompt for a generative AI model might be, "Please tell me how to analyze receipts and item photos taken by the user and update that information in the refrigerator inventory database."

[0502] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0503] Step 1:

[0504] After shopping, the user takes a photo of the receipt or item with their smartphone. The input is the captured image, and the output is an image file saved in the smartphone's storage. Specifically, the user opens the smartphone's camera app, takes a photo of the receipt or item, and saves it as an image file.

[0505] Step 2:

[0506] The device uploads the captured image to the cloud server. The input is the image file stored on the smartphone, and the output is the image data uploaded to the cloud server. Specifically, the device establishes communication with the cloud server and sends the image file captured by the user to the server.

[0507] Step 3:

[0508] The server analyzes the image using optical character recognition (OCR) technology and extracts product information. The input is the image data uploaded to the cloud server, and the output is text data such as the product name, quantity, and price. Specifically, the server uses OCR technology (e.g., Tesseract) to analyze the text portion of the image and extracts product information as text data.

[0509] Step 4:

[0510] The server records the analyzed item information in the inventory database inside the cooler and updates it. The input is text data extracted using OCR technology, and the output is updated inventory data. Specifically, the server accesses the inventory database and updates the inventory data based on the extracted item information.

[0511] Step 5:

[0512] The user communicates the ingredients used in cooking to the system through voice or image input. The input is the user's voice instruction or image data, and the output is information sent by the device to the server. Specifically, the user communicates the ingredients used to the system using the microphone or camera on their smartphone.

[0513] Step 6:

[0514] The device converts voice input into text and sends the information to the server. The input is the user's voice data, and the output is text data. Specifically, the device converts the voice into text using voice recognition technology (e.g., Google Cloud Speech-to-Text) and sends it to the server.

[0515] Step 7:

[0516] The server updates the inventory database in the refrigerator based on the received information. The input is text data or image data, and the output is updated inventory data. Specifically, the server accesses the inventory database and decreases the quantity of materials used.

[0517] Step 8:

[0518] The server retrieves flyer information from an external website every day and analyzes it as an image. The input is the website URL, and the output is flyer image data. Specifically, the server accesses the website at a specified time and downloads the flyer image.

[0519] Step 9:

[0520] The server analyzes the flyer image and registers the product information (product name, price, validity period, etc.) in a database. The input is the image data of the flyer, and the output is the product information registered in the database. Specifically, the server analyzes the flyer image using OCR technology, extracts the product information, and stores it in the database.

[0521] Step 10:

[0522] The system receives voice input from the user to discuss a menu, and the server generates an appropriate menu and shopping list based on current inventory and flyer information. The input is the voice instruction and inventory and flyer information, and the output is a suggested menu and shopping list. Specifically, the server analyzes the voice instruction and compares the inventory data with the flyer information to generate an appropriate menu.

[0523] Step 11:

[0524] The server analyzes the user's emotional information and adjusts the suggestions. The input is the user's voice tone and facial expression data, and the output is the adjusted suggestions. Specifically, the server analyzes the user's emotions using emotion recognition technology (e.g., Google Cloud Vision API) and adjusts the suggestions based on the emotions.

[0525] Step 12:

[0526] The device outputs the adjusted proposal content to the user by voice. The input is the adjusted proposal content, and the output is a voice notification to the user. Specifically, the device converts text data into voice and notifies the user of the proposal content.

[0527] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0528] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0529] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0530] [Second embodiment]

[0531] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0532] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0533] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0534] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0535] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0536] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0537] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0538] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0539] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0540] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0541] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0542] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0543] The present invention is a system that manages refrigerator inventory by analyzing receipts and product photos taken by users after shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information obtained daily.

[0544] Program processing overview

[0545] Inventory management using receipts and product photos after shopping

[0546] 1. The user takes a photo of the receipt or product with their smartphone.

[0547] 2. Upload the photos taken by the device (smartphone) to the cloud server.

[0548] 3. The server analyzes the received image using optical character recognition (OCR) technology and extracts product information (product name, quantity, price, etc.).

[0549] 4. The server records and updates the analyzed product information in the refrigerator's inventory database.

[0550] Control delivery by voice or image input during cooking

[0551] 1. When the user prepares a dish, they tell the system the ingredients they will use using voice or images.

[0552] 2. The device converts the voice input into text and sends the information to the server. If the input is an image, the image is sent to the server.

[0553] 3. The server updates the refrigerator's inventory database based on the received information, decrementing the quantity of ingredients used and notifying the user if the ingredient is out of stock.

[0554] Daily flyer information acquisition and analysis

[0555] 1. The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[0556] 2. The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[0557] 3. The extracted product information is registered in the flyer information database.

[0558] Voice menu consultation and suggestions

[0559] 1. The user asks the system verbally, "What would you like for dinner tonight?"

[0560] 2. The device converts the voice input into text and sends the information to the server.

[0561] 3. Based on the information received by the server, the current refrigerator inventory database is compared with the flyer information database to generate an appropriate menu and a shopping list of missing ingredients.

[0562] 4. The device will output a suggestion to the user via voice, such as "To make a chicken and broccoli stir-fry, you will need to buy broccoli from the supermarket."

[0563] Specific examples

[0564] Specific examples of inventory management

[0565] A user purchases milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which uses OCR technology to extract the product information (1 liter of milk) and updates the refrigerator's inventory database with the entry "Add 1 liter of milk."

[0566] Specific examples of inventory management

[0567] While cooking, the user says "I used milk" by voice input. The device converts the voice to text and sends it to the server as "I used 1 liter of milk." The server updates the inventory database and records it as "I reduced 1 liter of milk."

[0568] Examples of flyer information

[0569] At 9:00 AM, the server retrieves the flyer image from the supermarket's website, analyzes the product information "Broccoli 98 yen," and registers it in the flyer information database. It saves it as "Broccoli 98 yen, sale period: 3 days."

[0570] Specific examples of menu consultation

[0571] The user asks, "What should I have for dinner tonight?" The device converts this speech into text and sends it to the server. The server compares the current inventory database with the flyer information database, determines that "there is chicken in the fridge and broccoli on sale," and suggests "stir-fried chicken and broccoli."

[0572] This system provides a multifunctional mechanism that includes these steps, and centrally and automatically supports users in daily food management, menu planning, and shopping planning.

[0573] The processing flow will be explained below.

[0574] Manage refrigerator inventory using shopping photos and receipts

[0575] Step 1:

[0576] The user takes a photo of the receipt or product with their smartphone.

[0577] Open the camera app on your smartphone and take a photo of your receipt or purchased items.

[0578] Step 2:

[0579] The device uploads the photos it takes to the server.

[0580] Use the dedicated application and press the button to upload the photo to the cloud server.

[0581] Step 3:

[0582] The server applies OCR technology to analyze the received image.

[0583] The server invokes an image processing module to extract text information from the image.

[0584] Step 4:

[0585] The server extracts the parsed product information (product name, quantity, price, etc.).

[0586] Product name, quantity, and price are identified from the text information and converted into a database format.

[0587] Step 5:

[0588] The server records the extracted product information in the refrigerator's inventory database and updates it.

[0589] Adds the product information to the corresponding user's inventory database and updates existing inventory information.

[0590] Control delivery by voice or image input during cooking

[0591] Step 1:

[0592] When cooking, the user verbally tells the system what ingredients to use.

[0593] Use your smartphone's microphone to input voice instructions such as "I used chicken."

[0594] Step 2:

[0595] The device converts the voice input into text and sends the information to the server.

[0596] Uses voice recognition software to convert speech into text.

[0597] Step 3:

[0598] The server updates the refrigerator inventory database based on the received information.

[0599] The quantity of the specified ingredient is reduced and the inventory database is updated.

[0600] Step 4:

[0601] If the server is out of stock, the user is notified.

[0602] If the stock reaches 0, a notification message will be sent to the user.

[0603] Daily flyer information acquisition and analysis

[0604] Step 1:

[0605] The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[0606] Based on a fixed schedule, it automatically visits each supermarket's website and downloads flyer images.

[0607] Step 2:

[0608] The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[0609] Use the flyer image processing module to extract product information from the image.

[0610] Step 3:

[0611] The extracted product information is registered in a flyer information database.

[0612] The flyer information is stored in a database based on the product name, price, and validity period.

[0613] Voice menu consultation and suggestions

[0614] Step 1:

[0615] The user verbally asks the system, "What would you like for dinner tonight?"

[0616] Voice menu consultation is performed using the smartphone's microphone.

[0617] Step 2:

[0618] The device converts the voice input into text and sends the information to the server.

[0619] Use speech recognition software to convert speech to text.

[0620] Step 3:

[0621] Based on the information received by the server, the current refrigerator inventory database is compared with the flyer information database.

[0622] Cross-check the database to see what ingredients are available and what discounts are available.

[0623] Step 4:

[0624] Generate appropriate menus and shopping lists for missing ingredients.

[0625] Generate suggested recipes and add missing ingredients to your shopping list.

[0626] Step 5:

[0627] The terminal outputs the suggestions to the user by voice.

[0628] Using the smartphone's speaker, a voice suggestion is made: "To make chicken and broccoli stir-fry, you need to buy broccoli at the supermarket."

[0629] Through the above processing flow, this system efficiently supports the user in managing ingredients, planning menus, and making shopping plans.

[0630] Example 1

[0631] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0632] In modern life, efficient ingredient management and menu planning are important challenges for many households. However, manually managing ingredients and creating appropriate menus and shopping lists is time-consuming and labor-intensive, making it a burden for many people. In particular, manually collecting and using information from refrigerator inventory and supermarket flyers is cumbersome and inefficient. Additionally, it is difficult to update the inventory of ingredients used during cooking in real time, resulting in problems such as out-of-stock situations and food waste.

[0633] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0634] In this invention, the server includes means for taking pictures of receipts and products after a user has finished shopping and analyzing the product information from the captured images, means for automatically updating the inventory in the refrigerator based on the analyzed product information, means for receiving information about ingredients used in cooking when the user inputs the information by voice or image and updating the inventory in the refrigerator, means for acquiring flyer information from an external website every day, analyzing the information, and registering the product information in a database, means for receiving voice input from the user discussing menus and generating and suggesting appropriate menus and shopping lists based on the current inventory and the acquired flyer information, means for converting the user's voice input into text using a voice recognition system and sending the information to the server, and means for outputting suggestions by voice using a text-to-speech conversion system.This allows the user to efficiently manage their refrigerator inventory without any hassle and have appropriate menus and shopping lists automatically suggested.

[0635] A "user" is an entity that uses the system to manage ingredients, create menus, and plan shopping.

[0636] A "receipt" is a paper record of a purchase made at a store.

[0637] "Product photo" is an image of the purchased product.

[0638] "Means for analyzing product information from images" refers to a method for extracting information such as product name, quantity, and price from images using optical character recognition technology.

[0639] The "means for automatically updating inventory in the refrigerator" is a system that updates the database based on analyzed product information to reflect the new inventory status.

[0640] "Means for inputting information about ingredients used in cooking by voice or image" refers to a method in which the user communicates information about ingredients used to the system through voice or image.

[0641] A "voice recognition system" is a technology that converts voice input into text.

[0642] The "means for updating inventory" is a system that modifies the inventory database based on user input and maintains the latest inventory status.

[0643] "Means for obtaining daily flyer information from external websites" refers to a technology for downloading flyer images from websites of supermarkets and the like via the Internet.

[0644] The "means of analyzing and registering product information in a database" refers to a system that analyzes acquired flyer images using optical character recognition technology and stores the product information in a database.

[0645] The "means for receiving voice input for consulting a menu" is a method for receiving information in which a user requests menu suggestions by voice.

[0646] "Means for generating appropriate menus and shopping lists" refers to technology that creates optimal cooking menus and lists of ingredients that are in short supply for users based on current inventory information and flyer information.

[0647] "Means for converting a user's voice input into text using a voice recognition system" refers to a method that uses technology to convert voice data into text data.

[0648] The "means for outputting suggestions by voice using a text-to-voice conversion system" is a technology for converting generated text information into voice and conveying it to the user.

[0649] A "natural language processing model" is a technology that generates appropriate text and analyzes intent based on user input.

[0650] This invention is a system that manages refrigerator inventory by analyzing receipts and product photos taken by users after shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information obtained daily.

[0651] The system is implemented using the following hardware and software:

[0652] Hardware:

[0653] Smartphone (user device)

[0654] Cloud Server

[0655] Refrigerator (physical storage location)

[0656] software:

[0657] Optical character recognition technology (OCR): Google Cloud Vision API

[0658] Speech recognition system: Google Cloud Speech-to-Text, Amazon Transcribe

[0659] Text-to-speech system: Amazon Polly

[0660] Web scraping tools: Selenium, BeautifulSoup

[0661] Natural language processing model: GPT-3

[0662] Database management systems: MySQL, PostgreSQL

[0663] Voice UI Applications

[0664] After a user finishes shopping at the supermarket, they take a photo of the receipt or product with their smartphone. The device uploads the photo to a cloud server, which then analyzes the image using the Google Cloud Vision API. Using OCR technology, product information (product name, quantity, price, etc.) is extracted, and the server automatically registers and updates the analyzed information in the refrigerator's inventory database.

[0665] When cooking, the user can tell the system which ingredients they will use by voice or image input. The device converts the voice input into text using Google Cloud Speech-to-Text or Amazon Transcribe and sends the information to the server. When using image input, the device also sends the image to the server, which updates the inventory database based on the analysis results. The server reduces the quantity of ingredients used and notifies the user if they are out of stock.

[0666] The server also retrieves flyer images from each supermarket's website at a specified time every day using web scraping tools (Selenium, BeautifulSoup). The server then analyzes the flyer images using Tesseract OCR, extracts product information (product name, price, validity period, etc.), and registers it in the flyer information database.

[0667] When a user asks the system, "What would you like for dinner tonight?", the device converts the speech to text using Amazon Transcribe and sends it to the server. The server compares the current inventory database with the flyer information database and uses the GPT-3 natural language processing model to generate an appropriate menu and a shopping list of missing ingredients. The device then uses Amazon Polly to output the suggestions to the user.

[0668] Specific examples

[0669] For example, suppose a user buys milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which then uses the Google Cloud Vision API to extract the product information, such as "1 liter of milk." The information is then registered and updated in the refrigerator's inventory database as "1 liter of milk added."

[0670] If a user says "I used milk" while cooking, the device converts the speech into text using Google Cloud Speech-to-Text and sends it to the server as "I used 1 liter of milk." The server then updates the inventory database, recording "I reduced 1 liter of milk."

[0671] Every day at 9:00 a.m., the server uses Selenium to retrieve flyer images from the supermarket's website, analyzes the product information, such as "Broccoli 98 yen," using Tesseract OCR, and registers it in the flyer information database.

[0672] When a user asks, "What should I have for dinner tonight?", the device converts the speech into text using Amazon Transcribe and sends it to the server. The server compares the current inventory database with the flyer information database and uses the GPT-3 natural language processing model to suggest "chicken and broccoli stir-fry." The device then uses Amazon Polly to announce, "To make chicken and broccoli stir-fry, you need to buy broccoli at the supermarket."

[0673] Prompt Sentence Examples

[0674] Example prompt sentences when the user says "I used milk":

[0675] The user says "Milk used." Update the refrigerator inventory database. Decrease the quantity of milk by 1 liter and record the result.

[0676] Example prompt sentences when a user says "What should I have for dinner tonight?":

[0677] The user says, "What should I have for dinner tonight?" Match the current refrigerator inventory database with the flyer information database to suggest an appropriate dinner menu. If there are any ingredients missing, prepare a shopping list for them.

[0678] The above is a specific embodiment for carrying out the present invention.

[0679] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0680] Inventory management using receipts and product photos after shopping

[0681] Step 1:

[0682] After a user finishes shopping at a supermarket, they take a photo of the receipt or product with their smartphone. The user opens the smartphone's camera app and takes a photo of the receipt or product. This image becomes input data for the system.

[0683] Step 2:

[0684] The device (smartphone) takes a photo and uploads it to the cloud server. The upload program encodes the image data and sends it to the cloud server via a secure connection. The input is an image of the receipt or product, and the output is the storage of the image data on the cloud server.

[0685] Step 3:

[0686] The server analyzes the received photos using optical character recognition (OCR) technology. Specifically, it uses the Google Cloud Vision API to extract product information such as product name, quantity, and price from the image. The input is image data, and the output is analyzed product information (text data).

[0687] Step 4:

[0688] The server records and updates the parsed product information in the refrigerator's inventory database. The database management system (e.g., MySQL) adds the new product information and updates the existing inventory data. The input is the parsed product information, and the output is the updated inventory database.

[0689] Control delivery by voice or image input during cooking

[0690] Step 1:

[0691] When a user prepares a dish, they can tell the system which ingredients they will use by voice or image. Specifically, the user opens the smartphone app and either voice-inputs "I used milk" or takes a photo of the ingredients used. The input is voice data or image data.

[0692] Step 2:

[0693] The device converts voice input into text and sends that information to the server. "Google Cloud Speech-to-Text" is used as the voice recognition system. Voice is converted into text and the textual information is sent to the server. The input is voice data and the output is text data. In the case of image input, the image is sent directly to the server. The input is image data and the output is saving the image data to the server.

[0694] Step 3:

[0695] The server updates the refrigerator's inventory database based on the information it receives. Specifically, it analyzes the text data and reduces the quantity of ingredients used. If the item is out of stock, a program is activated to notify the user. The input is the text data, and the output is the updated inventory database and a notification.

[0696] Daily flyer information acquisition and analysis

[0697] Step 1:

[0698] The server retrieves flyer information as images from each supermarket's website at a specified time every day. Specifically, the web scraping tools "Selenium" and "BeautifulSoup" are used to download the flyer images. The input is the supermarket's website URL, and the output is the retrieved flyer image.

[0699] Step 2:

[0700] The server analyzes the flyer image it has acquired. Specifically, it uses "Tesseract OCR" to extract product information (product name, price, validity period, etc.). The input is the flyer image, and the output is the analyzed product information (text data).

[0701] Step 3:

[0702] The extracted product information is registered in a flyer information database. New flyer information is added and saved using a database management system (e.g., PostgreSQL). The input is the parsed product information, and the output is an updated flyer information database.

[0703] Voice menu consultation and suggestions

[0704] Step 1:

[0705] The user asks the system by voice, "What would you like for dinner tonight?" The user opens the app on their smartphone and enters voice input. The input is voice data.

[0706] Step 2:

[0707] The device converts voice input into text and sends that information to the server. Specifically, it uses "Amazon Transcribe." The voice is converted into text and the text information is sent to the server. The input is voice data and the output is text data.

[0708] Step 3:

[0709] Based on the information received by the server, the current refrigerator inventory database is compared with the flyer information database. The natural language processing model "GPT-3" is used to generate an appropriate menu and a shopping list of missing ingredients. The input is text data and database information, and the output is a menu and shopping list.

[0710] Step 4:

[0711] The device outputs the suggestions by voice. Using the text-to-speech conversion system "Amazon Polly," the generated text information is converted into voice and conveyed to the user. The input is the text data of the suggestions, and the output is a voice suggestion.

[0712] The above processing steps allow users to effortlessly manage refrigerator inventory, manage food delivery while cooking, and receive appropriate menu suggestions. Shopping lists are also automatically generated based on flyer information acquired daily.

[0713] (Application example 1)

[0714] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0715] Modern consumers have access to a wide variety of products and services, but lack efficient and effective means for daily food management and shopping planning. The present invention aims to automatically manage refrigerator inventory by analyzing receipts and product photos after a user's shopping trip, update inventory in real time based on voice or image input while cooking, and suggest appropriate menus and shopping lists using external flyer information. However, current technology does not offer a system that integrates these functions, resulting in significant labor-intensive tasks during shopping and actual cooking. In particular, product recognition and inventory updates during shopping are performed manually, resulting in inefficiencies and limitations in real-time inventory updates and menu suggestions.

[0716] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0717] In this invention, the server includes means for taking pictures of receipts and products after shopping and analyzing product information from the captured images, means for automatically updating inventory in the refrigerator based on the analyzed product information, means for receiving information on ingredients used in cooking when the user inputs the ingredients by voice or image and updating the inventory in the refrigerator, means for obtaining flyer information from an external website every day, analyzing the information, and registering the product information in a database, means for receiving voice input from the user discussing menus and generating and suggesting appropriate menus and shopping lists based on the current inventory and the obtained flyer information, means for scanning product barcodes when shopping, means for automatically adding product information obtained from the scanned barcodes to the database, and means for creating and updating shopping lists based on the user's voice input. This allows the user to manage refrigerator inventory in real time while shopping or cooking and receive suggestions for appropriate menus and shopping lists.

[0718] A "receipt" is a paper medium that lists information such as the purchased item, the amount, and the date and time of purchase.

[0719] "Product Photos" refers to images of the products purchased by a User.

[0720] "Analysis" refers to extracting necessary data from captured images and input information.

[0721] "Inventory" refers to the food and products stored in the refrigerator.

[0722] "Voice input" is a method in which a user inputs information by voice using a microphone device.

[0723] "Image input" is a method in which a user inputs information by means of an image using a camera device.

[0724] "Flyer information" refers to advertising materials including sale information and product information issued by supermarkets and the like.

[0725] A "barcode" is a series of lines or graphics printed on a product to allow computer-readable information about the product.

[0726] A "database" is a collection of information that stores managed data and allows for rapid searching and updating.

[0727] "Menu" means a list of dishes or meals from which a User can choose.

[0728] A "shopping list" refers to a list of products or ingredients that a user plans to purchase.

[0729] "Scanning" refers to reading information from a barcode or the like.

[0730] "Suggestion" refers to presenting appropriate options or actions to the user.

[0731] "User" refers to a consumer who uses the system to shop and cook.

[0732] A "system" refers to a structure in which a series of devices and programs function in conjunction with one another.

[0733] This invention is a system that manages refrigerator inventory by analyzing receipts and product photos after users have finished shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information obtained daily.The system uses hardware and software such as a cloud server, smartphones, smart glasses, and OCR (optical character recognition) technology.

[0734] The main process is as follows:

[0735] Post-shopping inventory management:

[0736] 1. The user takes a photo of the receipt or product with their smartphone and uploads it to the cloud server. This process uses the smartphone's camera and internet connection functions.

[0737] 2. The cloud server analyzes the uploaded image using OCR technology to extract information such as product name and quantity. This OCR technology is used to quickly and accurately digitize product information.

[0738] 3. The extracted product information is recorded in an inventory database on the cloud server, and the refrigerator's inventory is automatically updated.

[0739] Cooking Inventory Update:

[0740] 1. When cooking, the user inputs voice or images. For example, the user inputs "I used milk" into the smartphone.

[0741] 2. The smartphone converts the voice input into text and sends it to a cloud server. The speech recognition module handles this process.

[0742] 3. The cloud server updates the inventory database based on the received information, decrements the quantity of ingredients used, and notifies the user if the ingredients are out of stock.

[0743] Daily flyer updates:

[0744] 1. The cloud server retrieves flyer information as images from supermarket websites at a specified time every day. A web scraper performs this process automatically.

[0745] 2. The server uses OCR technology to analyze the flyer image, extract product information, and obtain data such as product name, price, and validity period.

[0746] 3. The extracted product information is registered in the flyer information database and becomes accessible to users.

[0747] Menu suggestions and shopping list generation:

[0748] 1. A user speaks, "What would you like for dinner tonight?", for example, through smart glasses.

[0749] 2. The smart device converts the voice into text and sends the information to a cloud server.

[0750] 3. The server compares the current inventory database and flyer information database to generate the appropriate menu and shopping list. A generative AI model performs this analysis and recommendation.

[0751] 4. The smart device will give the user voice suggestions, for example, "To make a chicken and broccoli stir-fry, you need to buy broccoli from the supermarket."

[0752] Inventory management when shopping:

[0753] 1. When a user shops in a physical store, they scan the product's barcode with their smartphone. The barcode scanner app then sends the product information to a cloud server.

[0754] 2. The cloud server parses the product information from the scanned barcode and adds it to the inventory database.

[0755] Example of a concrete example and prompt for the generative AI model:

[0756] Examples:

[0757] When a user purchases a lettuce at a physical store and scans the barcode with their smartphone, the cloud server updates the inventory database and records "1 lettuce added."

[0758] When a user asks the smart glasses, "What's for dinner tonight?", the cloud server will suggest "hamburger steak and salad" based on the refrigerator's inventory and store special offers.

[0759] Example prompt sentence:

[0760] When a user says, "What ingredients do I need for a hot pot?", the following answers are generated based on the current refrigerator inventory and special offers:

[0761] The recipe is "Chicken Hot Pot"

[0762] refrigerator inventory

[0763] Special sale information

[0764] This allows users to enjoy an efficient and profitable shopping and cooking experience.

[0765] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0766] Step 1:

[0767] The user takes a photo of a receipt or product with their smartphone. The input is the image of the receipt or product. The output is an image file stored in the smartphone's storage. The device prepares the captured image to be uploaded to the cloud server.

[0768] Step 2:

[0769] The device uploads the images it takes to the cloud server. The input is the image file stored on the smartphone. The output is the image data transferred to the cloud server. The device transmits the data via an internet connection.

[0770] Step 3:

[0771] The server analyzes the received image using OCR technology. The input is the image data transferred to the server, and the output is the extracted product information (product name, quantity, price, etc.). The server uses OCR technology to extract text data from the image.

[0772] Step 4:

[0773] The server records and updates the analyzed product information in the refrigerator's inventory database. The input is the product information extracted using OCR technology. The output is the updated inventory database. The server compares the product information with existing inventory data and updates the database.

[0774] Step 5:

[0775] When cooking, the user speaks the ingredients they will use into their smartphone. The input is the user's voice data. The output is the voice information converted into text. The device then uses a voice recognition module to convert the voice data into text.

[0776] Step 6:

[0777] The device converts voice input into text and sends the information to the server. The input is text data. The output is text information transferred to the cloud server. The device transmits the data via the Internet.

[0778] Step 7:

[0779] The server updates the inventory database based on the received information. The input is text information obtained by speech recognition. The output is the updated inventory database. The server processes the text information to reduce the amount of ingredients used.

[0780] Step 8:

[0781] The server uses a web scraper to retrieve supermarket flyer information at a specified time every day. The input is the website URL. The output is the retrieved flyer image. The server automatically retrieves flyer information using the web scraper.

[0782] Step 9:

[0783] The server uses OCR technology to analyze the acquired flyer image and extract product information. The input is the flyer image. The output is the analyzed product information (product name, price, validity period, etc.). The server uses OCR technology to extract text data from the flyer image.

[0784] Step 10:

[0785] The server registers the extracted product information in a flyer information database. The input is the product information extracted using OCR technology. The output is an updated flyer information database. The server stores the product information in the database.

[0786] Step 11:

[0787] The user speaks to the smart glasses, saying, "What would you like for dinner tonight?" The input is the user's voice data. The output is the voice information converted into text. The device uses a voice recognition module to convert the voice data into text.

[0788] Step 12:

[0789] The device converts voice input into text and sends the information to the server. The input is text data. The output is text information sent to the cloud server. The device sends the data over the Internet.

[0790] Step 13:

[0791] The server compares the current inventory database and flyer information database and generates the appropriate menu and shopping list. The input is the inventory database and flyer information database. The output is the generated menu and shopping list. The server generates the menu and list using a generative AI model.

[0792] Step 14:

[0793] The device makes audio suggestions to the user. The input is the generated menu and shopping list. The output is audio suggestions. The device uses a speaker to provide information to the user.

[0794] Step 15:

[0795] A user scans a product barcode with their smartphone in a physical store. The input is the product barcode data. The output is the barcode information stored on the smartphone. The device uses the camera function and a barcode scanner app.

[0796] Step 16:

[0797] The terminal sends the product information obtained by barcode scanning to the cloud server. The input is the barcode information. The output is the product information sent to the cloud server. The terminal sends the data via the Internet.

[0798] Step 17:

[0799] The server adds and updates the received product information to the inventory database. The input is the product information obtained by barcode scanning. The output is the updated inventory database. The server stores the product information in the database.

[0800] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0801] This system manages refrigerator inventory by analyzing receipts and product photos taken by the user after shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to the user based on supermarket flyer information acquired daily.The system also combines an emotion engine that recognizes the user's emotions using voice input and image analysis technology and adjusts the suggestions based on those emotions.

[0802] Program processing overview

[0803] Inventory management using shopping photos and receipts

[0804] 1. The user takes a photo of the receipt or product with their smartphone.

[0805] 2. Upload the photos taken by the device (smartphone) to the cloud server.

[0806] 3. The server analyzes the received image using optical character recognition (OCR) technology and extracts product information (product name, quantity, price, etc.).

[0807] 4. The server records and updates the analyzed product information in the refrigerator's inventory database.

[0808] Control delivery by voice or image input during cooking

[0809] 1. When the user prepares a dish, they tell the system the ingredients they will use using voice or images.

[0810] 2. The device converts the voice input into text and sends the information to the server. If the input is an image, the image is sent to the server.

[0811] 3. The server updates the refrigerator's inventory database based on the received information, decrementing the quantity of ingredients used and notifying the user if the ingredient is out of stock.

[0812] Daily flyer information acquisition and analysis

[0813] 1. The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[0814] 2. The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[0815] 3. The extracted product information is registered in the flyer information database.

[0816] Recognizing user emotions with an emotion engine

[0817] 1. Recognize emotions through voice input or camera when users consult the menu.

[0818] 2. The device converts the voice input into text and sends the information to the server. At the same time, the emotion engine analyzes the user's emotions from their tone of voice and facial expressions.

[0819] 3. The server adjusts the menu suggestions and shopping list contents based on the emotional information received.

[0820] Voice and emotion menu consultation and suggestions

[0821] 1. The user asks the system verbally, "What would you like for dinner tonight?"

[0822] 2. The device converts the voice input into text and sends the information to the server. At the same time, the emotion engine recognizes the user's emotions.

[0823] 3. The server compares the current refrigerator inventory database, flyer information database, and emotion information to generate an appropriate menu.

[0824] 4. The server creates a shopping list for ingredients that are in short supply and adjusts its suggestions based on feedback from the emotion engine.

[0825] 5. The device will then output a suggestion to the user, such as, "You seem a little tired today. How about a quick chicken and broccoli stir-fry?"

[0826] Specific examples

[0827] Specific examples of inventory management

[0828] A user purchases milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which uses OCR technology to extract the product information (1 liter of milk) and updates the refrigerator's inventory database with the entry "Add 1 liter of milk."

[0829] Specific examples of inventory management

[0830] While cooking, the user says "I used milk" by voice input. The device converts the voice to text and sends it to the server as "I used 1 liter of milk." The server updates the inventory database and records it as "I reduced 1 liter of milk."

[0831] Examples of flyer information

[0832] At 9:00 AM, the server retrieves the flyer image from the supermarket's website, analyzes the product information "Broccoli 98 yen," and registers it in the flyer information database. It saves it as "Broccoli 98 yen, sale period: 3 days."

[0833] Examples of emotion engines

[0834] The user asks, "What should we have for dinner tonight?" The emotion engine determines from the user's tone of voice and facial expression that they seem tired. The device converts this speech into text and sends it to the server. Based on the emotion engine's feedback, the server compares the refrigerator's inventory with flyer information and suggests, "You seem a little tired today, how about a simple chicken and broccoli stir-fry?"

[0835] This system provides a multifunctional mechanism that includes these steps and can support users in daily food management, menu planning, and shopping planning, including emotional information.

[0836] The processing flow will be explained below.

[0837] Manage refrigerator inventory using shopping photos and receipts

[0838] Step 1:

[0839] The user takes a photo of the receipt or product with their smartphone.

[0840] Open the camera app on your smartphone and take a photo of your receipt or purchased items.

[0841] Step 2:

[0842] The device uploads the photos it takes to the server.

[0843] Use the dedicated application and press the button to upload the photo to the cloud server.

[0844] Step 3:

[0845] The server applies OCR technology to analyze the received image.

[0846] The server invokes an image processing module to extract text information from the image.

[0847] Step 4:

[0848] The server extracts the parsed product information (product name, quantity, price, etc.).

[0849] Product name, quantity, and price are identified from the text information and converted into a database format.

[0850] Step 5:

[0851] The server records the extracted product information in the refrigerator's inventory database and updates it.

[0852] Adds the product information to the corresponding user's inventory database and updates existing inventory information.

[0853] Control delivery by voice or image input during cooking

[0854] Step 1:

[0855] When cooking, the user verbally tells the system what ingredients to use.

[0856] Use your smartphone's microphone to input voice instructions such as "I used chicken."

[0857] Step 2:

[0858] The device converts the voice input into text and sends the information to the server.

[0859] Uses voice recognition software to convert speech into text.

[0860] Step 3:

[0861] The server updates the refrigerator inventory database based on the received information.

[0862] The quantity of the specified ingredient is reduced and the inventory database is updated.

[0863] Step 4:

[0864] If the server is out of stock, the user is notified.

[0865] If the stock reaches 0, a notification message will be sent to the user.

[0866] Daily flyer information acquisition and analysis

[0867] Step 1:

[0868] The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[0869] Based on a fixed schedule, it automatically visits each supermarket's website and downloads flyer images.

[0870] Step 2:

[0871] The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[0872] Use the flyer image processing module to extract product information from the image.

[0873] Step 3:

[0874] The extracted product information is registered in a flyer information database.

[0875] The flyer information is stored in a database based on the product name, price, and validity period.

[0876] Recognizing user emotions with an emotion engine

[0877] Step 1:

[0878] When the user consults the menu, he or she uses voice input.

[0879] Using the smartphone's microphone, voice menu consultation is performed, asking, "What would you like for dinner tonight?"

[0880] Step 2:

[0881] The device converts the voice input into text and sends the information to the server.

[0882] Use speech recognition software to convert speech to text.

[0883] Step 3:

[0884] The emotion engine analyzes emotions from the user's tone of voice.

[0885] A voice tone analysis module is used to identify the user's emotions.

[0886] Step 4:

[0887] The server adjusts the suggested menu and shopping list based on the emotional information received.

[0888] Taking emotion information into consideration, simple dishes and dishes that the user likes are selected.

[0889] Voice and emotion menu consultation and suggestions

[0890] Step 1:

[0891] The user verbally asks the system, "What would you like for dinner tonight?"

[0892] Voice menu consultation is performed using the smartphone's microphone.

[0893] Step 2:

[0894] The device converts the voice input into text and sends the information to the server.

[0895] Use speech recognition software to convert speech to text.

[0896] Step 3:

[0897] The server compares the current refrigerator inventory database, flyer information database, and emotion information.

[0898] Cross-check the database to see what ingredients are available and what discounts are available.

[0899] Step 4:

[0900] Generate appropriate menus and shopping lists for missing ingredients.

[0901] Generate suggested recipes and add missing ingredients to your shopping list.

[0902] Step 5:

[0903] The terminal outputs the suggestions to the user by voice.

[0904] Using the smartphone's speaker, the system makes a voice suggestion such as, "You seem a little tired today, how about an easy-to-make chicken and broccoli stir-fry?"

[0905] Through the above processing flow, this system can efficiently support users in managing ingredients, planning menus, and making shopping plans, including emotional information.

[0906] Example 2

[0907] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0908] In today's busy lifestyles, food inventory management has become a complicated task for many households. Deciding on a menu and creating a shopping list for the necessary ingredients is also a time-consuming task. Furthermore, there is a demand for appropriate menu suggestions based on the user's emotions and physical condition, but current technology has difficulty meeting this demand.

[0909] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for taking pictures of receipts and products after shopping and analyzing product information from the captured images; means for automatically updating inventory in the refrigerator based on the analyzed product information; means for receiving information on ingredients used in cooking when the user inputs the ingredients by voice or image and updating the inventory in the refrigerator; means for acquiring flyer information from an external website every day, analyzing the information, and registering the product information in a database; means for receiving voice input from the user discussing menus, and generating and suggesting appropriate menus and shopping lists based on the current inventory and the acquired flyer information; means for recognizing emotions from the user's voice tone and facial expressions; and means for adjusting the suggestions based on the emotion recognition information. This enables efficient and accurate daily ingredient management, menu planning, and shopping planning, including emotional information.

[0910] "Receipts and product photos after shopping" refers to receipts showing purchase details that users receive when they purchase products at a store, and image data of purchased products.

[0911] "Product Information" means detailed data about the purchased item, such as product name, quantity, and price.

[0912] "Optical character recognition (OCR) technology" refers to the technology that scans images and handwritten characters and converts them into text data.

[0913] "Inventory database" refers to a database system for recording and managing inventory status and product information in a refrigerator.

[0914] "Voice input" means a means by which a user communicates information to a system using speech.

[0915] "Image input" means the means by which a user communicates information to a system using an image.

[0916] "Web scraping" refers to the technique of extracting data from websites using automated programs.

[0917] "Flyer information" refers to data including special sale items and price information provided by stores for promotional purposes.

[0918] "Emotion engine" refers to algorithms and technologies that analyze a user's tone of voice and facial expressions to recognize their emotional state.

[0919] "Suggestions" refers to information such as menus and shopping lists provided to users.

[0920] "Refrigerator inventory" refers to the ingredients and products stored in the user's refrigerator.

[0921] "Voice input for menu consultation" refers to voice input by the user to ask the system about the dish they would like to make.

[0922] This system analyzes receipts and product photos taken by users after shopping, manages refrigerator inventory, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information acquired daily.The system also combines an emotion engine that recognizes the user's emotions using voice input and image analysis technology and adjusts the suggestions based on those emotions.

[0923] Inventory management using shopping photos and receipts

[0924] Users take photos of receipts and products with their smartphones. The device uploads these images to a cloud server. The server then analyzes the images using optical character recognition (OCR) technology. Specifically, OCR software such as Tesseract OCR is used to extract product information such as product name, quantity, and price. The server then records this information in the refrigerator's inventory database and automatically updates it.

[0925] Examples:

[0926] A user buys milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which uses Tesseract OCR to extract the product information (1 liter of milk) and updates the refrigerator inventory with the entry "Add 1 liter of milk."

[0927] Control delivery by voice or image input during cooking

[0928] When a user prepares a dish, they tell the system which ingredients they will use using voice or images. The device converts the voice input into text and sends that information to the server. If an image is input, the image is sent to the server. The server updates the refrigerator's inventory database based on the received information and reduces the quantity of ingredients used. If the inventory falls below a certain level, the user is notified.

[0929] Examples:

[0930] While cooking, the user voice-inputs "Milk used." The device converts the voice data into text and sends it to the server as "1 liter of milk used." The server updates the inventory database and records "1 liter of milk decreased."

[0931] Daily flyer information acquisition and analysis

[0932] The server retrieves flyer information as image data from each supermarket's website at a specified time every day. Libraries such as Beautiful Soup and Selenium are used for web scraping. The server analyzes the flyer images retrieved using OCR technology and extracts product information (product name, price, validity period, etc.). This information is registered in the flyer information database.

[0933] Examples:

[0934] The server retrieves the flyer image from the supermarket's website at 9:00 AM, analyzes the product information "Broccoli 98 yen" using Beautiful Soup, and saves it in the flyer information database as "Broccoli 98 yen, sale period: 3 days."

[0935] Recognizing user emotions with an emotion engine

[0936] When a user consults about a menu, emotions are recognized through voice input or a camera. The device converts the voice input into text and sends the information to the server. At the same time, the device is equipped with an emotion analysis engine such as OpenCV or Face API, which analyzes emotions from the user's tone of voice and facial expressions. The server then adjusts the menu suggestions and shopping list contents based on this emotional information.

[0937] Examples:

[0938] The user asks, "What should we have for dinner tonight?" The device converts this speech into text, and at the same time, the emotion engine detects tired facial expressions and analyzes them. Based on the feedback from the emotion engine, the server adjusts the suggestions, such as "You seem a little tired today, so I'll recommend some easy-to-make dishes."

[0939] Voice and emotion menu consultation and suggestions

[0940] The user verbally asks the system, "What would you like for dinner tonight?" The device converts the voice input into text and sends the information to the server. At the same time, the emotion engine recognizes the user's emotions. The server compares the current refrigerator inventory database, flyer information database, and emotion information to generate an appropriate menu. The server creates a shopping list for missing ingredients and further adjusts this list based on feedback from the emotion engine. The device then outputs the suggestions to the user verbally.

[0941] Examples:

[0942] The user asks the system via voice, "What would you like for dinner tonight?" The device converts the text and sends the emotion data to the server, which then makes a suggestion to the user via voice, such as, "You seem a little tired today, how about a quick chicken and broccoli stir-fry?"

[0943] Example prompt sentence:

[0944] "Take a photo of your supermarket receipt and upload it"

[0945] "List the ingredients used while cooking by voice"

[0946] "I'd like some suggestions for dinner tonight."

[0947] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0948] Inventory management using shopping photos and receipts

[0949] Step 1:

[0950] After shopping, the user takes a photo of the receipt or product with their smartphone. The user launches the camera app and takes a photo, taking care to ensure that the receipt and product are clearly visible. This is the initial step in obtaining input data (receipt photo).

[0951] Step 2:

[0952] The device uploads the captured image to the cloud server. The device uses the HTTP protocol via an internet connection to send the image data to the server. The input is a photo of the receipt, and the output is the transmission of image data to the server.

[0953] Step 3:

[0954] The image received by the server is analyzed using OCR technology. Software such as Tesseract OCR is used to recognize characters in the image and extract text information such as product name, quantity, and price. The input is image data, and the output is analyzed product information.

[0955] Step 4:

[0956] The server records the extracted product information in the refrigerator's inventory database and updates it. The database update process keeps the inventory information up to date. The input is the analyzed product information, and the output is the updated inventory database.

[0957] Control delivery by voice or image input during cooking

[0958] Step 1:

[0959] When a user prepares a dish, they tell the system what ingredients they will use by voice or image. For example, they can say "I used milk." This is input data for recording the ingredients used.

[0960] Step 2:

[0961] The device converts the user's voice input into text data. It uses Google Cloud Speech-to-Text, a speech recognition technology, to convert speech to text. The input is voice data, and the output is text data.

[0962] Step 3:

[0963] The terminal sends the converted text data to the server via an Internet connection using the HTTP protocol. The input is text data, and the output is data sent to the server.

[0964] Step 4:

[0965] The server updates the refrigerator's inventory database based on the received data. It reduces the quantity of ingredients used and notifies the user when the stock falls below a certain level. The input is text data, and the output is the updated inventory database and a notification.

[0966] Daily flyer information acquisition and analysis

[0967] Step 1:

[0968] The server retrieves flyer images from each supermarket's website at a specified time every day using a web scraping tool (e.g., Beautiful Soup or Selenium). The input is the supermarket's website URL, and the output is the flyer image.

[0969] Step 2:

[0970] The server analyzes the flyer image acquired by the server using OCR technology and extracts product information. The input is the flyer image and the output is the analyzed product information.

[0971] Step 3:

[0972] The server registers the analyzed product information in a flyer information database. The database is updated and sale information is stored. The input is the analyzed product information, and the output is the updated flyer information database.

[0973] Recognizing user emotions with an emotion engine

[0974] Step 1:

[0975] Emotion recognition is performed through voice input or camera when the user is consulting a menu. For example, the user faces the camera and says, "What should I have for dinner tonight?" This is the initial step in obtaining input data (voice and image).

[0976] Step 2:

[0977] The device converts the user's voice input into text and sends that information to the server. At the same time, the emotion engine analyzes emotions from voice tone and facial expressions. OpenCV and Face API are used. The input is voice and image data, and the output is text data and emotion data.

[0978] Step 3:

[0979] The server adjusts menu suggestions based on emotional information. It compares the inventory database and flyer information to generate the optimal menu for the user. The input is emotional data and inventory / flyer information, and the output is the adjusted menu suggestions.

[0980] Voice and emotion menu consultation and suggestions

[0981] Step 1:

[0982] The user verbally asks the system what dish to make, for example, "What would you like for dinner tonight?" This is the input data for the suggestion.

[0983] Step 2:

[0984] The device converts voice input into text and sends it to the server. At the same time, the emotion engine recognizes the user's emotions. The input is voice data, and the output is text data and emotion data.

[0985] Step 3:

[0986] The server generates an appropriate menu based on the current refrigerator inventory database, flyer information database, and emotion information. The input is inventory and flyer information and emotion data, and the output is the generated menu.

[0987] Step 4:

[0988] The server creates a shopping list of missing ingredients and further refines the suggestions. The input is the generated menu, and the output is the shopping list and refined suggestions.

[0989] Step 5:

[0990] The device outputs a suggestion to the user by voice, for example, "You seem a little tired today, how about a quick chicken and broccoli stir-fry?" The input is the tailored suggestion, and the output is the voice prompt.

[0991] (Application example 2)

[0992] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0993] Conventional refrigerator inventory management systems have the problem that updating inventory is cumbersome and difficult for users to use. They also lack a way for users to easily grasp the ingredients and menu items they need, and are unable to make suggestions based on their emotions. Furthermore, while real-time inventory updates and emotion-based suggestions are required, conventional systems have been unable to comprehensively address these issues.

[0994] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0995] In this invention, the server includes means for taking pictures of receipts and items after shopping and analyzing item information from the captured images, means for automatically updating inventory in the refrigerator based on the analyzed item information, means for receiving information on ingredients used in cooking when the user inputs the information by voice or image and updating the inventory in the refrigerator, means for acquiring flyer information from an external website every day, analyzing the information, and registering the item information in a database, means for receiving voice input from the user discussing menus, and generating and suggesting appropriate menus and shopping lists based on the current inventory and the acquired flyer information, means for analyzing user emotional information, and means for adjusting the suggestions based on the analyzed emotional information. This makes it possible to efficiently manage inventory in the refrigerator and generate menu suggestions and shopping lists that reflect the user's emotions.

[0996] A "post-purchase receipt" is a paper statement issued after the purchase of an item, which contains information including the name, quantity, and price of the purchased item.

[0997] An "item photo" is an image of a purchased item taken by a user after shopping, and includes the appearance and label information of the item.

[0998] "Item information" refers to detailed information such as the name, quantity, and price of an item, and is data extracted by image analysis or character recognition.

[0999] "Inventory in refrigerator" is a database that shows the total amount of food and beverages stored in a refrigerator or other refrigeration device.

[1000] "Voice input" is a means by which the user verbally communicates to the system the materials to be used, the details of the proposal, etc., which is converted into text data using voice recognition technology.

[1001] "Image input" is a means by which a user sends an image of the material or item they are using to the system, and information is extracted using image analysis techniques.

[1002] "External websites" are web pages of commercial facilities or information providers that exist on the Internet, and on which flyer information and the like are made public.

[1003] "Flyer information" is image data or text data that includes special sale information and product price lists offered by commercial facilities.

[1004] A "database" is an electronic information aggregation system for systematically managing and storing analyzed product information, inventory information, and flyer information.

[1005] "User's emotional information" is data that indicates the emotional state of the user, which is analyzed from the user's tone of voice, facial expression, content of words, and the like.

[1006] "Suggestion content" refers to information such as menus and shopping lists that the system provides to users, and is generated based on inventory data, emotional information, and the like.

[1007] This invention begins when a user takes a photo of a receipt or item after shopping with their smartphone and uploads it to a cloud server. The server analyzes the image using optical character recognition (OCR) technology to extract item information (item name, quantity, price, etc.). This item information is recorded in an inventory database inside the refrigerator and automatically updated. When the user prepares a meal, they tell the system which ingredients they will be using via voice or image input. The device converts the voice to text and sends that information to the server. The server updates the inventory database based on the received information and reduces the quantity of ingredients used. In addition, flyer information is obtained daily from an external website, analyzed as an image, and item information (item name, price, expiration date, etc.) is extracted. This information is registered in the flyer information database.

[1008] The system's main hardware includes a smartphone (camera, microphone) and a cloud server. The main software used includes optical character recognition (OCR), voice input analysis, and image analysis technologies, which utilize cloud services such as Google Cloud Vision API and Google Cloud Speech-to-Text. When a user asks the system, "What would you like for dinner tonight?", the voice is recorded by the smartphone's microphone. The device converts the voice into text and sends it to the cloud server. At the same time, an emotion engine analyzes the user's emotions from their tone of voice and facial expressions. The server compares the current refrigerator inventory database, flyer information database, and emotion information to generate an appropriate menu. It creates a shopping list for any missing ingredients and adjusts the suggestions based on feedback from the emotion engine.

[1009] For example, a user purchases milk at a supermarket and takes a photo of the receipt with their smartphone. When this image is uploaded to the cloud server, OCR technology is used to extract the product information—"1 liter of milk." This information is recorded in the refrigerator's inventory database and updated as "1 liter of milk added." Furthermore, if the user voice-inputs "I used milk" while cooking, this voice is converted to text and sent to the cloud server. The server updates the inventory database based on this information, recording it as "1 liter of milk reduced." Furthermore, the server retrieves flyer images from each supermarket's website at a specified time each morning, analyzes the product information, such as "Broccoli 98 yen," and registers it in the flyer information database. This information is saved as "Broccoli 98 yen, special sale period: 3 days." The emotion engine determines that the user is tired based on their tone of voice and facial expression when they ask, "What should I have for dinner tonight?" The server then compares the refrigerator's inventory with the flyer information based on the emotion engine's feedback and suggests, "You seem tired today. How about a simple chicken and broccoli stir-fry?"

[1010] An example of a prompt for a generative AI model might be, "Please tell me how to analyze receipts and item photos taken by the user and update that information in the refrigerator inventory database."

[1011] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1012] Step 1:

[1013] After shopping, the user takes a photo of the receipt or item with their smartphone. The input is the captured image, and the output is an image file saved in the smartphone's storage. Specifically, the user opens the smartphone's camera app, takes a photo of the receipt or item, and saves it as an image file.

[1014] Step 2:

[1015] The device uploads the captured image to the cloud server. The input is the image file stored on the smartphone, and the output is the image data uploaded to the cloud server. Specifically, the device establishes communication with the cloud server and sends the image file captured by the user to the server.

[1016] Step 3:

[1017] The server analyzes the image using optical character recognition (OCR) technology and extracts product information. The input is the image data uploaded to the cloud server, and the output is text data such as the product name, quantity, and price. Specifically, the server uses OCR technology (e.g., Tesseract) to analyze the text portion of the image and extracts product information as text data.

[1018] Step 4:

[1019] The server records the analyzed item information in the inventory database inside the cooler and updates it. The input is text data extracted using OCR technology, and the output is updated inventory data. Specifically, the server accesses the inventory database and updates the inventory data based on the extracted item information.

[1020] Step 5:

[1021] The user communicates the ingredients used in cooking to the system through voice or image input. The input is the user's voice instruction or image data, and the output is information sent by the device to the server. Specifically, the user communicates the ingredients used to the system using the microphone or camera on their smartphone.

[1022] Step 6:

[1023] The device converts voice input into text and sends the information to the server. The input is the user's voice data, and the output is text data. Specifically, the device converts the voice into text using voice recognition technology (e.g., Google Cloud Speech-to-Text) and sends it to the server.

[1024] Step 7:

[1025] The server updates the inventory database in the refrigerator based on the received information. The input is text data or image data, and the output is updated inventory data. Specifically, the server accesses the inventory database and decreases the quantity of materials used.

[1026] Step 8:

[1027] The server retrieves flyer information from an external website every day and analyzes it as an image. The input is the website URL, and the output is flyer image data. Specifically, the server accesses the website at a specified time and downloads the flyer image.

[1028] Step 9:

[1029] The server analyzes the flyer image and registers the product information (product name, price, validity period, etc.) in a database. The input is the image data of the flyer, and the output is the product information registered in the database. Specifically, the server analyzes the flyer image using OCR technology, extracts the product information, and stores it in the database.

[1030] Step 10:

[1031] The system receives voice input from the user to discuss a menu, and the server generates an appropriate menu and shopping list based on current inventory and flyer information. The input is the voice instruction and inventory and flyer information, and the output is a suggested menu and shopping list. Specifically, the server analyzes the voice instruction and compares the inventory data with the flyer information to generate an appropriate menu.

[1032] Step 11:

[1033] The server analyzes the user's emotional information and adjusts the suggestions. The input is the user's voice tone and facial expression data, and the output is the adjusted suggestions. Specifically, the server analyzes the user's emotions using emotion recognition technology (e.g., Google Cloud Vision API) and adjusts the suggestions based on the emotions.

[1034] Step 12:

[1035] The device outputs the adjusted proposal content to the user by voice. The input is the adjusted proposal content, and the output is a voice notification to the user. Specifically, the device converts text data into voice and notifies the user of the proposal content.

[1036] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1037] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1038] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1039] [Third embodiment]

[1040] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1041] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1042] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1043] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1044] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1045] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1046] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1047] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1048] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1049] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1050] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1051] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1052] The present invention is a system that manages refrigerator inventory by analyzing receipts and product photos taken by users after shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information obtained daily.

[1053] Program processing overview

[1054] Inventory management using receipts and product photos after shopping

[1055] 1. The user takes a photo of the receipt or product with their smartphone.

[1056] 2. Upload the photos taken by the device (smartphone) to the cloud server.

[1057] 3. The server analyzes the received image using optical character recognition (OCR) technology and extracts product information (product name, quantity, price, etc.).

[1058] 4. The server records and updates the analyzed product information in the refrigerator's inventory database.

[1059] Control delivery by voice or image input during cooking

[1060] 1. When the user prepares a dish, they tell the system the ingredients they will use using voice or images.

[1061] 2. The device converts the voice input into text and sends the information to the server. If the input is an image, the image is sent to the server.

[1062] 3. The server updates the refrigerator's inventory database based on the received information, decrementing the quantity of ingredients used and notifying the user if the ingredient is out of stock.

[1063] Daily flyer information acquisition and analysis

[1064] 1. The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[1065] 2. The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[1066] 3. The extracted product information is registered in the flyer information database.

[1067] Voice menu consultation and suggestions

[1068] 1. The user asks the system verbally, "What would you like for dinner tonight?"

[1069] 2. The device converts the voice input into text and sends the information to the server.

[1070] 3. Based on the information received by the server, the current refrigerator inventory database is compared with the flyer information database to generate an appropriate menu and a shopping list of missing ingredients.

[1071] 4. The device will output a suggestion to the user via voice, such as "To make a chicken and broccoli stir-fry, you will need to buy broccoli from the supermarket."

[1072] Specific examples

[1073] Specific examples of inventory management

[1074] A user purchases milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which uses OCR technology to extract the product information (1 liter of milk) and updates the refrigerator's inventory database with the entry "Add 1 liter of milk."

[1075] Specific examples of inventory management

[1076] While cooking, the user says "I used milk" by voice input. The device converts the voice to text and sends it to the server as "I used 1 liter of milk." The server updates the inventory database and records it as "I reduced 1 liter of milk."

[1077] Examples of flyer information

[1078] At 9:00 AM, the server retrieves the flyer image from the supermarket's website, analyzes the product information "Broccoli 98 yen," and registers it in the flyer information database. It saves it as "Broccoli 98 yen, sale period: 3 days."

[1079] Specific examples of menu consultation

[1080] The user asks, "What should I have for dinner tonight?" The device converts this speech into text and sends it to the server. The server compares the current inventory database with the flyer information database, determines that "there is chicken in the fridge and broccoli on sale," and suggests "stir-fried chicken and broccoli."

[1081] This system provides a multifunctional mechanism that includes these steps, and centrally and automatically supports users in daily food management, menu planning, and shopping planning.

[1082] The processing flow will be explained below.

[1083] Manage refrigerator inventory using shopping photos and receipts

[1084] Step 1:

[1085] The user takes a photo of the receipt or product with their smartphone.

[1086] Open the camera app on your smartphone and take a photo of your receipt or purchased items.

[1087] Step 2:

[1088] The device uploads the photos it takes to the server.

[1089] Use the dedicated application and press the button to upload the photo to the cloud server.

[1090] Step 3:

[1091] The server applies OCR technology to analyze the received image.

[1092] The server invokes an image processing module to extract text information from the image.

[1093] Step 4:

[1094] The server extracts the parsed product information (product name, quantity, price, etc.).

[1095] Product name, quantity, and price are identified from the text information and converted into a database format.

[1096] Step 5:

[1097] The server records the extracted product information in the refrigerator's inventory database and updates it.

[1098] Adds the product information to the corresponding user's inventory database and updates existing inventory information.

[1099] Control delivery by voice or image input during cooking

[1100] Step 1:

[1101] When cooking, the user verbally tells the system what ingredients to use.

[1102] Use your smartphone's microphone to input voice instructions such as "I used chicken."

[1103] Step 2:

[1104] The device converts the voice input into text and sends the information to the server.

[1105] Uses voice recognition software to convert speech into text.

[1106] Step 3:

[1107] The server updates the refrigerator inventory database based on the received information.

[1108] The quantity of the specified ingredient is reduced and the inventory database is updated.

[1109] Step 4:

[1110] If the server is out of stock, the user is notified.

[1111] If the stock reaches 0, a notification message will be sent to the user.

[1112] Daily flyer information acquisition and analysis

[1113] Step 1:

[1114] The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[1115] Based on a fixed schedule, it automatically visits each supermarket's website and downloads flyer images.

[1116] Step 2:

[1117] The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[1118] Use the flyer image processing module to extract product information from the image.

[1119] Step 3:

[1120] The extracted product information is registered in a flyer information database.

[1121] The flyer information is stored in a database based on the product name, price, and validity period.

[1122] Voice menu consultation and suggestions

[1123] Step 1:

[1124] The user verbally asks the system, "What would you like for dinner tonight?"

[1125] Voice menu consultation is performed using the smartphone's microphone.

[1126] Step 2:

[1127] The device converts the voice input into text and sends the information to the server.

[1128] Use speech recognition software to convert speech to text.

[1129] Step 3:

[1130] Based on the information received by the server, the current refrigerator inventory database is compared with the flyer information database.

[1131] Cross-check the database to see what ingredients are available and what discounts are available.

[1132] Step 4:

[1133] Generate appropriate menus and shopping lists for missing ingredients.

[1134] Generate suggested recipes and add missing ingredients to your shopping list.

[1135] Step 5:

[1136] The terminal outputs the suggestions to the user by voice.

[1137] Using the smartphone's speaker, a voice suggestion is made: "To make chicken and broccoli stir-fry, you need to buy broccoli at the supermarket."

[1138] Through the above processing flow, this system efficiently supports the user in managing ingredients, planning menus, and making shopping plans.

[1139] Example 1

[1140] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1141] In modern life, efficient ingredient management and menu planning are important challenges for many households. However, manually managing ingredients and creating appropriate menus and shopping lists is time-consuming and labor-intensive, making it a burden for many people. In particular, manually collecting and using information from refrigerator inventory and supermarket flyers is cumbersome and inefficient. Additionally, it is difficult to update the inventory of ingredients used during cooking in real time, resulting in problems such as out-of-stock situations and food waste.

[1142] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1143] In this invention, the server includes means for taking pictures of receipts and products after a user has finished shopping and analyzing the product information from the captured images, means for automatically updating the inventory in the refrigerator based on the analyzed product information, means for receiving information about ingredients used in cooking when the user inputs the information by voice or image and updating the inventory in the refrigerator, means for acquiring flyer information from an external website every day, analyzing the information, and registering the product information in a database, means for receiving voice input from the user discussing menus and generating and suggesting appropriate menus and shopping lists based on the current inventory and the acquired flyer information, means for converting the user's voice input into text using a voice recognition system and sending the information to the server, and means for outputting suggestions by voice using a text-to-speech conversion system.This allows the user to efficiently manage their refrigerator inventory without any hassle and have appropriate menus and shopping lists automatically suggested.

[1144] A "user" is an entity that uses the system to manage ingredients, create menus, and plan shopping.

[1145] A "receipt" is a paper record of a purchase made at a store.

[1146] "Product photo" is an image of the purchased product.

[1147] "Means for analyzing product information from images" refers to a method for extracting information such as product name, quantity, and price from images using optical character recognition technology.

[1148] The "means for automatically updating inventory in the refrigerator" is a system that updates the database based on analyzed product information to reflect the new inventory status.

[1149] "Means for inputting information about ingredients used in cooking by voice or image" refers to a method in which the user communicates information about ingredients used to the system through voice or image.

[1150] A "voice recognition system" is a technology that converts voice input into text.

[1151] The "means for updating inventory" is a system that modifies the inventory database based on user input and maintains the latest inventory status.

[1152] "Means for obtaining daily flyer information from external websites" refers to a technology for downloading flyer images from websites of supermarkets and the like via the Internet.

[1153] The "means of analyzing and registering product information in a database" refers to a system that analyzes acquired flyer images using optical character recognition technology and stores the product information in a database.

[1154] The "means for receiving voice input for consulting a menu" is a method for receiving information in which a user requests menu suggestions by voice.

[1155] "Means for generating appropriate menus and shopping lists" refers to technology that creates optimal cooking menus and lists of ingredients that are in short supply for users based on current inventory information and flyer information.

[1156] "Means for converting a user's voice input into text using a voice recognition system" refers to a method that uses technology to convert voice data into text data.

[1157] The "means for outputting suggestions by voice using a text-to-voice conversion system" is a technology for converting generated text information into voice and conveying it to the user.

[1158] A "natural language processing model" is a technology that generates appropriate text and analyzes intent based on user input.

[1159] This invention is a system that manages refrigerator inventory by analyzing receipts and product photos taken by users after shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information obtained daily.

[1160] The system is implemented using the following hardware and software:

[1161] Hardware:

[1162] Smartphone (user device)

[1163] Cloud Server

[1164] Refrigerator (physical storage location)

[1165] software:

[1166] Optical character recognition technology (OCR): Google Cloud Vision API

[1167] Speech recognition system: Google Cloud Speech-to-Text, Amazon Transcribe

[1168] Text-to-speech system: Amazon Polly

[1169] Web scraping tools: Selenium, BeautifulSoup

[1170] Natural language processing model: GPT-3

[1171] Database management systems: MySQL, PostgreSQL

[1172] Voice UI Applications

[1173] After a user finishes shopping at the supermarket, they take a photo of the receipt or product with their smartphone. The device uploads the photo to a cloud server, which then analyzes the image using the Google Cloud Vision API. Using OCR technology, product information (product name, quantity, price, etc.) is extracted, and the server automatically registers and updates the analyzed information in the refrigerator's inventory database.

[1174] When cooking, the user can tell the system which ingredients they will use by voice or image input. The device converts the voice input into text using Google Cloud Speech-to-Text or Amazon Transcribe and sends the information to the server. When using image input, the device also sends the image to the server, which updates the inventory database based on the analysis results. The server reduces the quantity of ingredients used and notifies the user if they are out of stock.

[1175] The server also retrieves flyer images from each supermarket's website at a specified time every day using web scraping tools (Selenium, BeautifulSoup). The server then analyzes the flyer images using Tesseract OCR, extracts product information (product name, price, validity period, etc.), and registers it in the flyer information database.

[1176] When a user asks the system, "What would you like for dinner tonight?", the device converts the speech to text using Amazon Transcribe and sends it to the server. The server compares the current inventory database with the flyer information database and uses the GPT-3 natural language processing model to generate an appropriate menu and a shopping list of missing ingredients. The device then uses Amazon Polly to output the suggestions to the user.

[1177] Specific examples

[1178] For example, suppose a user buys milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which then uses the Google Cloud Vision API to extract the product information, such as "1 liter of milk." The information is then registered and updated in the refrigerator's inventory database as "1 liter of milk added."

[1179] If a user says "I used milk" while cooking, the device converts the speech into text using Google Cloud Speech-to-Text and sends it to the server as "I used 1 liter of milk." The server then updates the inventory database, recording "I reduced 1 liter of milk."

[1180] Every day at 9:00 a.m., the server uses Selenium to retrieve flyer images from the supermarket's website, analyzes the product information, such as "Broccoli 98 yen," using Tesseract OCR, and registers it in the flyer information database.

[1181] When a user asks, "What should I have for dinner tonight?", the device converts the speech into text using Amazon Transcribe and sends it to the server. The server compares the current inventory database with the flyer information database and uses the GPT-3 natural language processing model to suggest "chicken and broccoli stir-fry." The device then uses Amazon Polly to announce, "To make chicken and broccoli stir-fry, you need to buy broccoli at the supermarket."

[1182] Prompt Sentence Examples

[1183] Example prompt sentences when the user says "I used milk":

[1184] The user says "Milk used." Update the refrigerator inventory database. Decrease the quantity of milk by 1 liter and record the result.

[1185] Example prompt sentences when a user says "What should I have for dinner tonight?":

[1186] The user says, "What should I have for dinner tonight?" Match the current refrigerator inventory database with the flyer information database to suggest an appropriate dinner menu. If there are any ingredients missing, prepare a shopping list for them.

[1187] The above is a specific embodiment for carrying out the present invention.

[1188] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1189] Inventory management using receipts and product photos after shopping

[1190] Step 1:

[1191] After a user finishes shopping at a supermarket, they take a photo of the receipt or product with their smartphone. The user opens the smartphone's camera app and takes a photo of the receipt or product. This image becomes input data for the system.

[1192] Step 2:

[1193] The device (smartphone) takes a photo and uploads it to the cloud server. The upload program encodes the image data and sends it to the cloud server via a secure connection. The input is an image of the receipt or product, and the output is the storage of the image data on the cloud server.

[1194] Step 3:

[1195] The server analyzes the received photos using optical character recognition (OCR) technology. Specifically, it uses the Google Cloud Vision API to extract product information such as product name, quantity, and price from the image. The input is image data, and the output is analyzed product information (text data).

[1196] Step 4:

[1197] The server records and updates the parsed product information in the refrigerator's inventory database. The database management system (e.g., MySQL) adds the new product information and updates the existing inventory data. The input is the parsed product information, and the output is the updated inventory database.

[1198] Control delivery by voice or image input during cooking

[1199] Step 1:

[1200] When a user prepares a dish, they can tell the system which ingredients they will use by voice or image. Specifically, the user opens the smartphone app and either voice-inputs "I used milk" or takes a photo of the ingredients used. The input is voice data or image data.

[1201] Step 2:

[1202] The device converts voice input into text and sends that information to the server. "Google Cloud Speech-to-Text" is used as the voice recognition system. Voice is converted into text and the textual information is sent to the server. The input is voice data and the output is text data. In the case of image input, the image is sent directly to the server. The input is image data and the output is saving the image data to the server.

[1203] Step 3:

[1204] The server updates the refrigerator's inventory database based on the information it receives. Specifically, it analyzes the text data and reduces the quantity of ingredients used. If the item is out of stock, a program is activated to notify the user. The input is the text data, and the output is the updated inventory database and a notification.

[1205] Daily flyer information acquisition and analysis

[1206] Step 1:

[1207] The server retrieves flyer information as images from each supermarket's website at a specified time every day. Specifically, the web scraping tools "Selenium" and "BeautifulSoup" are used to download the flyer images. The input is the supermarket's website URL, and the output is the retrieved flyer image.

[1208] Step 2:

[1209] The server analyzes the flyer image it has acquired. Specifically, it uses "Tesseract OCR" to extract product information (product name, price, validity period, etc.). The input is the flyer image, and the output is the analyzed product information (text data).

[1210] Step 3:

[1211] The extracted product information is registered in a flyer information database. New flyer information is added and saved using a database management system (e.g., PostgreSQL). The input is the parsed product information, and the output is an updated flyer information database.

[1212] Voice menu consultation and suggestions

[1213] Step 1:

[1214] The user asks the system by voice, "What would you like for dinner tonight?" The user opens the app on their smartphone and enters voice input. The input is voice data.

[1215] Step 2:

[1216] The device converts voice input into text and sends that information to the server. Specifically, it uses "Amazon Transcribe." The voice is converted into text and the text information is sent to the server. The input is voice data and the output is text data.

[1217] Step 3:

[1218] Based on the information received by the server, the current refrigerator inventory database is compared with the flyer information database. The natural language processing model "GPT-3" is used to generate an appropriate menu and a shopping list of missing ingredients. The input is text data and database information, and the output is a menu and shopping list.

[1219] Step 4:

[1220] The device outputs the suggestions by voice. Using the text-to-speech conversion system "Amazon Polly," the generated text information is converted into voice and conveyed to the user. The input is the text data of the suggestions, and the output is a voice suggestion.

[1221] The above processing steps allow users to effortlessly manage refrigerator inventory, manage food delivery while cooking, and receive appropriate menu suggestions. Shopping lists are also automatically generated based on flyer information acquired daily.

[1222] (Application example 1)

[1223] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1224] Modern consumers have access to a wide variety of products and services, but lack efficient and effective means for daily food management and shopping planning. The present invention aims to automatically manage refrigerator inventory by analyzing receipts and product photos after a user's shopping trip, update inventory in real time based on voice or image input while cooking, and suggest appropriate menus and shopping lists using external flyer information. However, current technology does not offer a system that integrates these functions, resulting in significant labor-intensive tasks during shopping and actual cooking. In particular, product recognition and inventory updates during shopping are performed manually, resulting in inefficiencies and limitations in real-time inventory updates and menu suggestions.

[1225] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1226] In this invention, the server includes means for taking pictures of receipts and products after shopping and analyzing product information from the captured images, means for automatically updating inventory in the refrigerator based on the analyzed product information, means for receiving information on ingredients used in cooking when the user inputs the ingredients by voice or image and updating the inventory in the refrigerator, means for obtaining flyer information from an external website every day, analyzing the information, and registering the product information in a database, means for receiving voice input from the user discussing menus and generating and suggesting appropriate menus and shopping lists based on the current inventory and the obtained flyer information, means for scanning product barcodes when shopping, means for automatically adding product information obtained from the scanned barcodes to the database, and means for creating and updating shopping lists based on the user's voice input. This allows the user to manage refrigerator inventory in real time while shopping or cooking and receive suggestions for appropriate menus and shopping lists.

[1227] A "receipt" is a paper medium that lists information such as the purchased item, the amount, and the date and time of purchase.

[1228] "Product Photos" refers to images of the products purchased by a User.

[1229] "Analysis" refers to extracting necessary data from captured images and input information.

[1230] "Inventory" refers to the food and products stored in the refrigerator.

[1231] "Voice input" is a method in which a user inputs information by voice using a microphone device.

[1232] "Image input" is a method in which a user inputs information by means of an image using a camera device.

[1233] "Flyer information" refers to advertising materials including sale information and product information issued by supermarkets and the like.

[1234] A "barcode" is a series of lines or graphics printed on a product to allow computer-readable information about the product.

[1235] A "database" is a collection of information that stores managed data and allows for rapid searching and updating.

[1236] "Menu" means a list of dishes or meals from which a User can choose.

[1237] A "shopping list" refers to a list of products or ingredients that a user plans to purchase.

[1238] "Scanning" refers to reading information from a barcode or the like.

[1239] "Suggestion" refers to presenting appropriate options or actions to the user.

[1240] "User" refers to a consumer who uses the system to shop and cook.

[1241] A "system" refers to a structure in which a series of devices and programs function in conjunction with one another.

[1242] This invention is a system that manages refrigerator inventory by analyzing receipts and product photos after users have finished shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information obtained daily.The system uses hardware and software such as a cloud server, smartphones, smart glasses, and OCR (optical character recognition) technology.

[1243] The main process is as follows:

[1244] Post-shopping inventory management:

[1245] 1. The user takes a photo of the receipt or product with their smartphone and uploads it to the cloud server. This process uses the smartphone's camera and internet connection functions.

[1246] 2. The cloud server analyzes the uploaded image using OCR technology to extract information such as product name and quantity. This OCR technology is used to quickly and accurately digitize product information.

[1247] 3. The extracted product information is recorded in an inventory database on the cloud server, and the refrigerator's inventory is automatically updated.

[1248] Cooking Inventory Update:

[1249] 1. When cooking, the user inputs voice or images. For example, the user inputs "I used milk" into the smartphone.

[1250] 2. The smartphone converts the voice input into text and sends it to a cloud server. The speech recognition module handles this process.

[1251] 3. The cloud server updates the inventory database based on the received information, decrements the quantity of ingredients used, and notifies the user if the ingredients are out of stock.

[1252] Daily flyer updates:

[1253] 1. The cloud server retrieves flyer information as images from supermarket websites at a specified time every day. A web scraper performs this process automatically.

[1254] 2. The server uses OCR technology to analyze the flyer image, extract product information, and obtain data such as product name, price, and validity period.

[1255] 3. The extracted product information is registered in the flyer information database and becomes accessible to users.

[1256] Menu suggestions and shopping list generation:

[1257] 1. A user speaks, "What would you like for dinner tonight?", for example, through smart glasses.

[1258] 2. The smart device converts the voice into text and sends the information to a cloud server.

[1259] 3. The server compares the current inventory database and flyer information database to generate the appropriate menu and shopping list. A generative AI model performs this analysis and recommendation.

[1260] 4. The smart device will give the user voice suggestions, for example, "To make a chicken and broccoli stir-fry, you need to buy broccoli from the supermarket."

[1261] Inventory management when shopping:

[1262] 1. When a user shops in a physical store, they scan the product's barcode with their smartphone. The barcode scanner app then sends the product information to a cloud server.

[1263] 2. The cloud server parses the product information from the scanned barcode and adds it to the inventory database.

[1264] Example of a concrete example and prompt for the generative AI model:

[1265] Examples:

[1266] When a user purchases a lettuce at a physical store and scans the barcode with their smartphone, the cloud server updates the inventory database and records "1 lettuce added."

[1267] When a user asks the smart glasses, "What's for dinner tonight?", the cloud server will suggest "hamburger steak and salad" based on the refrigerator's inventory and store special offers.

[1268] Example prompt sentence:

[1269] When a user says, "What ingredients do I need for a hot pot?", the following answers are generated based on the current refrigerator inventory and special offers:

[1270] The recipe is "Chicken Hot Pot"

[1271] refrigerator inventory

[1272] Special sale information

[1273] This allows users to enjoy an efficient and profitable shopping and cooking experience.

[1274] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1275] Step 1:

[1276] The user takes a photo of a receipt or product with their smartphone. The input is the image of the receipt or product. The output is an image file stored in the smartphone's storage. The device prepares the captured image to be uploaded to the cloud server.

[1277] Step 2:

[1278] The device uploads the images it takes to the cloud server. The input is the image file stored on the smartphone. The output is the image data transferred to the cloud server. The device transmits the data via an internet connection.

[1279] Step 3:

[1280] The server analyzes the received image using OCR technology. The input is the image data transferred to the server, and the output is the extracted product information (product name, quantity, price, etc.). The server uses OCR technology to extract text data from the image.

[1281] Step 4:

[1282] The server records and updates the analyzed product information in the refrigerator's inventory database. The input is the product information extracted using OCR technology. The output is the updated inventory database. The server compares the product information with existing inventory data and updates the database.

[1283] Step 5:

[1284] When cooking, the user speaks the ingredients they will use into their smartphone. The input is the user's voice data. The output is the voice information converted into text. The device then uses a voice recognition module to convert the voice data into text.

[1285] Step 6:

[1286] The device converts voice input into text and sends the information to the server. The input is text data. The output is text information transferred to the cloud server. The device transmits the data via the Internet.

[1287] Step 7:

[1288] The server updates the inventory database based on the received information. The input is text information obtained by speech recognition. The output is the updated inventory database. The server processes the text information to reduce the amount of ingredients used.

[1289] Step 8:

[1290] The server uses a web scraper to retrieve supermarket flyer information at a specified time every day. The input is the website URL. The output is the retrieved flyer image. The server automatically retrieves flyer information using the web scraper.

[1291] Step 9:

[1292] The server uses OCR technology to analyze the acquired flyer image and extract product information. The input is the flyer image. The output is the analyzed product information (product name, price, validity period, etc.). The server uses OCR technology to extract text data from the flyer image.

[1293] Step 10:

[1294] The server registers the extracted product information in a flyer information database. The input is the product information extracted using OCR technology. The output is an updated flyer information database. The server stores the product information in the database.

[1295] Step 11:

[1296] The user speaks to the smart glasses, saying, "What would you like for dinner tonight?" The input is the user's voice data. The output is the voice information converted into text. The device uses a voice recognition module to convert the voice data into text.

[1297] Step 12:

[1298] The device converts voice input into text and sends the information to the server. The input is text data. The output is text information sent to the cloud server. The device sends the data over the Internet.

[1299] Step 13:

[1300] The server compares the current inventory database and flyer information database and generates the appropriate menu and shopping list. The input is the inventory database and flyer information database. The output is the generated menu and shopping list. The server generates the menu and list using a generative AI model.

[1301] Step 14:

[1302] The device makes audio suggestions to the user. The input is the generated menu and shopping list. The output is audio suggestions. The device uses a speaker to provide information to the user.

[1303] Step 15:

[1304] A user scans a product barcode with their smartphone in a physical store. The input is the product barcode data. The output is the barcode information stored on the smartphone. The device uses the camera function and a barcode scanner app.

[1305] Step 16:

[1306] The terminal sends the product information obtained by barcode scanning to the cloud server. The input is the barcode information. The output is the product information sent to the cloud server. The terminal sends the data via the Internet.

[1307] Step 17:

[1308] The server adds and updates the received product information to the inventory database. The input is the product information obtained by barcode scanning. The output is the updated inventory database. The server stores the product information in the database.

[1309] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1310] This system manages refrigerator inventory by analyzing receipts and product photos taken by the user after shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to the user based on supermarket flyer information acquired daily.The system also combines an emotion engine that recognizes the user's emotions using voice input and image analysis technology and adjusts the suggestions based on those emotions.

[1311] Program processing overview

[1312] Inventory management using shopping photos and receipts

[1313] 1. The user takes a photo of the receipt or product with their smartphone.

[1314] 2. Upload the photos taken by the device (smartphone) to the cloud server.

[1315] 3. The server analyzes the received image using optical character recognition (OCR) technology and extracts product information (product name, quantity, price, etc.).

[1316] 4. The server records and updates the analyzed product information in the refrigerator's inventory database.

[1317] Control delivery by voice or image input during cooking

[1318] 1. When the user prepares a dish, they tell the system the ingredients they will use using voice or images.

[1319] 2. The device converts the voice input into text and sends the information to the server. If the input is an image, the image is sent to the server.

[1320] 3. The server updates the refrigerator's inventory database based on the received information, decrementing the quantity of ingredients used and notifying the user if the ingredient is out of stock.

[1321] Daily flyer information acquisition and analysis

[1322] 1. The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[1323] 2. The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[1324] 3. The extracted product information is registered in the flyer information database.

[1325] Recognizing user emotions with an emotion engine

[1326] 1. Recognize emotions through voice input or camera when users consult the menu.

[1327] 2. The device converts the voice input into text and sends the information to the server. At the same time, the emotion engine analyzes the user's emotions from their tone of voice and facial expressions.

[1328] 3. The server adjusts the menu suggestions and shopping list contents based on the emotional information received.

[1329] Voice and emotion menu consultation and suggestions

[1330] 1. The user asks the system verbally, "What would you like for dinner tonight?"

[1331] 2. The device converts the voice input into text and sends the information to the server. At the same time, the emotion engine recognizes the user's emotions.

[1332] 3. The server compares the current refrigerator inventory database, flyer information database, and emotion information to generate an appropriate menu.

[1333] 4. The server creates a shopping list for ingredients that are in short supply and adjusts its suggestions based on feedback from the emotion engine.

[1334] 5. The device will then output a suggestion to the user, such as, "You seem a little tired today. How about a quick chicken and broccoli stir-fry?"

[1335] Specific examples

[1336] Specific examples of inventory management

[1337] A user purchases milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which uses OCR technology to extract the product information (1 liter of milk) and updates the refrigerator's inventory database with the entry "Add 1 liter of milk."

[1338] Specific examples of inventory management

[1339] While cooking, the user says "I used milk" by voice input. The device converts the voice to text and sends it to the server as "I used 1 liter of milk." The server updates the inventory database and records it as "I reduced 1 liter of milk."

[1340] Examples of flyer information

[1341] At 9:00 AM, the server retrieves the flyer image from the supermarket's website, analyzes the product information "Broccoli 98 yen," and registers it in the flyer information database. It saves it as "Broccoli 98 yen, sale period: 3 days."

[1342] Examples of emotion engines

[1343] The user asks, "What should we have for dinner tonight?" The emotion engine determines from the user's tone of voice and facial expression that they seem tired. The device converts this speech into text and sends it to the server. Based on the emotion engine's feedback, the server compares the refrigerator's inventory with flyer information and suggests, "You seem a little tired today, how about a simple chicken and broccoli stir-fry?"

[1344] This system provides a multifunctional mechanism that includes these steps and can support users in daily food management, menu planning, and shopping planning, including emotional information.

[1345] The processing flow will be explained below.

[1346] Manage refrigerator inventory using shopping photos and receipts

[1347] Step 1:

[1348] The user takes a photo of the receipt or product with their smartphone.

[1349] Open the camera app on your smartphone and take a photo of your receipt or purchased items.

[1350] Step 2:

[1351] The device uploads the photos it takes to the server.

[1352] Use the dedicated application and press the button to upload the photo to the cloud server.

[1353] Step 3:

[1354] The server applies OCR technology to analyze the received image.

[1355] The server invokes an image processing module to extract text information from the image.

[1356] Step 4:

[1357] The server extracts the parsed product information (product name, quantity, price, etc.).

[1358] Product name, quantity, and price are identified from the text information and converted into a database format.

[1359] Step 5:

[1360] The server records the extracted product information in the refrigerator's inventory database and updates it.

[1361] Adds the product information to the corresponding user's inventory database and updates existing inventory information.

[1362] Control delivery by voice or image input during cooking

[1363] Step 1:

[1364] When cooking, the user verbally tells the system what ingredients to use.

[1365] Use your smartphone's microphone to input voice instructions such as "I used chicken."

[1366] Step 2:

[1367] The device converts the voice input into text and sends the information to the server.

[1368] Uses voice recognition software to convert speech into text.

[1369] Step 3:

[1370] The server updates the refrigerator inventory database based on the received information.

[1371] The quantity of the specified ingredient is reduced and the inventory database is updated.

[1372] Step 4:

[1373] If the server is out of stock, the user is notified.

[1374] If the stock reaches 0, a notification message will be sent to the user.

[1375] Daily flyer information acquisition and analysis

[1376] Step 1:

[1377] The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[1378] Based on a fixed schedule, it automatically visits each supermarket's website and downloads flyer images.

[1379] Step 2:

[1380] The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[1381] Use the flyer image processing module to extract product information from the image.

[1382] Step 3:

[1383] The extracted product information is registered in a flyer information database.

[1384] The flyer information is stored in a database based on the product name, price, and validity period.

[1385] Recognizing user emotions with an emotion engine

[1386] Step 1:

[1387] When the user consults the menu, he or she uses voice input.

[1388] Using the smartphone's microphone, voice menu consultation is performed, asking, "What would you like for dinner tonight?"

[1389] Step 2:

[1390] The device converts the voice input into text and sends the information to the server.

[1391] Use speech recognition software to convert speech to text.

[1392] Step 3:

[1393] The emotion engine analyzes emotions from the user's tone of voice.

[1394] A voice tone analysis module is used to identify the user's emotions.

[1395] Step 4:

[1396] The server adjusts the suggested menu and shopping list based on the emotional information received.

[1397] Taking emotion information into consideration, simple dishes and dishes that the user likes are selected.

[1398] Voice and emotion menu consultation and suggestions

[1399] Step 1:

[1400] The user verbally asks the system, "What would you like for dinner tonight?"

[1401] Voice menu consultation is performed using the smartphone's microphone.

[1402] Step 2:

[1403] The device converts the voice input into text and sends the information to the server.

[1404] Use speech recognition software to convert speech to text.

[1405] Step 3:

[1406] The server compares the current refrigerator inventory database, flyer information database, and emotion information.

[1407] Cross-check the database to see what ingredients are available and what discounts are available.

[1408] Step 4:

[1409] Generate appropriate menus and shopping lists for missing ingredients.

[1410] Generate suggested recipes and add missing ingredients to your shopping list.

[1411] Step 5:

[1412] The terminal outputs the suggestions to the user by voice.

[1413] Using the smartphone's speaker, the system makes a voice suggestion such as, "You seem a little tired today, how about an easy-to-make chicken and broccoli stir-fry?"

[1414] Through the above processing flow, this system can efficiently support users in managing ingredients, planning menus, and making shopping plans, including emotional information.

[1415] Example 2

[1416] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1417] In today's busy lifestyles, food inventory management has become a complicated task for many households. Deciding on a menu and creating a shopping list for the necessary ingredients is also a time-consuming task. Furthermore, there is a demand for appropriate menu suggestions based on the user's emotions and physical condition, but current technology has difficulty meeting this demand.

[1418] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for taking pictures of receipts and products after shopping and analyzing product information from the captured images; means for automatically updating inventory in the refrigerator based on the analyzed product information; means for receiving information on ingredients used in cooking when the user inputs the ingredients by voice or image and updating the inventory in the refrigerator; means for acquiring flyer information from an external website every day, analyzing the information, and registering the product information in a database; means for receiving voice input from the user discussing menus, and generating and suggesting appropriate menus and shopping lists based on the current inventory and the acquired flyer information; means for recognizing emotions from the user's voice tone and facial expressions; and means for adjusting the suggestions based on the emotion recognition information. This enables efficient and accurate daily ingredient management, menu planning, and shopping planning, including emotional information.

[1419] "Receipts and product photos after shopping" refers to receipts showing purchase details that users receive when they purchase products at a store, and image data of purchased products.

[1420] "Product Information" means detailed data about the purchased item, such as product name, quantity, and price.

[1421] "Optical character recognition (OCR) technology" refers to the technology that scans images and handwritten characters and converts them into text data.

[1422] "Inventory database" refers to a database system for recording and managing inventory status and product information in a refrigerator.

[1423] "Voice input" means a means by which a user communicates information to a system using speech.

[1424] "Image input" means the means by which a user communicates information to a system using an image.

[1425] "Web scraping" refers to the technique of extracting data from websites using automated programs.

[1426] "Flyer information" refers to data including special sale items and price information provided by stores for promotional purposes.

[1427] "Emotion engine" refers to algorithms and technologies that analyze a user's tone of voice and facial expressions to recognize their emotional state.

[1428] "Suggestions" refers to information such as menus and shopping lists provided to users.

[1429] "Refrigerator inventory" refers to the ingredients and products stored in the user's refrigerator.

[1430] "Voice input for menu consultation" refers to voice input by the user to ask the system about the dish they would like to make.

[1431] This system analyzes receipts and product photos taken by users after shopping, manages refrigerator inventory, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information acquired daily.The system also combines an emotion engine that recognizes the user's emotions using voice input and image analysis technology and adjusts the suggestions based on those emotions.

[1432] Inventory management using shopping photos and receipts

[1433] Users take photos of receipts and products with their smartphones. The device uploads these images to a cloud server. The server then analyzes the images using optical character recognition (OCR) technology. Specifically, OCR software such as Tesseract OCR is used to extract product information such as product name, quantity, and price. The server then records this information in the refrigerator's inventory database and automatically updates it.

[1434] Examples:

[1435] A user buys milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which uses Tesseract OCR to extract the product information (1 liter of milk) and updates the refrigerator inventory with the entry "Add 1 liter of milk."

[1436] Control delivery by voice or image input during cooking

[1437] When a user prepares a dish, they tell the system which ingredients they will use using voice or images. The device converts the voice input into text and sends that information to the server. If an image is input, the image is sent to the server. The server updates the refrigerator's inventory database based on the received information and reduces the quantity of ingredients used. If the inventory falls below a certain level, the user is notified.

[1438] Examples:

[1439] While cooking, the user voice-inputs "Milk used." The device converts the voice data into text and sends it to the server as "1 liter of milk used." The server updates the inventory database and records "1 liter of milk decreased."

[1440] Daily flyer information acquisition and analysis

[1441] The server retrieves flyer information as image data from each supermarket's website at a specified time every day. Libraries such as Beautiful Soup and Selenium are used for web scraping. The server analyzes the flyer images retrieved using OCR technology and extracts product information (product name, price, validity period, etc.). This information is registered in the flyer information database.

[1442] Examples:

[1443] The server retrieves the flyer image from the supermarket's website at 9:00 AM, analyzes the product information "Broccoli 98 yen" using Beautiful Soup, and saves it in the flyer information database as "Broccoli 98 yen, sale period: 3 days."

[1444] Recognizing user emotions with an emotion engine

[1445] When a user consults about a menu, emotions are recognized through voice input or a camera. The device converts the voice input into text and sends the information to the server. At the same time, the device is equipped with an emotion analysis engine such as OpenCV or Face API, which analyzes emotions from the user's tone of voice and facial expressions. The server then adjusts the menu suggestions and shopping list contents based on this emotional information.

[1446] Examples:

[1447] The user asks, "What should we have for dinner tonight?" The device converts this speech into text, and at the same time, the emotion engine detects tired facial expressions and analyzes them. Based on the feedback from the emotion engine, the server adjusts the suggestions, such as "You seem a little tired today, so I'll recommend some easy-to-make dishes."

[1448] Voice and emotion menu consultation and suggestions

[1449] The user verbally asks the system, "What would you like for dinner tonight?" The device converts the voice input into text and sends the information to the server. At the same time, the emotion engine recognizes the user's emotions. The server compares the current refrigerator inventory database, flyer information database, and emotion information to generate an appropriate menu. The server creates a shopping list for missing ingredients and further adjusts this list based on feedback from the emotion engine. The device then outputs the suggestions to the user verbally.

[1450] Examples:

[1451] The user asks the system via voice, "What would you like for dinner tonight?" The device converts the text and sends the emotion data to the server, which then makes a suggestion to the user via voice, such as, "You seem a little tired today, how about a quick chicken and broccoli stir-fry?"

[1452] Example prompt sentence:

[1453] "Take a photo of your supermarket receipt and upload it"

[1454] "List the ingredients used while cooking by voice"

[1455] "I'd like some suggestions for dinner tonight."

[1456] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1457] Inventory management using shopping photos and receipts

[1458] Step 1:

[1459] After shopping, the user takes a photo of the receipt or product with their smartphone. The user launches the camera app and takes a photo, taking care to ensure that the receipt and product are clearly visible. This is the initial step in obtaining input data (receipt photo).

[1460] Step 2:

[1461] The device uploads the captured image to the cloud server. The device uses the HTTP protocol via an internet connection to send the image data to the server. The input is a photo of the receipt, and the output is the transmission of image data to the server.

[1462] Step 3:

[1463] The image received by the server is analyzed using OCR technology. Software such as Tesseract OCR is used to recognize characters in the image and extract text information such as product name, quantity, and price. The input is image data, and the output is analyzed product information.

[1464] Step 4:

[1465] The server records the extracted product information in the refrigerator's inventory database and updates it. The database update process keeps the inventory information up to date. The input is the analyzed product information, and the output is the updated inventory database.

[1466] Control delivery by voice or image input during cooking

[1467] Step 1:

[1468] When a user prepares a dish, they tell the system what ingredients they will use by voice or image. For example, they can say "I used milk." This is input data for recording the ingredients used.

[1469] Step 2:

[1470] The device converts the user's voice input into text data. It uses Google Cloud Speech-to-Text, a speech recognition technology, to convert speech to text. The input is voice data, and the output is text data.

[1471] Step 3:

[1472] The terminal sends the converted text data to the server via an Internet connection using the HTTP protocol. The input is text data, and the output is data sent to the server.

[1473] Step 4:

[1474] The server updates the refrigerator's inventory database based on the received data. It reduces the quantity of ingredients used and notifies the user when the stock falls below a certain level. The input is text data, and the output is the updated inventory database and a notification.

[1475] Daily flyer information acquisition and analysis

[1476] Step 1:

[1477] The server retrieves flyer images from each supermarket's website at a specified time every day using a web scraping tool (e.g., Beautiful Soup or Selenium). The input is the supermarket's website URL, and the output is the flyer image.

[1478] Step 2:

[1479] The server analyzes the flyer image acquired by the server using OCR technology and extracts product information. The input is the flyer image and the output is the analyzed product information.

[1480] Step 3:

[1481] The server registers the analyzed product information in a flyer information database. The database is updated and sale information is stored. The input is the analyzed product information, and the output is the updated flyer information database.

[1482] Recognizing user emotions with an emotion engine

[1483] Step 1:

[1484] Emotion recognition is performed through voice input or camera when the user is consulting a menu. For example, the user faces the camera and says, "What should I have for dinner tonight?" This is the initial step to obtain input data (voice and image).

[1485] Step 2:

[1486] The device converts the user's voice input into text and sends that information to the server. At the same time, the emotion engine analyzes emotions from voice tone and facial expressions. OpenCV and Face API are used. The input is voice and image data, and the output is text data and emotion data.

[1487] Step 3:

[1488] The server adjusts menu suggestions based on emotional information. It compares the inventory database and flyer information to generate the optimal menu for the user. The input is emotional data and inventory / flyer information, and the output is the adjusted menu suggestions.

[1489] Voice and emotion menu consultation and suggestions

[1490] Step 1:

[1491] The user verbally asks the system what dish to make, for example, "What would you like for dinner tonight?" This is the input data for the suggestion.

[1492] Step 2:

[1493] The device converts voice input into text and sends it to the server. At the same time, the emotion engine recognizes the user's emotions. The input is voice data, and the output is text data and emotion data.

[1494] Step 3:

[1495] The server generates an appropriate menu based on the current refrigerator inventory database, flyer information database, and emotion information. The input is inventory and flyer information and emotion data, and the output is the generated menu.

[1496] Step 4:

[1497] The server creates a shopping list of missing ingredients and further refines the suggestions. The input is the generated menu, and the output is the shopping list and refined suggestions.

[1498] Step 5:

[1499] The device outputs a suggestion to the user by voice, for example, "You seem a little tired today, how about a quick chicken and broccoli stir-fry?" The input is the tailored suggestion, and the output is the voice prompt.

[1500] (Application example 2)

[1501] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1502] Conventional refrigerator inventory management systems have the problem that updating inventory is cumbersome and difficult for users to use. They also lack a way for users to easily grasp the ingredients and menu items they need, and are unable to make suggestions based on their emotions. Furthermore, while real-time inventory updates and emotion-based suggestions are required, conventional systems have been unable to comprehensively address these issues.

[1503] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1504] In this invention, the server includes means for taking pictures of receipts and items after shopping and analyzing item information from the captured images, means for automatically updating inventory in the refrigerator based on the analyzed item information, means for receiving information on ingredients used in cooking when the user inputs the information by voice or image and updating the inventory in the refrigerator, means for acquiring flyer information from an external website every day, analyzing the information, and registering the item information in a database, means for receiving voice input from the user discussing menus, and generating and suggesting appropriate menus and shopping lists based on the current inventory and the acquired flyer information, means for analyzing user emotional information, and means for adjusting the suggestions based on the analyzed emotional information. This makes it possible to efficiently manage inventory in the refrigerator and generate menu suggestions and shopping lists that reflect the user's emotions.

[1505] A "post-purchase receipt" is a paper statement issued after the purchase of an item, which contains information including the name, quantity, and price of the purchased item.

[1506] An "item photo" is an image of a purchased item taken by a user after shopping, and includes the appearance and label information of the item.

[1507] "Item information" refers to detailed information such as the name, quantity, and price of an item, and is data extracted by image analysis or character recognition.

[1508] "Inventory in refrigerator" is a database that shows the total amount of food and beverages stored in a refrigerator or other refrigeration device.

[1509] "Voice input" is a means by which the user verbally communicates to the system the materials to be used, the details of the proposal, etc., which is converted into text data using voice recognition technology.

[1510] "Image input" is a means by which a user sends an image of the material or item they are using to the system, and information is extracted using image analysis techniques.

[1511] "External websites" are web pages of commercial facilities or information providers that exist on the Internet, and on which flyer information and the like are made public.

[1512] "Flyer information" is image data or text data that includes special sale information and product price lists offered by commercial facilities.

[1513] A "database" is an electronic information aggregation system for systematically managing and storing analyzed product information, inventory information, and flyer information.

[1514] "User's emotional information" is data that indicates the emotional state of the user, which is analyzed from the user's tone of voice, facial expression, content of words, and the like.

[1515] "Suggestion content" refers to information such as menus and shopping lists that the system provides to users, and is generated based on inventory data, emotional information, and the like.

[1516] This invention begins when a user takes a photo of a receipt or item after shopping with their smartphone and uploads it to a cloud server. The server analyzes the image using optical character recognition (OCR) technology to extract item information (item name, quantity, price, etc.). This item information is recorded in an inventory database inside the refrigerator and automatically updated. When the user prepares a meal, they tell the system which ingredients they will be using via voice or image input. The device converts the voice to text and sends that information to the server. The server updates the inventory database based on the received information and reduces the quantity of ingredients used. In addition, flyer information is obtained daily from an external website, analyzed as an image, and item information (item name, price, expiration date, etc.) is extracted. This information is registered in the flyer information database.

[1517] The system's main hardware includes a smartphone (camera, microphone) and a cloud server. The main software used includes optical character recognition (OCR), voice input analysis, and image analysis technologies, which utilize cloud services such as Google Cloud Vision API and Google Cloud Speech-to-Text. When a user asks the system, "What would you like for dinner tonight?", the voice is recorded by the smartphone's microphone. The device converts the voice into text and sends it to the cloud server. At the same time, an emotion engine analyzes the user's emotions from their tone of voice and facial expressions. The server compares the current refrigerator inventory database, flyer information database, and emotion information to generate an appropriate menu. It creates a shopping list for any missing ingredients and adjusts the suggestions based on feedback from the emotion engine.

[1518] For example, a user purchases milk at a supermarket and takes a photo of the receipt with their smartphone. When this image is uploaded to the cloud server, OCR technology is used to extract the product information—"1 liter of milk." This information is recorded in the refrigerator's inventory database and updated as "1 liter of milk added." Furthermore, if the user voice-inputs "I used milk" while cooking, this voice is converted to text and sent to the cloud server. The server updates the inventory database based on this information, recording it as "1 liter of milk reduced." Furthermore, the server retrieves flyer images from each supermarket's website at a specified time each morning, analyzes the product information, such as "Broccoli 98 yen," and registers it in the flyer information database. This information is saved as "Broccoli 98 yen, special sale period: 3 days." The emotion engine determines that the user is tired based on their tone of voice and facial expression when they ask, "What should I have for dinner tonight?" The server then compares the refrigerator's inventory with the flyer information based on the emotion engine's feedback and suggests, "You seem tired today. How about a simple chicken and broccoli stir-fry?"

[1519] An example of a prompt for a generative AI model might be, "Please tell me how to analyze receipts and item photos taken by the user and update that information in the refrigerator inventory database."

[1520] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1521] Step 1:

[1522] After shopping, the user takes a photo of the receipt or item with their smartphone. The input is the captured image, and the output is an image file saved in the smartphone's storage. Specifically, the user opens the smartphone's camera app, takes a photo of the receipt or item, and saves it as an image file.

[1523] Step 2:

[1524] The device uploads the captured image to the cloud server. The input is the image file stored on the smartphone, and the output is the image data uploaded to the cloud server. Specifically, the device establishes communication with the cloud server and sends the image file captured by the user to the server.

[1525] Step 3:

[1526] The server analyzes the image using optical character recognition (OCR) technology and extracts product information. The input is the image data uploaded to the cloud server, and the output is text data such as the product name, quantity, and price. Specifically, the server uses OCR technology (e.g., Tesseract) to analyze the text portion of the image and extracts product information as text data.

[1527] Step 4:

[1528] The server records the analyzed item information in the inventory database inside the cooler and updates it. The input is text data extracted using OCR technology, and the output is updated inventory data. Specifically, the server accesses the inventory database and updates the inventory data based on the extracted item information.

[1529] Step 5:

[1530] The user communicates the ingredients used in cooking to the system through voice or image input. The input is the user's voice instruction or image data, and the output is information sent by the device to the server. Specifically, the user communicates the ingredients used to the system using the microphone or camera on their smartphone.

[1531] Step 6:

[1532] The device converts voice input into text and sends the information to the server. The input is the user's voice data, and the output is text data. Specifically, the device converts the voice into text using voice recognition technology (e.g., Google Cloud Speech-to-Text) and sends it to the server.

[1533] Step 7:

[1534] The server updates the inventory database in the refrigerator based on the received information. The input is text data or image data, and the output is updated inventory data. Specifically, the server accesses the inventory database and decreases the quantity of materials used.

[1535] Step 8:

[1536] The server retrieves flyer information from an external website every day and analyzes it as an image. The input is the website URL, and the output is flyer image data. Specifically, the server accesses the website at a specified time and downloads the flyer image.

[1537] Step 9:

[1538] The server analyzes the flyer image and registers the product information (product name, price, validity period, etc.) in a database. The input is the image data of the flyer, and the output is the product information registered in the database. Specifically, the server analyzes the flyer image using OCR technology, extracts the product information, and stores it in the database.

[1539] Step 10:

[1540] The system receives voice input from the user to discuss a menu, and the server generates an appropriate menu and shopping list based on current inventory and flyer information. The input is the voice instruction and inventory and flyer information, and the output is a suggested menu and shopping list. Specifically, the server analyzes the voice instruction and compares the inventory data with the flyer information to generate an appropriate menu.

[1541] Step 11:

[1542] The server analyzes the user's emotional information and adjusts the suggestions. The input is the user's voice tone and facial expression data, and the output is the adjusted suggestions. Specifically, the server analyzes the user's emotions using emotion recognition technology (e.g., Google Cloud Vision API) and adjusts the suggestions based on the emotions.

[1543] Step 12:

[1544] The device outputs the adjusted proposal content to the user by voice. The input is the adjusted proposal content, and the output is a voice notification to the user. Specifically, the device converts text data into voice and notifies the user of the proposal content.

[1545] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1546] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1547] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1548] [Fourth embodiment]

[1549] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1550] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1551] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1552] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1553] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1554] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1555] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1556] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1557] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1558] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1559] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1560] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1561] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1562] The present invention is a system that manages refrigerator inventory by analyzing receipts and product photos taken by users after shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information obtained daily.

[1563] Program processing overview

[1564] Inventory management using receipts and product photos after shopping

[1565] 1. The user takes a photo of the receipt or product with their smartphone.

[1566] 2. Upload the photos taken by the device (smartphone) to the cloud server.

[1567] 3. The server analyzes the received image using optical character recognition (OCR) technology and extracts product information (product name, quantity, price, etc.).

[1568] 4. The server records and updates the analyzed product information in the refrigerator's inventory database.

[1569] Control delivery by voice or image input during cooking

[1570] 1. When the user prepares a dish, they tell the system the ingredients they will use using voice or images.

[1571] 2. The device converts the voice input into text and sends the information to the server. If the input is an image, the image is sent to the server.

[1572] 3. The server updates the refrigerator's inventory database based on the received information, decrementing the quantity of ingredients used and notifying the user if the ingredient is out of stock.

[1573] Daily flyer information acquisition and analysis

[1574] 1. The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[1575] 2. The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[1576] 3. The extracted product information is registered in the flyer information database.

[1577] Voice menu consultation and suggestions

[1578] 1. The user asks the system verbally, "What would you like for dinner tonight?"

[1579] 2. The device converts the voice input into text and sends the information to the server.

[1580] 3. Based on the information received by the server, the current refrigerator inventory database is compared with the flyer information database to generate an appropriate menu and a shopping list of missing ingredients.

[1581] 4. The device will output a suggestion to the user via voice, such as "To make a chicken and broccoli stir-fry, you will need to buy broccoli from the supermarket."

[1582] Specific examples

[1583] Specific examples of inventory management

[1584] A user purchases milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which uses OCR technology to extract the product information (1 liter of milk) and updates the refrigerator's inventory database with the entry "Add 1 liter of milk."

[1585] Specific examples of inventory management

[1586] While cooking, the user says "I used milk" by voice input. The device converts the voice to text and sends it to the server as "I used 1 liter of milk." The server updates the inventory database and records it as "I reduced 1 liter of milk."

[1587] Examples of flyer information

[1588] At 9:00 AM, the server retrieves the flyer image from the supermarket's website, analyzes the product information "Broccoli 98 yen," and registers it in the flyer information database. It saves it as "Broccoli 98 yen, sale period: 3 days."

[1589] Specific examples of menu consultation

[1590] The user asks, "What should I have for dinner tonight?" The device converts this speech into text and sends it to the server. The server compares the current inventory database with the flyer information database, determines that "there is chicken in the fridge and broccoli on sale," and suggests "stir-fried chicken and broccoli."

[1591] This system provides a multifunctional mechanism that includes these steps, and centrally and automatically supports users in daily food management, menu planning, and shopping planning.

[1592] The processing flow will be explained below.

[1593] Manage refrigerator inventory using shopping photos and receipts

[1594] Step 1:

[1595] The user takes a photo of the receipt or product with their smartphone.

[1596] Open the camera app on your smartphone and take a photo of your receipt or purchased items.

[1597] Step 2:

[1598] The device uploads the photos it takes to the server.

[1599] Use the dedicated application and press the button to upload the photo to the cloud server.

[1600] Step 3:

[1601] The server applies OCR technology to analyze the received image.

[1602] The server invokes an image processing module to extract text information from the image.

[1603] Step 4:

[1604] The server extracts the parsed product information (product name, quantity, price, etc.).

[1605] Product name, quantity, and price are identified from the text information and converted into a database format.

[1606] Step 5:

[1607] The server records the extracted product information in the refrigerator's inventory database and updates it.

[1608] Adds the product information to the corresponding user's inventory database and updates existing inventory information.

[1609] Control delivery by voice or image input during cooking

[1610] Step 1:

[1611] When cooking, the user verbally tells the system what ingredients to use.

[1612] Use your smartphone's microphone to input voice instructions such as "I used chicken."

[1613] Step 2:

[1614] The device converts the voice input into text and sends the information to the server.

[1615] Uses voice recognition software to convert speech into text.

[1616] Step 3:

[1617] The server updates the refrigerator inventory database based on the received information.

[1618] The quantity of the specified ingredient is reduced and the inventory database is updated.

[1619] Step 4:

[1620] If the server is out of stock, the user is notified.

[1621] If the stock reaches 0, a notification message will be sent to the user.

[1622] Daily flyer information acquisition and analysis

[1623] Step 1:

[1624] The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[1625] Based on a fixed schedule, it automatically visits each supermarket's website and downloads flyer images.

[1626] Step 2:

[1627] The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[1628] Use the flyer image processing module to extract product information from the image.

[1629] Step 3:

[1630] The extracted product information is registered in a flyer information database.

[1631] The flyer information is stored in a database based on the product name, price, and validity period.

[1632] Voice menu consultation and suggestions

[1633] Step 1:

[1634] The user verbally asks the system, "What would you like for dinner tonight?"

[1635] Voice menu consultation is performed using the smartphone's microphone.

[1636] Step 2:

[1637] The device converts the voice input into text and sends the information to the server.

[1638] Use speech recognition software to convert speech to text.

[1639] Step 3:

[1640] Based on the information received by the server, the current refrigerator inventory database is compared with the flyer information database.

[1641] Cross-check the database to see what ingredients are available and what discounts are available.

[1642] Step 4:

[1643] Generate appropriate menus and shopping lists for missing ingredients.

[1644] Generate suggested recipes and add missing ingredients to your shopping list.

[1645] Step 5:

[1646] The terminal outputs the suggestions to the user by voice.

[1647] Using the smartphone's speaker, a voice suggestion is made: "To make chicken and broccoli stir-fry, you need to buy broccoli at the supermarket."

[1648] Through the above processing flow, this system efficiently supports the user in managing ingredients, planning menus, and making shopping plans.

[1649] Example 1

[1650] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1651] In modern life, efficient ingredient management and menu planning are important challenges for many households. However, manually managing ingredients and creating appropriate menus and shopping lists is time-consuming and labor-intensive, making it a burden for many people. In particular, manually collecting and using information from refrigerator inventory and supermarket flyers is cumbersome and inefficient. Additionally, it is difficult to update the inventory of ingredients used during cooking in real time, resulting in problems such as out-of-stock situations and food waste.

[1652] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1653] In this invention, the server includes means for taking pictures of receipts and products after a user has finished shopping and analyzing the product information from the captured images, means for automatically updating the inventory in the refrigerator based on the analyzed product information, means for receiving information about ingredients used in cooking when the user inputs the information by voice or image and updating the inventory in the refrigerator, means for acquiring flyer information from an external website every day, analyzing the information, and registering the product information in a database, means for receiving voice input from the user discussing menus and generating and suggesting appropriate menus and shopping lists based on the current inventory and the acquired flyer information, means for converting the user's voice input into text using a voice recognition system and sending the information to the server, and means for outputting suggestions by voice using a text-to-speech conversion system.This allows the user to efficiently manage their refrigerator inventory without any hassle and have appropriate menus and shopping lists automatically suggested.

[1654] A "user" is an entity that uses the system to manage ingredients, create menus, and plan shopping.

[1655] A "receipt" is a paper record of a purchase made at a store.

[1656] "Product photo" is an image of the purchased product.

[1657] "Means for analyzing product information from images" refers to a method for extracting information such as product name, quantity, and price from images using optical character recognition technology.

[1658] The "means for automatically updating inventory in the refrigerator" is a system that updates the database based on analyzed product information to reflect the new inventory status.

[1659] "Means for inputting information about ingredients used in cooking by voice or image" refers to a method in which the user communicates information about ingredients used to the system through voice or image.

[1660] A "voice recognition system" is a technology that converts voice input into text.

[1661] The "means for updating inventory" is a system that modifies the inventory database based on user input and maintains the latest inventory status.

[1662] "Means for obtaining daily flyer information from external websites" refers to a technology for downloading flyer images from websites of supermarkets and the like via the Internet.

[1663] The "means of analyzing and registering product information in a database" refers to a system that analyzes acquired flyer images using optical character recognition technology and stores the product information in a database.

[1664] The "means for receiving voice input for consulting a menu" is a method for receiving information in which a user requests menu suggestions by voice.

[1665] "Means for generating appropriate menus and shopping lists" refers to technology that creates optimal cooking menus and lists of ingredients that are in short supply for users based on current inventory information and flyer information.

[1666] "Means for converting a user's voice input into text using a voice recognition system" refers to a method that uses technology to convert voice data into text data.

[1667] The "means for outputting suggestions by voice using a text-to-voice conversion system" is a technology for converting generated text information into voice and conveying it to the user.

[1668] A "natural language processing model" is a technology that generates appropriate text and analyzes intent based on user input.

[1669] This invention is a system that manages refrigerator inventory by analyzing receipts and product photos taken by users after shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information obtained daily.

[1670] The system is implemented using the following hardware and software:

[1671] Hardware:

[1672] Smartphone (user device)

[1673] Cloud Server

[1674] Refrigerator (physical storage location)

[1675] software:

[1676] Optical character recognition technology (OCR): Google Cloud Vision API

[1677] Speech recognition system: Google Cloud Speech-to-Text, Amazon Transcribe

[1678] Text-to-speech system: Amazon Polly

[1679] Web scraping tools: Selenium, BeautifulSoup

[1680] Natural language processing model: GPT-3

[1681] Database management systems: MySQL, PostgreSQL

[1682] Voice UI Applications

[1683] After a user finishes shopping at the supermarket, they take a photo of the receipt or product with their smartphone. The device uploads the photo to a cloud server, which then analyzes the image using the Google Cloud Vision API. Using OCR technology, product information (product name, quantity, price, etc.) is extracted, and the server automatically registers and updates the analyzed information in the refrigerator's inventory database.

[1684] When cooking, the user can tell the system which ingredients they will use by voice or image input. The device converts the voice input into text using Google Cloud Speech-to-Text or Amazon Transcribe and sends the information to the server. When using image input, the device also sends the image to the server, which updates the inventory database based on the analysis results. The server reduces the quantity of ingredients used and notifies the user if they are out of stock.

[1685] The server also retrieves flyer images from each supermarket's website at a specified time every day using web scraping tools (Selenium, BeautifulSoup). The server then analyzes the flyer images using Tesseract OCR, extracts product information (product name, price, validity period, etc.), and registers it in the flyer information database.

[1686] When a user asks the system, "What would you like for dinner tonight?", the device converts the speech to text using Amazon Transcribe and sends it to the server. The server compares the current inventory database with the flyer information database and uses the GPT-3 natural language processing model to generate an appropriate menu and a shopping list of missing ingredients. The device then uses Amazon Polly to output the suggestions to the user.

[1687] Specific examples

[1688] For example, suppose a user buys milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which then uses the Google Cloud Vision API to extract the product information, such as "1 liter of milk." The information is then registered and updated in the refrigerator's inventory database as "1 liter of milk added."

[1689] If a user says "I used milk" while cooking, the device converts the speech into text using Google Cloud Speech-to-Text and sends it to the server as "I used 1 liter of milk." The server then updates the inventory database, recording "I reduced 1 liter of milk."

[1690] Every day at 9:00 a.m., the server uses Selenium to retrieve flyer images from the supermarket's website, analyzes the product information, such as "Broccoli 98 yen," using Tesseract OCR, and registers it in the flyer information database.

[1691] When a user asks, "What should I have for dinner tonight?", the device converts the speech into text using Amazon Transcribe and sends it to the server. The server compares the current inventory database with the flyer information database and uses the GPT-3 natural language processing model to suggest "chicken and broccoli stir-fry." The device then uses Amazon Polly to announce, "To make chicken and broccoli stir-fry, you need to buy broccoli at the supermarket."

[1692] Prompt Sentence Examples

[1693] Example prompt sentences when the user says "I used milk":

[1694] The user says "Milk used." Update the refrigerator inventory database. Decrease the quantity of milk by 1 liter and record the result.

[1695] Example prompt sentences when a user says "What should I have for dinner tonight?":

[1696] The user says, "What should I have for dinner tonight?" Match the current refrigerator inventory database with the flyer information database to suggest an appropriate dinner menu. If there are any ingredients missing, prepare a shopping list for them.

[1697] The above is a specific embodiment for carrying out the present invention.

[1698] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1699] Inventory management using receipts and product photos after shopping

[1700] Step 1:

[1701] After a user finishes shopping at a supermarket, they take a photo of the receipt or product with their smartphone. The user opens the smartphone's camera app and takes a photo of the receipt or product. This image becomes input data for the system.

[1702] Step 2:

[1703] The device (smartphone) takes a photo and uploads it to the cloud server. The upload program encodes the image data and sends it to the cloud server via a secure connection. The input is an image of the receipt or product, and the output is the storage of the image data on the cloud server.

[1704] Step 3:

[1705] The server analyzes the received photos using optical character recognition (OCR) technology. Specifically, it uses the Google Cloud Vision API to extract product information such as product name, quantity, and price from the image. The input is image data, and the output is analyzed product information (text data).

[1706] Step 4:

[1707] The server records and updates the parsed product information in the refrigerator's inventory database. The database management system (e.g., MySQL) adds the new product information and updates the existing inventory data. The input is the parsed product information, and the output is the updated inventory database.

[1708] Control delivery by voice or image input during cooking

[1709] Step 1:

[1710] When a user prepares a dish, they can tell the system which ingredients they will use by voice or image. Specifically, the user opens the smartphone app and either voice-inputs "I used milk" or takes a photo of the ingredients used. The input is voice data or image data.

[1711] Step 2:

[1712] The device converts voice input into text and sends that information to the server. "Google Cloud Speech-to-Text" is used as the voice recognition system. Voice is converted into text and the textual information is sent to the server. The input is voice data and the output is text data. In the case of image input, the image is sent directly to the server. The input is image data and the output is saving the image data to the server.

[1713] Step 3:

[1714] The server updates the refrigerator's inventory database based on the information it receives. Specifically, it analyzes the text data and reduces the quantity of ingredients used. If the item is out of stock, a program is activated to notify the user. The input is the text data, and the output is the updated inventory database and a notification.

[1715] Daily flyer information acquisition and analysis

[1716] Step 1:

[1717] The server retrieves flyer information as images from each supermarket's website at a specified time every day. Specifically, the web scraping tools "Selenium" and "BeautifulSoup" are used to download the flyer images. The input is the supermarket's website URL, and the output is the retrieved flyer image.

[1718] Step 2:

[1719] The server analyzes the flyer image it has acquired. Specifically, it uses "Tesseract OCR" to extract product information (product name, price, validity period, etc.). The input is the flyer image, and the output is the analyzed product information (text data).

[1720] Step 3:

[1721] The extracted product information is registered in a flyer information database. New flyer information is added and saved using a database management system (e.g., PostgreSQL). The input is the parsed product information, and the output is an updated flyer information database.

[1722] Voice menu consultation and suggestions

[1723] Step 1:

[1724] The user asks the system by voice, "What would you like for dinner tonight?" The user opens the app on their smartphone and enters voice input. The input is voice data.

[1725] Step 2:

[1726] The device converts voice input into text and sends that information to the server. Specifically, it uses "Amazon Transcribe." The voice is converted into text and the text information is sent to the server. The input is voice data and the output is text data.

[1727] Step 3:

[1728] Based on the information received by the server, the current refrigerator inventory database is compared with the flyer information database. The natural language processing model "GPT-3" is used to generate an appropriate menu and a shopping list of missing ingredients. The input is text data and database information, and the output is a menu and shopping list.

[1729] Step 4:

[1730] The device outputs the suggestions by voice. Using the text-to-speech conversion system "Amazon Polly," the generated text information is converted into voice and conveyed to the user. The input is the text data of the suggestions, and the output is a voice suggestion.

[1731] The above processing steps allow users to effortlessly manage refrigerator inventory, manage food delivery while cooking, and receive appropriate menu suggestions. Shopping lists are also automatically generated based on flyer information acquired daily.

[1732] (Application example 1)

[1733] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1734] Modern consumers have access to a wide variety of products and services, but lack efficient and effective means for daily food management and shopping planning. The present invention aims to automatically manage refrigerator inventory by analyzing receipts and product photos after a user's shopping trip, update inventory in real time based on voice or image input while cooking, and suggest appropriate menus and shopping lists using external flyer information. However, current technology does not offer a system that integrates these functions, resulting in significant labor-intensive tasks during shopping and actual cooking. In particular, product recognition and inventory updates during shopping are performed manually, resulting in inefficiencies and limitations in real-time inventory updates and menu suggestions.

[1735] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1736] In this invention, the server includes means for taking pictures of receipts and products after shopping and analyzing product information from the captured images, means for automatically updating inventory in the refrigerator based on the analyzed product information, means for receiving information on ingredients used in cooking when the user inputs the ingredients by voice or image and updating the inventory in the refrigerator, means for obtaining flyer information from an external website every day, analyzing the information, and registering the product information in a database, means for receiving voice input from the user discussing menus and generating and suggesting appropriate menus and shopping lists based on the current inventory and the obtained flyer information, means for scanning product barcodes when shopping, means for automatically adding product information obtained from the scanned barcodes to the database, and means for creating and updating shopping lists based on the user's voice input. This allows the user to manage refrigerator inventory in real time while shopping or cooking and receive suggestions for appropriate menus and shopping lists.

[1737] A "receipt" is a paper medium that lists information such as the purchased item, the amount, and the date and time of purchase.

[1738] "Product Photos" refers to images of the products purchased by a User.

[1739] "Analysis" refers to extracting necessary data from captured images and input information.

[1740] "Inventory" refers to the food and products stored in the refrigerator.

[1741] "Voice input" is a method in which a user inputs information by voice using a microphone device.

[1742] "Image input" is a method in which a user inputs information by means of an image using a camera device.

[1743] "Flyer information" refers to advertising materials including sale information and product information issued by supermarkets and the like.

[1744] A "barcode" is a series of lines or graphics printed on a product to allow computer-readable information about the product.

[1745] A "database" is a collection of information that stores managed data and allows for rapid searching and updating.

[1746] "Menu" means a list of dishes or meals from which a User can choose.

[1747] A "shopping list" refers to a list of products or ingredients that a user plans to purchase.

[1748] "Scanning" refers to reading information from a barcode or the like.

[1749] "Suggestion" refers to presenting appropriate options or actions to the user.

[1750] "User" refers to a consumer who uses the system to shop and cook.

[1751] A "system" refers to a structure in which a series of devices and programs function in conjunction with one another.

[1752] This invention is a system that manages refrigerator inventory by analyzing receipts and product photos after users have finished shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to users based on supermarket flyer information obtained daily.The system uses hardware and software such as a cloud server, smartphones, smart glasses, and OCR (optical character recognition) technology.

[1753] The main process is as follows:

[1754] Post-shopping inventory management:

[1755] 1. The user takes a photo of the receipt or product with their smartphone and uploads it to the cloud server. This process uses the smartphone's camera and internet connection functions.

[1756] 2. The cloud server analyzes the uploaded image using OCR technology to extract information such as product name and quantity. This OCR technology is used to quickly and accurately digitize product information.

[1757] 3. The extracted product information is recorded in an inventory database on the cloud server, and the refrigerator's inventory is automatically updated.

[1758] Cooking Inventory Update:

[1759] 1. When cooking, the user inputs voice or images. For example, the user inputs "I used milk" into the smartphone.

[1760] 2. The smartphone converts the voice input into text and sends it to a cloud server. The speech recognition module handles this process.

[1761] 3. The cloud server updates the inventory database based on the received information, decrements the quantity of ingredients used, and notifies the user if the ingredients are out of stock.

[1762] Daily flyer updates:

[1763] 1. The cloud server retrieves flyer information as images from supermarket websites at a specified time every day. A web scraper performs this process automatically.

[1764] 2. The server uses OCR technology to analyze the flyer image, extract product information, and obtain data such as product name, price, and validity period.

[1765] 3. The extracted product information is registered in the flyer information database and becomes accessible to users.

[1766] Menu suggestions and shopping list generation:

[1767] 1. A user speaks, "What would you like for dinner tonight?", for example, through smart glasses.

[1768] 2. The smart device converts the voice into text and sends the information to a cloud server.

[1769] 3. The server compares the current inventory database and flyer information database to generate the appropriate menu and shopping list. A generative AI model performs this analysis and recommendation.

[1770] 4. The smart device will give the user voice suggestions, for example, "To make a chicken and broccoli stir-fry, you need to buy broccoli from the supermarket."

[1771] Inventory management when shopping:

[1772] 1. When a user shops in a physical store, they scan the product's barcode with their smartphone. The barcode scanner app then sends the product information to a cloud server.

[1773] 2. The cloud server parses the product information from the scanned barcode and adds it to the inventory database.

[1774] Example of a concrete example and prompt for the generative AI model:

[1775] Examples:

[1776] When a user purchases a lettuce at a physical store and scans the barcode with their smartphone, the cloud server updates the inventory database and records "1 lettuce added."

[1777] When a user asks the smart glasses, "What's for dinner tonight?", the cloud server will suggest "hamburger steak and salad" based on the refrigerator's inventory and store special offers.

[1778] Example prompt sentence:

[1779] When a user says, "What ingredients do I need for a hot pot?", the following answers are generated based on the current refrigerator inventory and special offers:

[1780] The recipe is "Chicken Hot Pot"

[1781] refrigerator inventory

[1782] Special sale information

[1783] This allows users to enjoy an efficient and profitable shopping and cooking experience.

[1784] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1785] Step 1:

[1786] The user takes a photo of a receipt or product with their smartphone. The input is the image of the receipt or product. The output is an image file stored in the smartphone's storage. The device prepares the captured image to be uploaded to the cloud server.

[1787] Step 2:

[1788] The device uploads the images it takes to the cloud server. The input is the image file stored on the smartphone. The output is the image data transferred to the cloud server. The device transmits the data via an internet connection.

[1789] Step 3:

[1790] The server analyzes the received image using OCR technology. The input is the image data transferred to the server, and the output is the extracted product information (product name, quantity, price, etc.). The server uses OCR technology to extract text data from the image.

[1791] Step 4:

[1792] The server records and updates the analyzed product information in the refrigerator's inventory database. The input is the product information extracted using OCR technology. The output is the updated inventory database. The server compares the product information with existing inventory data and updates the database.

[1793] Step 5:

[1794] When cooking, the user speaks the ingredients they will use into their smartphone. The input is the user's voice data. The output is the voice information converted into text. The device then uses a voice recognition module to convert the voice data into text.

[1795] Step 6:

[1796] The device converts voice input into text and sends the information to the server. The input is text data. The output is text information transferred to the cloud server. The device transmits the data via the Internet.

[1797] Step 7:

[1798] The server updates the inventory database based on the received information. The input is text information obtained by speech recognition. The output is the updated inventory database. The server processes the text information to reduce the amount of ingredients used.

[1799] Step 8:

[1800] The server uses a web scraper to retrieve supermarket flyer information at a specified time every day. The input is the website URL. The output is the retrieved flyer image. The server automatically retrieves flyer information using the web scraper.

[1801] Step 9:

[1802] The server uses OCR technology to analyze the acquired flyer image and extract product information. The input is the flyer image. The output is the analyzed product information (product name, price, validity period, etc.). The server uses OCR technology to extract text data from the flyer image.

[1803] Step 10:

[1804] The server registers the extracted product information in a flyer information database. The input is the product information extracted using OCR technology. The output is an updated flyer information database. The server stores the product information in the database.

[1805] Step 11:

[1806] The user speaks to the smart glasses, saying, "What would you like for dinner tonight?" The input is the user's voice data. The output is the voice information converted into text. The device uses a voice recognition module to convert the voice data into text.

[1807] Step 12:

[1808] The device converts voice input into text and sends the information to the server. The input is text data. The output is text information sent to the cloud server. The device sends the data over the Internet.

[1809] Step 13:

[1810] The server compares the current inventory database and flyer information database and generates the appropriate menu and shopping list. The input is the inventory database and flyer information database. The output is the generated menu and shopping list. The server generates the menu and list using a generative AI model.

[1811] Step 14:

[1812] The device makes audio suggestions to the user. The input is the generated menu and shopping list. The output is audio suggestions. The device uses a speaker to provide information to the user.

[1813] Step 15:

[1814] A user scans a product barcode with their smartphone in a physical store. The input is the product barcode data. The output is the barcode information stored on the smartphone. The device uses the camera function and a barcode scanner app.

[1815] Step 16:

[1816] The terminal sends the product information obtained by barcode scanning to the cloud server. The input is the barcode information. The output is the product information sent to the cloud server. The terminal sends the data via the Internet.

[1817] Step 17:

[1818] The server adds and updates the received product information to the inventory database. The input is the product information obtained by barcode scanning. The output is the updated inventory database. The server stores the product information in the database.

[1819] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1820] This system manages refrigerator inventory by analyzing receipts and product photos taken by the user after shopping, updates inventory while receiving voice or image input when cooking, and suggests appropriate menus and shopping lists to the user based on supermarket flyer information acquired daily.The system also combines an emotion engine that recognizes the user's emotions using voice input and image analysis technology and adjusts the suggestions based on those emotions.

[1821] Program processing overview

[1822] Inventory management using shopping photos and receipts

[1823] 1. The user takes a photo of the receipt or product with their smartphone.

[1824] 2. Upload the photos taken by the device (smartphone) to the cloud server.

[1825] 3. The server analyzes the received image using optical character recognition (OCR) technology and extracts product information (product name, quantity, price, etc.).

[1826] 4. The server records and updates the analyzed product information in the refrigerator's inventory database.

[1827] Control delivery by voice or image input during cooking

[1828] 1. When the user prepares a dish, they tell the system the ingredients they will use using voice or images.

[1829] 2. The device converts the voice input into text and sends the information to the server. If the input is an image, the image is sent to the server.

[1830] 3. The server updates the refrigerator's inventory database based on the received information, decrementing the quantity of ingredients used and notifying the user if the ingredient is out of stock.

[1831] Daily flyer information acquisition and analysis

[1832] 1. The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[1833] 2. The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[1834] 3. The extracted product information is registered in the flyer information database.

[1835] Recognizing user emotions with an emotion engine

[1836] 1. Recognize emotions through voice input or camera when users consult the menu.

[1837] 2. The device converts the voice input into text and sends the information to the server. At the same time, the emotion engine analyzes the user's emotions from their tone of voice and facial expressions.

[1838] 3. The server adjusts the menu suggestions and shopping list contents based on the emotional information received.

[1839] Voice and emotion menu consultation and suggestions

[1840] 1. The user asks the system verbally, "What would you like for dinner tonight?"

[1841] 2. The device converts the voice input into text and sends the information to the server. At the same time, the emotion engine recognizes the user's emotions.

[1842] 3. The server compares the current refrigerator inventory database, flyer information database, and emotion information to generate an appropriate menu.

[1843] 4. The server creates a shopping list for ingredients that are in short supply and adjusts its suggestions based on feedback from the emotion engine.

[1844] 5. The device will then output a suggestion to the user, such as, "You seem a little tired today. How about a quick chicken and broccoli stir-fry?"

[1845] Specific examples

[1846] Specific examples of inventory management

[1847] A user purchases milk at a supermarket and takes a photo of the receipt with their smartphone. The device uploads the photo to the server, which uses OCR technology to extract the product information (1 liter of milk) and updates the refrigerator's inventory database with the entry "Add 1 liter of milk."

[1848] Specific examples of inventory management

[1849] While cooking, the user says "I used milk" by voice input. The device converts the voice to text and sends it to the server as "I used 1 liter of milk." The server updates the inventory database and records it as "I reduced 1 liter of milk."

[1850] Examples of flyer information

[1851] At 9:00 AM, the server retrieves the flyer image from the supermarket's website, analyzes the product information "Broccoli 98 yen," and registers it in the flyer information database. It saves it as "Broccoli 98 yen, sale period: 3 days."

[1852] Examples of emotion engines

[1853] The user asks, "What should we have for dinner tonight?" The emotion engine determines from the user's tone of voice and facial expression that they seem tired. The device converts this speech into text and sends it to the server. Based on the emotion engine's feedback, the server compares the refrigerator's inventory with flyer information and suggests, "You seem a little tired today, how about a simple chicken and broccoli stir-fry?"

[1854] This system provides a multifunctional mechanism that includes these steps and can support users in daily food management, menu planning, and shopping planning, including emotional information.

[1855] The processing flow will be explained below.

[1856] Manage refrigerator inventory using shopping photos and receipts

[1857] Step 1:

[1858] The user takes a photo of the receipt or product with their smartphone.

[1859] Open the camera app on your smartphone and take a photo of your receipt or purchased items.

[1860] Step 2:

[1861] The device uploads the photos it takes to the server.

[1862] Use the dedicated application and press the button to upload the photo to the cloud server.

[1863] Step 3:

[1864] The server applies OCR technology to analyze the received image.

[1865] The server invokes an image processing module to extract text information from the image.

[1866] Step 4:

[1867] The server extracts the parsed product information (product name, quantity, price, etc.).

[1868] Product name, quantity, and price are identified from the text information and converted into a database format.

[1869] Step 5:

[1870] The server records the extracted product information in the refrigerator's inventory database and updates it.

[1871] Adds the product information to the corresponding user's inventory database and updates existing inventory information.

[1872] Control delivery by voice or image input during cooking

[1873] Step 1:

[1874] When cooking, the user verbally tells the system what ingredients to use.

[1875] Use your smartphone's microphone to input voice instructions such as "I used chicken."

[1876] Step 2:

[1877] The device converts the voice input into text and sends the information to the server.

[1878] Uses voice recognition software to convert speech into text.

[1879] Step 3:

[1880] The server updates the refrigerator inventory database based on the received information.

[1881] The quantity of the specified ingredient is reduced and the inventory database is updated.

[1882] Step 4:

[1883] If the server is out of stock, the user is notified.

[1884] If the stock reaches 0, a notification message will be sent to the user.

[1885] Daily flyer information acquisition and analysis

[1886] Step 1:

[1887] The server retrieves flyer information as images from each supermarket's website at a specified time every day.

[1888] Based on a fixed schedule, it automatically visits each supermarket's website and downloads flyer images.

[1889] Step 2:

[1890] The server analyzes the flyer image it has acquired and extracts product information (product name, price, validity period, etc.).

[1891] Use the flyer image processing module to extract product information from the image.

[1892] Step 3:

[1893] The extracted product information is registered in a flyer information database.

[1894] The flyer information is stored in a database based on the product name, price, and validity period.

[1895] Recognizing user emotions with an emotion engine

[1896] Step 1:

[1897] When the user consults the menu, he or she uses voice input.

[1898] Using the smartphone's microphone, voice menu consultation is performed, asking, "What would you like for dinner tonight?"

[1899] Step 2:

[1900] The device converts the voice input into text and sends the information to the server.

[1901] Use speech recognition software to convert speech to text.

[1902] Step 3:

[1903] The emotion engine analyzes emotions from the user's tone of voice.

[1904] A voic...

Claims

1. A method for taking photos of receipts and products after shopping and analyzing product information from the images taken; A means for automatically updating the inventory in the refrigerator based on the analyzed product information; a means for receiving information on ingredients used in cooking by voice input or image input by a user and updating the inventory in the refrigerator; A method to obtain flyer information from external websites every day, analyze it, and register product information in a database. A means for receiving a voice input from a user requesting a menu, and generating and suggesting an appropriate menu and shopping list based on current inventory and acquired flyer information; A system including:

2. 10. The system of claim 1, wherein an optical character recognition device is used to analyze product information from the image.

3. 10. The system of claim 1, wherein a voice recognition device is used to convert a user's voice input into text and update the inventory database accordingly.

Images