System

Abstract

A system is provided.SOLUTION: A system comprising: imaging means for imaging an article in a food preserving apparatus; recognizing means for analyzing image data obtained by the imaging means and recognizing identification information and a preservation limit of the article; proposing means for proposing an optimal use method related to the article based on the identification information and the preservation limit; acquiring means for acquiring bargain information of a seller near a user from an external database; and notifying means for notifying the user of the bargain information.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In today's busy lifestyles, efficient food management at home is an important issue. However, it is not easy to manage multiple foods in the refrigerator and keep track of expiration dates. Furthermore, there is a lack of support for reducing food waste and planning cooking. Furthermore, there is a need for a means to easily obtain information about sales at nearby supermarkets and use it to plan purchases during everyday shopping. The purpose of this invention is to solve these problems. [Means for solving the problem]

[0005] The present invention solves these problems by providing a system that includes an imaging means for imaging items in a food storage device, a recognition means for analyzing image data obtained by the imaging means and recognizing the identification information and shelf life of the items, a suggestion means for suggesting the optimal method of use related to the items based on the identification information and shelf life, a transmission means for transmitting the recognized item information to an external database, an acquisition means for acquiring, from the external database, sales information from retailers near the user, and a notification means for notifying the user of the sales information acquired by the acquisition means.

[0006] The "imaging means" refers to a device such as a camera or sensor for taking pictures of items in the food storage device.

[0007] The "recognition means" refers to a processing device or software that analyzes image data obtained by the imaging means and identifies the identification information and storage period of the item.

[0008] The "suggestion means" is a device or software that suggests the optimum method of use or recipe based on the identification information and shelf life of the item identified by the recognition means.

[0009] The "transmitting means" is a communication device for transmitting the recognized product information to an external database or server.

[0010] The "acquisition means" refers to a device or software for acquiring sale information of retailers in the user's vicinity from an external database or server.

[0011] The "notification means" is a device or software for notifying the user of acquired sale information and suggested recipes.

[0012] A "food storage device" is a household or commercial device for storing food, such as a refrigerator or freezer.

[0013] "Items" refers to food and ingredients stored in a food storage device.

[0014] "Identification information" is information for identifying the name and type of an item.

[0015] The "shelf life" is information that indicates the date by which an item can be consumed in the appropriate condition.

[0016] An "external database" is a database that exists on the Internet or cloud and is used to manage food and sale information, etc.

[0017] A "retailer" is a business that sells products, such as a supermarket or retail store near the user. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is a system that efficiently manages the items stored in a food storage device (e.g., a refrigerator) and provides users with information on optimal usage methods and nearby special sales based on the managed items. This system includes three elements: a "terminal" that captures images of the items stored in the refrigerator, a "server" that analyzes the captured image data and identifies the items and their expiration dates, and a "user" that notifies the user based on the identified information.

[0040] Basic system configuration

[0041] 1. Imaging means (terminal)

[0042] The terminal is installed inside the refrigerator and uses a 360-degree camera to capture images of the items inside the refrigerator.

[0043] Image data of the item is acquired and temporarily stored in local memory.

[0044] 2. Recognition means (server)

[0045] It receives image data sent from the terminal and uses an object recognition algorithm to analyze the item's identification information (name, type) and label information.

[0046] Optical character recognition (OCR) technology is used to recognize the expiration date printed on the product label.

[0047] 3. Proposed method (server)

[0048] Based on the recognized identity and shelf life, the system suggests the best way to use the item (e.g., a recipe).

[0049] Recipe selection also takes into account the user's food preferences and past history.

[0050] 4. Transmission means (terminal)

[0051] The recognized item information is transmitted from the terminal to the server.

[0052] The data is packaged in JSON format and sent over Wi-Fi.

[0053] 5. Acquisition method (server)

[0054] Get special offers from vendors near the user from an external database.

[0055] The acquired sale information is sorted and information relevant to the user is identified.

[0056] 6. Notification Method (Server)

[0057] The acquired sale information and suggested recipes are notified to the user's smartphone app.

[0058] Use push notifications to provide users with real-time information.

[0059] Specific processing of the program

[0060] Scan food in the refrigerator

[0061] The device automatically enters scanning mode when the refrigerator door is closed, activating its 360-degree camera to scan each shelf inside the refrigerator. The image data captured by the camera is analyzed using an object recognition algorithm to identify the location and appearance of each food item. OCR technology is used to read the food label information (name, expiration date). This information is stored in local memory and later sent to the server.

[0062] Scan data transmission and analysis

[0063] The device sends the collected information about ingredients to a server via Wi-Fi. The server analyzes the received data and stores it in a database. The names, expiration dates, and location information of ingredients are organized by category, allowing the user to understand the status of their refrigerator.

[0064] Optimal recipe suggestions

[0065] The server then suggests optimal recipes to the user based on the stored ingredient information. The server then references the user's food preferences and past history to search for and generate a list of relevant recipes. This recipe list is then sent to the user's smartphone app via push notification.

[0066] Get information on nearby sales

[0067] The server retrieves sales information from nearby retailers from an external database based on the user's location information. The retrieved sales information is matched with the information about ingredients in the user's refrigerator to identify related sales information. This sales information is also notified to the user's smartphone app.

[0068] Specific examples

[0069] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. It determines that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023, and sends this information to the server. The server then confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. Along with this suggestion, the user is also notified of a 20% off sale on tomatoes at a nearby supermarket. The user can check this information on their smartphone app and plan their cooking and shopping accordingly.

[0070] The above is a description of specific embodiments for carrying out the present invention.

[0071] The processing flow will be explained below.

[0072] Step 1:

[0073] The device automatically enters scanning mode when the refrigerator door is closed, activating its built-in 360-degree camera to capture images of each shelf inside the refrigerator.

[0074] Step 2:

[0075] The device analyzes the image data captured by the camera using an object recognition algorithm, which identifies the location and type of each food item in the refrigerator.

[0076] Step 3:

[0077] The device uses OCR (optical character recognition) technology to read the label information (name, expiration date) of each ingredient, and this information is temporarily stored in the device's local memory.

[0078] Step 4:

[0079] The device sends the acquired information to a server via Wi-Fi. The data is packaged in JSON format and encrypted.

[0080] Step 5:

[0081] The server analyzes the received data and stores it in a database, organizing the names of ingredients, expiration dates, and location information by category.

[0082] Step 6:

[0083] The server periodically scans the database for ingredients that are nearing their expiration date and generates an alert for any ingredients that are nearing their expiration date.

[0084] Step 7:

[0085] The server refers to the user's preferences and past history, suggests recipes using ingredients that are close to their expiration date, searches for relevant recipes in the recipe database, and generates a list.

[0086] Step 8:

[0087] The server notifies the user of the generated recipe list and alert information via the user's smartphone app using the push notification function.

[0088] Step 9:

[0089] Users receive notifications on their smartphone app, which can include audio alerts and banners.

[0090] Step 10:

[0091] The user opens the app to check the ingredients in the refrigerator and see the suggested recipes. The app displays a list of ingredients, expiration dates, and suggested recipes.

[0092] Step 11:

[0093] The server retrieves sales information from nearby retailers from an external database based on the user's location information, and obtains the latest sales data through the supermarket's API.

[0094] Step 12:

[0095] The server analyzes the acquired sales information and identifies items that match the food information in the user's refrigerator. The identified sales information is organized and added to a recommendation list.

[0096] Step 13:

[0097] The server sends the recommendation list to the user's smartphone app, where the user can check the sale information through notifications.

[0098] Step 14:

[0099] Users can check the sale information within the app, which includes store addresses, discount rates, product details, and more.

[0100] Step 15:

[0101] Users can add ingredients they need to a list and create a shopping list, which can be edited and saved within the app.

[0102] The above is the processing flow of the specific program of the system. The specific operation has been explained in detail for each step.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] In modern households, food waste is a problem due to the inability to properly manage items stored in food storage devices. Other issues include overlooking items that are approaching their expiration date and taking time to find the perfect recipe. Furthermore, to efficiently do daily shopping, it is necessary to quickly obtain information about sales from nearby retailers, but current systems are not able to adequately handle this.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes an imaging means for capturing images of items in the food preservation device, a recognition means for analyzing the image data captured by the imaging means to recognize the identification information and shelf life of the items, a suggestion means for suggesting optimal usage methods for the items based on the identification information and shelf life, a transmission means for transmitting the recognized item information to an external database, a acquisition means for acquiring sales information from retailers near the user from the external database, a notification means for notifying the user of the sales information acquired by the acquisition means, a means for the imaging means to scan the interior of the refrigerator with a 360-degree camera triggered by a door sensor, and a means for the notification means to provide information to the user's mobile information terminal using a push notification function. This enables expiration date management of items in the food preservation device, suggestions for optimal usage methods, and efficient shopping.

[0108] A "food preservation device" is a device for preserving food at low temperatures, and mainly refers to a refrigerator or freezer.

[0109] "Imaging means" refers to a device for photographing items in the food storage device, and includes a camera and a sensor.

[0110] The "recognition means" refers to a technique or device for analyzing image data obtained by the imaging means and recognizing the identification information and shelf life of an item.

[0111] The "suggestion means" is a function or device for suggesting the most suitable method of use related to the item based on the identification information and the expiration date obtained by the recognition means.

[0112] The "transmitting means" refers to a function or device for transmitting the recognized product information to an external database.

[0113] The "acquisition means" refers to a function or device for acquiring sale information from an external database.

[0114] The "notification means" is a function or device for notifying the user of the acquired sale information.

[0115] A "door sensor" is a sensor that detects whether the refrigerator door is open or closed.

[0116] A "360-degree camera" is a camera that can capture images in all directions.

[0117] "Mobile information terminal" refers to a portable information processing device such as a mobile phone, smartphone, or tablet.

[0118] The "push notification function" is a function that delivers information from a server to a user's mobile information terminal in real time.

[0119] A "food management system" is a system that efficiently manages the items in a refrigerator and provides optimal usage methods and special sale information based on that information.

[0120] This system efficiently manages the items stored in a food storage device and provides users with information on optimal usage methods and nearby special sales based on that information. This system includes three elements: a terminal that captures images of the items stored in the refrigerator, a server that analyzes the captured image data and identifies the items and their expiration dates, and a user interface that notifies the user based on the identified information.

[0121] Specifically, the following hardware and software are used:

[0122] Imaging means (terminal)

[0123] The device is installed inside the refrigerator and captures images of the items inside using a 360-degree camera. Image data of the items is acquired and temporarily stored in local memory. When the door sensor detects that the refrigerator door is closed, the 360-degree camera is activated and begins scanning.

[0124] Recognition means (server)

[0125] The server receives the image data sent from the device and analyzes the item's identification information (name, type) and label information using an object recognition algorithm. Specifically, it uses a TensorFlow-based object recognition model. It also uses Tesseract as OCR technology to recognize the expiration date printed on the item's label.

[0126] Proposal method (server)

[0127] The server then suggests optimal ways to use the item (e.g., recipes) based on the recognized identification information and expiration date. The server also takes into account the user's food preferences and past history when selecting recipes. Based on the stored ingredient information, the server selects and lists the optimal recipes.

[0128] Transmission means (terminal)

[0129] The device sends the recognized item information to the server, packaged in JSON format, and transmitted via Wi-Fi.

[0130] Acquisition method (server)

[0131] The server retrieves sales information from nearby retailers from an external database, sorts the sales information, and identifies those relevant to the user. It then uses GeoAPI to obtain the user's location and sends an API request to the external database based on that location.

[0132] Notification method (server)

[0133] The server then notifies the user of the sale information and suggested recipes in real time via the smartphone app using Firebase Cloud Messaging (FCM).

[0134] Specific examples

[0135] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. It determines that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023, and sends this information to the server. The server then confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. Along with this suggestion, the user is also notified of a 20% off sale on tomatoes at a nearby supermarket. The user can check this information on their smartphone app and plan their cooking and shopping accordingly.

[0136] Example prompts for generative AI models

[0137] Here is an example prompt for the generative AI model to write specific steps for this system:

[0138] Please explain in detail the following system steps, including examples:

[0139] This system efficiently manages the contents of a food storage device (refrigerator) and provides users with information on optimal use and nearby special offers. It works by following these steps:

[0140] 1. Initiating a refrigerator scan: The device detects that the refrigerator door is closed, and the 360-degree camera activates to capture images of each shelf.

[0141] 2. Acquisition and temporary storage of image data: The image data captured by the camera is acquired and temporarily stored in local memory.

[0142] 3. Sending image data: The device sends the acquired image data to the server via Wi-Fi.

[0143] 4. Image data analysis: The server analyzes the received image data and extracts ingredient information.

[0144] 5. Storing and organizing ingredient information: The server stores and organizes ingredient information in a database.

[0145] 6. Optimal recipe suggestion: The server suggests optimal recipes to the user based on the stored ingredient information.

[0146] 7. Get nearby sales information: The server retrieves nearby sales information from an external database and selects the most relevant ones.

[0147] 8. User notification: The server pushes suggested recipes and special offers to the user's smartphone app.

[0148] For example, imagine you have milk, tomatoes, and carrots in your refrigerator. Your device scans them and sends that information to a server. The server sees that the tomatoes are nearing their expiration date and sends you a recipe for cream of tomato soup and a sale on tomatoes.

[0149] The above is a specific embodiment for carrying out the invention.

[0150] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0151] Step 1: Start the fridge scan

[0152] When the door sensor detects that the refrigerator door is closed, the device automatically enters scanning mode. The 360-degree camera starts up and takes pictures of each shelf in turn. The input is the sensor signal indicating that the door is closed, and the output is the image data captured by the camera.

[0153] Specific operation: After the door sensor detects the door closing, the camera waits 5 seconds and then starts shooting in all directions, scanning each shelf 10 degrees at a time, capturing a total of 36 image data.

[0154] Step 2: Acquire and temporarily save image data

[0155] The image data captured by the camera is acquired and temporarily stored in local memory. The input is image data taken by a 360-degree camera, and the output is a high-resolution JPEG image file stored in local memory.

[0156] How it works: Each scanned image is saved to local memory in real time and stored in a temporary storage area. The image is saved in JPEG format and a timestamp is added to each image.

[0157] Step 3: Sending image data

[0158] The device sends image data stored in its local memory to the server via Wi-Fi. The data is packaged in JSON format and sent as high-priority data. The input is the image data in the local memory, and the output is the JSON package sent to the server.

[0159] Specific behavior: The device checks for a Wi-Fi connection, sends data in 5MB chunks, and clears local memory after the transmission is complete.

[0160] Step 4: Analyzing the image data

[0161] The server analyzes the received image data and uses an object recognition algorithm to extract the identification information (name, type) of each ingredient. It also uses OCR technology to read the expiration date printed on the ingredient label. The input is a JSON package sent from the device, and the output is data containing the identification information and expiration date.

[0162] Specific operation: The server uses a TensorFlow-based object recognition model and only recognizes results with a confidence level of 0.9 or higher. OCR analysis is performed using the Tesseract library, and label data is extracted as text data.

[0163] Step 5: Save and organize ingredient information

[0164] The server stores the recognition results in a database and organizes the names, types, shelf life, and location information of ingredients by category. The input is data including identification information and shelf life, and the output is a database organized by category.

[0165] How it works: Recognition results are stored in a MySQL database and immediately indexed. Each item is categorized for efficient searching.

[0166] Step 6: Propose the best recipe

[0167] The server compares the stored ingredient information with the user's preferences and past history to suggest optimal recipes. The input is an organized database and the user's preferences and history data, and the output is a list of recommended recipes.

[0168] Specific operation: The server refers to the user profile and uses a matching algorithm to select three candidate recipes, generate a list of recommended recipes, and send it to the next notification step.

[0169] Step 7: Get local sales information

[0170] The server retrieves nearby sales information from an external retailer database based on the user's location information. The input is the user's location information and sale information from the external database, and the output is highly relevant sale information.

[0171] How it works: The server uses GeoAPI to obtain the user's location information and sends an API request to an external database based on that information. The obtained sale information is matched with food information, and the most relevant information is selected.

[0172] Step 8: Notify users

[0173] The server sends suggested recipes and sales information to the user's mobile device using push notifications. The input is a list of suggested recipes and related sales information, and the output is the notification sent to the user's mobile device.

[0174] How it works: The server uses Firebase Cloud Messaging (FCM) to send real-time notifications to users' smartphones, including detailed recipe instructions and special offers, designed to allow users to take immediate action.

[0175] (Application example 1)

[0176] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0177] Conventional food preservation devices have the problem that it is difficult to manage the inventory of food in the refrigerator and grasp the expiration date, resulting in a lot of food going to waste. Furthermore, when users shop in physical stores, they are unable to create an optimal purchasing plan that is linked to existing inventory, resulting in duplicate or unnecessary purchases. This leads to problems of wasting resources and increasing the burden on household finances.

[0178] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0179] In this invention, the server includes an imaging means for imaging items in the food preservation device, a recognition means for analyzing image data obtained by the imaging means to recognize the identification information and shelf life of the items, a suggestion means for suggesting optimal usage methods for the items based on the identification information and shelf life, a transmission means for transmitting the recognized item information to an external database, an acquisition means for acquiring sales information from retailers near the user from the external database, a notification means for notifying the user of the sales information acquired by the acquisition means, and a shopping support means for providing the recognized item information and sales information and suggesting an optimal purchasing plan when the user is shopping at a physical store. This allows the user to shop efficiently while referring to inventory information in the refrigerator in real time.

[0180] A "food preservation device" is a device used to maintain the quality of food and preserve it for a long period of time.

[0181] "Goods" refers to food and beverages stored in the refrigerator.

[0182] The "imaging means" is a device including a camera and a sensor for acquiring image data of an object.

[0183] "Image data" refers to visual information acquired by an imaging means expressed in digital form.

[0184] The "recognition means" is a device or system that analyzes the acquired image data and extracts the identification information and storage period of the item.

[0185] "Identification information" refers to information for identifying the name, type, characteristics, etc. of an item.

[0186] "Shelf life" is information that indicates the date by which an item can be consumed.

[0187] The "suggestion means" is a device or system that suggests to the user the most appropriate way to use the product based on the identification information and the shelf life.

[0188] The "transmitting means" is a device or system for transmitting the recognized product information to an external database.

[0189] "External database" refers to a database that stores and manages product information and special sale information and provides necessary information.

[0190] The "acquisition means" is a device or system for acquiring sale information of retailers in the user's neighborhood from an external database.

[0191] The "notification means" is a device or system for notifying the user of the acquired sale information and suggested usage methods.

[0192] A "shopping support means" is a device or system that provides recognized product information and sale information to a user when shopping at a physical store, and suggests an optimal purchasing plan.

[0193] "Brick and mortar store" refers to a store that sells products in person at a physical location.

[0194] The present invention is a system that efficiently manages the items in a food storage device and provides users with optimal usage instructions and nearby sales information based on the managed items. This system is composed of the following elements:

[0195] 1. Imaging means (terminal)

[0196] The device is installed inside the refrigerator and captures images of the items inside using a 360-degree camera. The image data captured by the camera is temporarily stored in local memory.

[0197] The specific hardware used is a 360-degree camera (e.g., Ricoh Theta) and a sensor module (e.g., Raspberry Pi).

[0198] 2. Recognition means (server)

[0199] The server receives the image data sent from the terminal and analyzes the identification information and storage period of the item.

[0200] Image recognition uses object recognition algorithms (e.g., YOLO) and optical character recognition (OCR) technology (e.g., Tesseract OCR) to identify the item's identity and shelf life.

[0201] 3. Proposed method (server)

[0202] Based on the recognized product information, the server suggests optimal usage methods, such as cooking recipes, to the user.

[0203] The system selects appropriate recipes based on the user's food preferences and past usage history.

[0204] 4. Transmission means (terminal)

[0205] The device packages the recognized item information in JSON format and sends it to the server via Wi-Fi.

[0206] 5. Acquisition method (server)

[0207] The server retrieves sales information from an external database for merchants in the user's vicinity.

[0208] The sale information is matched with information about items whose expiration dates are approaching for the user.

[0209] 6. Notification Method (Server)

[0210] The server then notifies the user of the acquired sale information and suggested recipes via push notification to their smartphone.

[0211] 7. Shopping Support Methods (User)

[0212] When users shop at a physical store, the smartphone app displays food items with upcoming expiration dates, inventory information, and special offers.

[0213] This allows users to shop efficiently and reduce waste.

[0214] Specific operation example

[0215] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. It determines that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023, and sends this information to the server. The server then confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. Along with this suggestion, the user is also notified of a 20% off sale on tomatoes at a nearby supermarket. The user can check this information on their smartphone app and plan their cooking and shopping accordingly.

[0216] Prompt Sentence Examples

[0217] "The system analyzes the image of food inventory in the refrigerator and identifies foods that are nearing their expiration date. While the user is shopping in a physical store, it takes into account the refrigerator's inventory and sales information and suggests the optimal shopping list in real time. It also includes recipes based on ingredients that are nearing their expiration date."

[0218] The above is a description of specific embodiments for carrying out the present invention.

[0219] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0220] Step 1:

[0221] The device starts scanning the inside of the refrigerator. When the refrigerator door is closed, the 360-degree camera automatically starts up and scans the interior. The image data acquired by the image capture is temporarily stored in local memory.

[0222] Input: Current state of the refrigerator

[0223] Data processing: 360-degree camera imaging

[0224] Output: 360-degree image data of the inside of the refrigerator

[0225] Step 2:

[0226] The device sends the scanned data to the server, and the image data stored in the local memory is packaged in JSON format and sent to the cloud server via Wi-Fi.

[0227] Input: 360-degree image data

[0228] Data processing: Packaging into JSON format

[0229] Output: Image data sent to the server

[0230] Step 3:

[0231] The server analyzes the received data. The cloud server analyzes the received image data and identifies the item's identity using an object recognition algorithm (e.g., YOLO). It also uses OCR technology (e.g., Tesseract OCR) to extract label information (name, expiration date), and stores this information in a database.

[0232] Input: Image data sent to the server

[0233] Data processing: object recognition, OCR analysis, saving to database

[0234] Output: Identification information and retention period

[0235] Step 4:

[0236] The server proposes the optimal way to use the product. Based on the recognized identification information and expiration date, the server proposes the optimal way to use the product, such as cooking recipes, to the user. The server selects recipes taking into consideration the user's food preferences and past history.

[0237] Input: Identification information, retention period, user preferences and history

[0238] Data processing: Recipe selection

[0239] Output: Suggested recipe information

[0240] Step 5:

[0241] The device receives the suggested recipe information from the server and notifies the user's smartphone. The suggested recipe information is sent to the user's smartphone via push notification.

[0242] Input: Suggested recipe information

[0243] Data processing: Notification to smartphone

[0244] Output: Recipe notification to smartphone

[0245] Step 6:

[0246] The server retrieves sale information from an external database, and based on the user's location information, retrieves sale information from nearby retailers and matches it with items with an approaching expiration date.

[0247] Input: User's location

[0248] Data processing: Acquiring and matching sale information

[0249] Output: Related deals

[0250] Step 7:

[0251] The server notifies the user of the special sale information it has acquired, and displays the information on the smartphone so that the user can refer to it when shopping at a physical store.

[0252] Input: Related sales information

[0253] Data processing: Notification to smartphone

[0254] Output: Special sale information notification to smartphone

[0255] Step 8:

[0256] Users can use a smartphone app to receive purchasing assistance in physical stores. The app displays items with upcoming expiration dates and sales information, and suggests optimal purchasing plans. This allows users to shop efficiently.

[0257] Input: Information notified to your smartphone

[0258] Data processing: Displaying expiration dates and special sale information, suggesting purchasing plans

[0259] Output: Purchase support information for the user

[0260] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0261] This invention is a system that efficiently manages items in a food storage device (e.g., a refrigerator) and provides users with optimal usage methods and nearby sale information that also take into account the user's emotions. This system includes three elements: a "terminal" that captures images of items in the refrigerator, a "server" that analyzes the image data and the user's emotions and makes suggestions based on the item's identification information, shelf life, and user emotions, and a "user" that notifies the user of the information.

[0262] Basic system configuration

[0263] 1. Imaging means (terminal)

[0264] The terminal is installed inside the refrigerator and uses a 360-degree camera to capture images of the items inside the refrigerator.

[0265] Image data of the item is acquired and temporarily stored in local memory.

[0266] 2. Recognition means (server)

[0267] It receives image data sent from the terminal and uses an object recognition algorithm to analyze the item's identification information (name, type) and label information.

[0268] OCR (optical character recognition) technology is used to recognize the expiration date printed on the product label.

[0269] 3. Proposed method (server)

[0270] Based on the recognized identity and shelf life, the system suggests the best way to use the item (e.g., a recipe).

[0271] The recipe selection incorporates an emotion engine that reflects the user's emotions, and by analyzing the user's current emotional state, it makes more personalized suggestions.

[0272] 4. Emotion engine (server)

[0273] It includes sensors that detect emotions from the user's voice input and facial expressions, and analyzes the information acquired by these sensors to identify the user's emotional state.

[0274] Based on the identified emotion information, a suggestion suited to the emotion of the user is made.

[0275] 5. Transmission means (terminal)

[0276] The recognized item information is transmitted from the terminal to the server.

[0277] The data is packaged in JSON format and sent over Wi-Fi.

[0278] 6. Acquisition method (server)

[0279] Get special offers from vendors near the user from an external database.

[0280] The acquired sale information is sorted and information relevant to the user is identified.

[0281] 7. Notification Method (Server)

[0282] The acquired sale information and suggested recipes are notified to the user's smartphone app.

[0283] Use push notifications to provide users with real-time information.

[0284] Specific processing of the program

[0285] Scan food in the refrigerator

[0286] The device automatically enters scanning mode when the refrigerator door is closed, activating its 360-degree camera to scan each shelf inside the refrigerator. The image data captured by the camera is analyzed using an object recognition algorithm to identify the location and appearance of each food item. OCR technology is used to read the food label information (name, expiration date). This information is stored in local memory and later sent to the server.

[0287] Scan data transmission and analysis

[0288] The device sends the collected information about ingredients to a server via Wi-Fi. The server analyzes the received data and stores it in a database. The names, expiration dates, and location information of ingredients are organized by category, allowing the user to understand the status of their refrigerator.

[0289] Optimal recipe suggestions

[0290] The server suggests optimal recipes to the user based on the stored ingredient information. Furthermore, it uses an emotion engine to select recipes that take into account the user's current emotional state. The server analyzes the user's voice input and facial recognition information to identify the user's emotional state. For example, if the user is feeling stressed, it suggests easy-to-make recipes or dishes that will help them relax.

[0291] Example: Emotion-based recipe suggestions

[0292] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. The data sent to the server includes the expiration date of the milk (May 1, 2023) and the expiration date of the tomatoes (April 28, 2023).

[0293] The server determines that the tomatoes are close to expiring and suggests a recipe for tomato cream soup. However, the emotion engine then detects that the user is feeling stressed from their voice and facial expressions. Based on this information, the server also suggests a recipe for a relaxing herbal soup that will help relieve stress.

[0294] Get information on nearby sales

[0295] The server retrieves sales information from nearby retailers from an external database based on the user's location information. It then sorts the retrieved information and identifies sales information relevant to the user. This sales information is also notified to the user's smartphone app.

[0296] Example: Notifications with sentiment and sales information

[0297] The smartphone app notifies users of a 20% off sale on tomatoes at a nearby supermarket along with recipes based on their emotions (e.g., herb soup). Because the app responds to the user's emotional state, it also provides suggestions for relaxation along with the recipe. Utilizing this information, users can plan their cooking and shopping accordingly.

[0298] The above is a description of a specific embodiment for carrying out the present invention. This system takes into account the user's emotions and makes more personalized suggestions, thereby improving the convenience of ingredient management and cooking.

[0299] The processing flow will be explained below.

[0300] Step 1:

[0301] The device automatically enters scanning mode when the refrigerator door is closed, activating the 360-degree camera and capturing images of each shelf inside the refrigerator.

[0302] Step 2:

[0303] The device temporarily stores image data captured by the camera in its local memory. The captured data includes all items in the refrigerator.

[0304] Step 3:

[0305] The device analyzes the image data and uses object recognition algorithms to identify the location and type of each item, thereby determining what is stored in the refrigerator.

[0306] Step 4:

[0307] The terminal uses OCR (optical character recognition) technology to read the label information (name, expiration date) of each item and stores this information in local memory.

[0308] Step 5:

[0309] The device sends the acquired ingredient information to a server via Wi-Fi, where the data is packaged in JSON format and encrypted.

[0310] Step 6:

[0311] The server analyzes the received data and stores it in a database. It organizes the names of ingredients, expiration dates, and location information by category, and keeps track of the status of the user's refrigerator.

[0312] Step 7:

[0313] The server periodically scans the database for ingredients that are nearing their expiration date and generates an alert for any ingredients that are nearing their expiration date.

[0314] Step 8:

[0315] The server activates the emotion engine and uses sensors to capture the user's voice input and facial expressions, thereby detecting the user's current emotional state.

[0316] Step 9:

[0317] The server selects recipe suggestions based on the user's emotional state. For example, if the user is feeling stressed, it will suggest recipes with a relaxing effect.

[0318] Step 10:

[0319] The server generates an optimal recipe list based on the stored ingredient information and emotion information, including suggestions based on ingredient usage and emotion.

[0320] Step 11:

[0321] The server notifies the user's smartphone app of the generated recipe list and alert information using the push notification function.

[0322] Step 12:

[0323] Users receive a notification via a smartphone app that displays a list of ingredients, expiration dates, and suggested recipes based on their emotions.

[0324] Step 13:

[0325] The server retrieves sales information from nearby retailers from an external database based on the user's location information, and obtains the latest sales data through the supermarket's API.

[0326] Step 14:

[0327] The server analyzes the sales information and identifies items that match the food information in the user's refrigerator. The sales information is organized into items relevant to the user.

[0328] Step 15:

[0329] The server sends the sale information to the user's smartphone app, where the user can check the sale information through notifications.

[0330] Step 16:

[0331] Users can check the sale information within the app, which includes store addresses, discount rates, product details, and more.

[0332] Step 17:

[0333] Users can add ingredients they need to a list and create a shopping list, which can be edited and saved within the app.

[0334] Examples:

[0335] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. The data sent to the server includes the expiration date of the milk (May 1, 2023) and the expiration date of the tomatoes (April 28, 2023). The server determines that the tomatoes are close to expiring and uses an emotion engine to detect the user's current emotional state. For example, if a user is feeling stressed, the device might suggest a recipe for a relaxing herbal soup and also notify them of a 20% off sale on tomatoes at a nearby supermarket.

[0336] The above is the specific processing flow of the system that combines the emotion engine. We have explained the detailed operation of each step.

[0337] Example 2

[0338] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0339] In recent years, managing items in food storage devices has become extremely complicated, leading to problems such as users often overlooking expiration dates and being unable to use food efficiently. Furthermore, there is a lack of recipe suggestions that take into account the user's emotions and stress levels, making it difficult to suggest the most suitable dishes for the user. Another problem is that users are unable to effectively utilize sales information from nearby retailers.

[0340] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an imaging means for capturing images of items in the food storage device, a recognition means for analyzing the image data captured by the imaging means to recognize the item's identification information and expiration date, a suggestion means for suggesting optimal usage methods for the items based on the identification information and expiration date, an emotion analysis means for analyzing the user's emotional state, a suggestion means for making suggestions appropriate to the user's emotions based on the emotion analysis means, a transmission means for transmitting the recognized item information to an external database, an acquisition means for acquiring sales information from retailers near the user from the external database, and a notification means for notifying the user of the sales information acquired by the acquisition means. This allows the user to efficiently manage food, avoid overlooking expiration dates, and receive optimal recipe suggestions based on their emotional state. It also allows the user to effectively utilize information about local sales.

[0341] A "food preservation device" is a device for preserving food at a constant temperature and humidity.

[0342] The "imaging means" is a means for taking a photograph of the items in the food storage device using a device such as a camera.

[0343] "Image data" refers to data of still images or moving images obtained by an imaging means.

[0344] The "recognition means" is a means for analyzing image data and identifying the identification information and storage period of an item.

[0345] "Identification information" is information for identifying the name and type of an item.

[0346] A "shelf life" is the date by which an item is properly stored.

[0347] The "suggestion means" is a means for providing the user with the optimum method of using the product (for example, a recipe) based on the identification information and the shelf life.

[0348] The "emotion analysis means" is a means for analyzing the user's emotional state through voice input or facial recognition.

[0349] The "transmitting means" is a means for transmitting the recognized product information to an external database.

[0350] The "acquisition means" is a means for acquiring sale information of retailers in the user's neighborhood from an external database.

[0351] The "notification means" is a means for notifying the user of the acquired sale information.

[0352] An "external database" is a database that can be accessed via the Internet and that stores various product information and special sale information.

[0353] "Special sale information" is information about discounts and campaigns being carried out by retailers in the user's neighborhood.

[0354] System basic configuration and operation

[0355] This invention is a system that efficiently manages items stored in a food storage device and provides users with information on optimal usage methods and nearby special sales, taking into account the user's emotions. This system includes three elements: a "terminal" that captures images of the items stored in the food storage device, a "server" that analyzes the captured image data and the user's emotions and makes suggestions based on the item's identification information, shelf life, and user emotions, and a "user" that notifies the user of the information.

[0356] Hardware and software used

[0357] Hardware: 360-degree camera installed inside the refrigerator, Wi-Fi module

[0358] Software: Object recognition algorithms, OCR technology, sentiment analysis engine, push notification system

[0359] Specific operation flow

[0360] Scan food in the refrigerator

[0361] The device automatically enters scanning mode when the refrigerator door is closed. It activates the 360-degree camera to scan each shelf in the refrigerator and capture images of the items in the food storage unit. The image data captured by the camera is analyzed using an object recognition algorithm to identify the location and appearance of each food item. OCR technology is used to read the food label information (name, expiration date). This information is temporarily stored in the device's local memory.

[0362] Scan data transmission and analysis

[0363] The device sends the collected food information to a server via Wi-Fi. The sent data is packaged in JSON format and includes information such as the food name, expiration date, and location. The server analyzes the received data, stores the food name, expiration date, and location information in a database, and organizes them by category to understand the status of the user's refrigerator.

[0364] User sentiment analysis

[0365] The server analyzes the user's voice input and facial recognition information and uses an emotion analysis engine to determine the user's emotional state: whether the user is stressed or relaxed.

[0366] Optimal recipe suggestions

[0367] The server then suggests optimal recipes based on the stored ingredient information and the user's emotional state, obtained from an emotion analysis engine. For example, if the expiration date of tomatoes is approaching, the server will suggest recipes that use tomatoes. If the user is looking to relax, the server will also suggest a recipe for a relaxing herbal soup.

[0368] Get information on nearby sales

[0369] The server retrieves sales information from nearby retailers from an external database based on the user's location information, and identifies the retrieved sales information as relevant to the user.

[0370] User Notification

[0371] The server notifies the user of the acquired sales information and suggested recipes via the smartphone app, using the push notification function to provide the information to the user in real time.

[0372] Specific examples

[0373] For example, if there is milk, tomatoes, and carrots in the refrigerator, the device scans them and recognizes the label information. The data sent to the server includes the fact that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023. The server confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. However, at this point, the emotion engine detects from the user's voice and facial expression that they are feeling stressed. Based on this information, the server also suggests a recipe for a relaxing herbal soup that will help relieve stress.

[0374] Prompt Sentence Examples

[0375] Create a program that scans ingredients in the refrigerator and obtains their names and expiration dates using object recognition and OCR technology. Also, incorporate logic to suggest appropriate recipes based on the user's emotional state. The ingredient information will be sent to a server via Wi-Fi, where it will be analyzed and stored in a database. Also implement a function to send push notifications of special offers and recipes to the user's smartphone app.

[0376] The above is a specific example of how to implement the invention. This system streamlines food ingredient management and enables personalized recommendations that take into account the user's emotional state. It also makes shopping more efficient by utilizing information on nearby sales.

[0377] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0378] Program processing flow

[0379] Step 1:

[0380] The device automatically enters scanning mode when the refrigerator door is closed. It then activates the 360-degree camera and scans each shelf inside the refrigerator. Specifically, the camera takes a full picture of the inside of the refrigerator and acquires the resulting image (input data). This is then input into an object recognition algorithm to identify the location and appearance of each ingredient (data processing). OCR technology is used to read the label information (name, expiration date) on the ingredients. This information is temporarily stored in local memory (output data).

[0381] Step 2:

[0382] The device sends the scanned data stored in its local memory to a server via Wi-Fi. Specifically, the scanned data is packaged in JSON format (input data). The data includes the name of the ingredient, its expiration date, and its location. This data is then sent to the server via a Wi-Fi module (output data).

[0383] Step 3:

[0384] The server receives the scanned data sent from the device and stores it in a database. Specifically, it analyzes the received JSON data (input data) and organizes the names of ingredients, expiration dates, and location information by category (data calculation). The organized data is then stored in a database (output data). This allows the user to understand the status of their refrigerator.

[0385] Step 4:

[0386] Users provide voice input and facial recognition information through a smartphone app. Specifically, the smartphone's microphone and camera capture the user's voice data and facial expression data (input data). This information is then sent to the server (output data).

[0387] Step 5:

[0388] The server analyzes the voice data and facial expression data and identifies the user's emotional state using an emotion analysis engine. Specifically, the analysis engine determines whether the user is feeling stressed based on the voice and facial expression (data calculation). The emotional state is saved as internal data (output data).

[0389] Step 6:

[0390] The server proposes optimal recipes to users based on the stored ingredient information and emotional state. Specifically, it refers to the emotional state and ingredient data (input data) and selects the optimal recipe based on an algorithm (data processing). The selected recipe is saved as output data.

[0391] Step 7:

[0392] The server retrieves sales information from nearby retailers from an external database based on the user's location information. Specifically, the server sends the location information (input data) as a query to the external database to retrieve sales information (data calculation). The retrieved sales information is then stored on the server (output data).

[0393] Step 8:

[0394] The server notifies the user's smartphone app of the acquired sales information and suggested recipes. Specifically, it integrates the acquired sales information and optimal recipes (input data) into a push notification message (data processing), and then pushes this message to the smartphone app (output data).

[0395] Through these steps, the system streamlines the management of ingredients in the user's refrigerator and provides personalized recipe suggestions and sale information based on the user's emotional state.

[0396] (Application example 2)

[0397] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0398] Conventional food storage device item management systems manage item identification information and expiration dates, and provide users with sale information and usage instructions. However, they do not provide optimal usage suggestions or warnings or advice that take into account the user's emotional state. As a result, they are unable to fully alleviate the user's psychological burden and stress. This limits the user experience and the system's usefulness.

[0399] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes emotion recognition means for detecting the emotional state of the user, and warning means for providing appropriate warnings and advice based on the emotional state detected by the emotion recognition means. This makes it possible to suggest cooking recipes according to the user's emotional state, and to provide warnings and advice in real time.

[0400] A "food preservation device" is a device for preserving food at low temperatures, generally referring to a refrigerator or freezer.

[0401] "Imaging means" refers to a device for imaging an item, and includes a camera and a sensor.

[0402] "Image data" refers to visual information acquired by imaging means such as a camera or sensor.

[0403] "Recognition means" refers to a system or algorithm for analyzing image data and recognizing the identification information and shelf life of an item.

[0404] "Identification information" is information for identifying an item, such as the name or type of the item.

[0405] "Shelf life" is information that indicates the period during which food or goods can maintain their quality.

[0406] The "suggestion means" is a system that suggests to the user the most appropriate way to use the item based on the identification information and the expiration date obtained by the recognition means.

[0407] The "transmission means" is a communication means for transmitting the recognized product information to an external database.

[0408] The "acquisition means" is a system that acquires sale information of retailers in the user's neighborhood from an external database.

[0409] The "notification means" is a means for notifying the user of the acquired sale information or proposals, and includes a terminal such as a smartphone.

[0410] The "emotion recognition means" is a system that analyzes the user's voice and facial expressions to detect their emotional state.

[0411] The "warning means" is a system that provides appropriate warnings and advice to the user based on the emotional state detected by the emotion recognition means.

[0412] The present invention is a system that efficiently manages items stored in a food storage device and provides optimal usage instructions and nearby sale information that take into account the user's feelings. Specific embodiments for implementing the present invention will be described below.

[0413] System configuration

[0414] This system consists of eight main parts: imaging means (terminal), recognition means (server), emotion recognition means (server), warning means (server), suggestion means (server), acquisition means (server), transmission means (terminal), and notification means (server).

[0415] 1. Imaging means (terminal):

[0416] The terminal captures images of items using cameras and sensors installed inside the food storage device (e.g., refrigerator). The image data acquired by this imaging means becomes the basic data for acquiring the item's identification information and shelf life.

[0417] 2. Recognition means (server):

[0418] The image data sent from the terminal is analyzed to recognize the item's identification information (name, type) and expiration date. Optical character recognition (OCR) technology is used here to read the label information attached to the item.

[0419] 3. Emotion recognition means (server):

[0420] The system analyzes voice input and facial expression data captured by a camera to detect the user's emotional state. This emotion recognition means allows the system to grasp the user's emotional state in real time.

[0421] 4. Alert Method (Server):

[0422] Based on the emotional state detected by the emotion recognition means, the system provides appropriate warnings and advice to the user. For example, if the user is feeling stressed or anxious, the system will provide advice on how to relax or call attention to the situation.

[0423] 5. Proposed method (server):

[0424] The system suggests optimal recipes based on the item's identification information and expiration date obtained through the recognition method. It also takes into account the user's current emotional state to provide more personalized suggestions.

[0425] 6. Acquisition method (server):

[0426] The service retrieves sales information from vendors in the user's neighborhood from an external database, allowing the user to obtain information on how to purchase fresh ingredients at lower prices.

[0427] 7. Transmission means (terminal):

[0428] The recognized item information is sent to a server, mainly via network communication such as Wi-Fi.

[0429] 8. Notification Method (Server):

[0430] The recipes generated by the suggestion means and the sale information collected by the acquisition means are notified to the user's device such as a smartphone, etc. This includes a function to send push notifications in real time.

[0431] Processing and its concrete examples

[0432] The server analyzes the image data sent from the device and extracts the item's identification information and expiration date. Technologies used include OpenCV and OCR. Next, an emotion recognition algorithm is used to analyze the user's emotions using their voice and facial expression data.

[0433] As a concrete example, if there is milk, tomatoes, and carrots in the refrigerator, the device will take a picture of these items and send the image data to the server. The server will then obtain the item's identification information (e.g., milk, tomatoes, carrots) based on the image data and recognize their respective expiration dates (e.g., May 1, 2023, April 28, 2023). Based on this information, the proposed method will provide the optimal recipe (e.g., tomato cream soup).

[0434] Additionally, if the emotion recognition means detects that the user is feeling stressed, it will also suggest relaxing recipes (e.g., herbal soups) that can help relieve stress.

[0435] Prompt Sentence Examples

[0436] "Develop a system that monitors the user's emotions in real time using a camera while they are using smart glasses, and provides advice and safety information based on their emotional state. If the user is in a 'stressed' state, notify them of nearby safety and alert information. Please also provide example program code."

[0437] The present invention provides more personalized information and enhances user convenience by making suggestions and warnings that take into account the user's emotional state.

[0438] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0439] Step 1:

[0440] Input: A user opens the refrigerator, stores items, and then closes the refrigerator door.

[0441] Processing: The device's camera automatically activates and takes a 360-degree image of the items in the refrigerator.

[0442] Output: Image data of items in the refrigerator.

[0443] How it works: A camera scans the inside of the refrigerator, capturing the location and appearance of each item.

[0444] Step 2:

[0445] Input: Image data of items in the refrigerator.

[0446] Processing: The device temporarily stores the acquired image data in its local memory.

[0447] Output: Image data of the item stored in local memory.

[0448] Operation: Stores image data in the device's storage.

[0449] Step 3:

[0450] Input: Image data of the item stored in local memory.

[0451] Processing: The device sends the image data to the server via Wi-Fi.

[0452] Output: Image data sent to the server.

[0453] How it works: The device uploads image data to a server via a Wi-Fi connection.

[0454] Step 4:

[0455] Input: Image data sent to the server.

[0456] Processing: The server analyzes the received image data and uses object recognition algorithms to extract the item's identity (name, type) and shelf life.

[0457] Output: Item identification and shelf life data.

[0458] How it works: The server uses object recognition algorithms and OCR technology to read the label information on the item.

[0459] Step 5:

[0460] Input: Item identification and shelf life data.

[0461] Processing: The server analyzes the user's voice input and facial expression data to recognize emotions.

[0462] Output: Emotion recognition result (e.g. stress, relaxation, etc.).

[0463] How it works: The server uses emotion recognition algorithms to identify the user's emotions from audio and image data.

[0464] Step 6:

[0465] Input: Item identification, shelf life data, and emotion recognition results.

[0466] Processing: Based on this data, the server proposes the optimal usage method (recipe).

[0467] Output: Suggested recipe information.

[0468] How it works: The server searches for suitable recipes from the database and generates suggestions taking into account emotion recognition results.

[0469] Step 7:

[0470] Input: The user's location.

[0471] Processing: The server retrieves sales information from vendors in the user's vicinity from an external database.

[0472] Output:Sale information.

[0473] How it works: The server uses an external API to retrieve special offers based on the user's location.

[0474] Step 8:

[0475] Input: Suggested recipe information and special offer information.

[0476] Processing: The server notifies the user's smartphone app of this information.

[0477] Output: A notification that appears on the user's smartphone.

[0478] How it works: The server uses push notifications to send information to users in real time.

[0479] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0480] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0481] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0482] [Second embodiment]

[0483] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0484] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0485] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0486] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0487] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0488] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0489] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0490] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0491] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0492] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0493] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0494] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0495] This invention is a system that efficiently manages the items stored in a food storage device (e.g., a refrigerator) and provides users with information on optimal usage methods and nearby special sales based on the managed items. This system includes three elements: a "terminal" that captures images of the items stored in the refrigerator, a "server" that analyzes the captured image data and identifies the items and their expiration dates, and a "user" that notifies the user based on the identified information.

[0496] Basic system configuration

[0497] 1. Imaging means (terminal)

[0498] The terminal is installed inside the refrigerator and uses a 360-degree camera to capture images of the items inside the refrigerator.

[0499] Image data of the item is acquired and temporarily stored in local memory.

[0500] 2. Recognition means (server)

[0501] It receives image data sent from the terminal and uses an object recognition algorithm to analyze the item's identification information (name, type) and label information.

[0502] Optical character recognition (OCR) technology is used to recognize the expiration date printed on the product label.

[0503] 3. Proposed method (server)

[0504] Based on the recognized identity and shelf life, the system suggests the best way to use the item (e.g., a recipe).

[0505] Recipe selection also takes into account the user's food preferences and past history.

[0506] 4. Transmission means (terminal)

[0507] The recognized item information is transmitted from the terminal to the server.

[0508] The data is packaged in JSON format and sent over Wi-Fi.

[0509] 5. Acquisition method (server)

[0510] Get special offers from vendors near the user from an external database.

[0511] The acquired sale information is sorted and information relevant to the user is identified.

[0512] 6. Notification Method (Server)

[0513] The acquired sale information and suggested recipes are notified to the user's smartphone app.

[0514] Use push notifications to provide users with real-time information.

[0515] Specific processing of the program

[0516] Scan food in the refrigerator

[0517] The device automatically enters scanning mode when the refrigerator door is closed, activating its 360-degree camera to scan each shelf inside the refrigerator. The image data captured by the camera is analyzed using an object recognition algorithm to identify the location and appearance of each food item. OCR technology is used to read the food label information (name, expiration date). This information is stored in local memory and later sent to the server.

[0518] Scan data transmission and analysis

[0519] The device sends the collected information about ingredients to a server via Wi-Fi. The server analyzes the received data and stores it in a database. The names, expiration dates, and location information of ingredients are organized by category, allowing the user to understand the status of their refrigerator.

[0520] Optimal recipe suggestions

[0521] The server then suggests optimal recipes to the user based on the stored ingredient information. The server then references the user's food preferences and past history to search for and generate a list of relevant recipes. This recipe list is then sent to the user's smartphone app via push notification.

[0522] Get information on nearby sales

[0523] The server retrieves sales information from nearby retailers from an external database based on the user's location information. The retrieved sales information is matched with the information about ingredients in the user's refrigerator to identify related sales information. This sales information is also notified to the user's smartphone app.

[0524] Specific examples

[0525] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. It determines that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023, and sends this information to the server. The server then confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. Along with this suggestion, the user is also notified of a 20% off sale on tomatoes at a nearby supermarket. The user can check this information on their smartphone app and plan their cooking and shopping accordingly.

[0526] The above is a description of specific embodiments for carrying out the present invention.

[0527] The processing flow will be explained below.

[0528] Step 1:

[0529] The device automatically enters scanning mode when the refrigerator door is closed, activating its built-in 360-degree camera to capture images of each shelf inside the refrigerator.

[0530] Step 2:

[0531] The device analyzes the image data captured by the camera using an object recognition algorithm, which identifies the location and type of each food item in the refrigerator.

[0532] Step 3:

[0533] The device uses OCR (optical character recognition) technology to read the label information (name, expiration date) of each ingredient, and this information is temporarily stored in the device's local memory.

[0534] Step 4:

[0535] The device sends the acquired information to a server via Wi-Fi. The data is packaged in JSON format and encrypted.

[0536] Step 5:

[0537] The server analyzes the received data and stores it in a database, organizing the names of ingredients, expiration dates, and location information by category.

[0538] Step 6:

[0539] The server periodically scans the database for ingredients that are nearing their expiration date and generates an alert for any ingredients that are nearing their expiration date.

[0540] Step 7:

[0541] The server refers to the user's preferences and past history, suggests recipes using ingredients that are close to their expiration date, searches for relevant recipes in the recipe database, and generates a list.

[0542] Step 8:

[0543] The server notifies the user of the generated recipe list and alert information via the user's smartphone app using the push notification function.

[0544] Step 9:

[0545] Users receive notifications on their smartphone app, which can include audio alerts and banners.

[0546] Step 10:

[0547] The user opens the app to check the ingredients in the refrigerator and see the suggested recipes. The app displays a list of ingredients, expiration dates, and suggested recipes.

[0548] Step 11:

[0549] The server retrieves sales information from nearby retailers from an external database based on the user's location information, and obtains the latest sales data through the supermarket's API.

[0550] Step 12:

[0551] The server analyzes the acquired sales information and identifies items that match the food information in the user's refrigerator. The identified sales information is organized and added to a recommendation list.

[0552] Step 13:

[0553] The server sends the recommendation list to the user's smartphone app, where the user can check the sale information through notifications.

[0554] Step 14:

[0555] Users can check the sale information within the app, which includes store addresses, discount rates, product details, and more.

[0556] Step 15:

[0557] Users can add ingredients they need to a list and create a shopping list, which can be edited and saved within the app.

[0558] The above is the processing flow of the specific program of the system. The specific operation has been explained in detail for each step.

[0559] Example 1

[0560] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0561] In modern households, food waste is a problem due to the inability to properly manage items stored in food storage devices. Other issues include overlooking items that are approaching their expiration date and taking time to find the perfect recipe. Furthermore, to efficiently do daily shopping, it is necessary to quickly obtain information about sales from nearby retailers, but current systems are not able to adequately handle this.

[0562] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0563] In this invention, the server includes an imaging means for capturing images of items in the food preservation device, a recognition means for analyzing the image data captured by the imaging means to recognize the identification information and shelf life of the items, a suggestion means for suggesting optimal usage methods for the items based on the identification information and shelf life, a transmission means for transmitting the recognized item information to an external database, a acquisition means for acquiring sales information from retailers near the user from the external database, a notification means for notifying the user of the sales information acquired by the acquisition means, a means for the imaging means to scan the interior of the refrigerator with a 360-degree camera triggered by a door sensor, and a means for the notification means to provide information to the user's mobile information terminal using a push notification function. This enables expiration date management of items in the food preservation device, suggestions for optimal usage methods, and efficient shopping.

[0564] A "food preservation device" is a device for preserving food at low temperatures, and mainly refers to a refrigerator or freezer.

[0565] "Imaging means" refers to a device for photographing items in the food storage device, and includes a camera and a sensor.

[0566] The "recognition means" refers to a technique or device for analyzing image data obtained by the imaging means and recognizing the identification information and shelf life of an item.

[0567] The "suggestion means" is a function or device for suggesting the most suitable method of use related to the item based on the identification information and the expiration date obtained by the recognition means.

[0568] The "transmitting means" refers to a function or device for transmitting the recognized product information to an external database.

[0569] The "acquisition means" refers to a function or device for acquiring sale information from an external database.

[0570] The "notification means" is a function or device for notifying the user of the acquired sale information.

[0571] A "door sensor" is a sensor that detects whether the refrigerator door is open or closed.

[0572] A "360-degree camera" is a camera that can capture images in all directions.

[0573] "Mobile information terminal" refers to a portable information processing device such as a mobile phone, smartphone, or tablet.

[0574] The "push notification function" is a function that delivers information from a server to a user's mobile information terminal in real time.

[0575] A "food management system" is a system that efficiently manages the items in a refrigerator and provides optimal usage methods and special sale information based on that information.

[0576] This system efficiently manages the items stored in a food storage device and provides users with information on optimal usage methods and nearby special sales based on that information. This system includes three elements: a terminal that captures images of the items stored in the refrigerator, a server that analyzes the captured image data and identifies the items and their expiration dates, and a user interface that notifies the user based on the identified information.

[0577] Specifically, the following hardware and software are used:

[0578] Imaging means (terminal)

[0579] The device is installed inside the refrigerator and captures images of the items inside using a 360-degree camera. Image data of the items is acquired and temporarily stored in local memory. When the door sensor detects that the refrigerator door is closed, the 360-degree camera is activated and begins scanning.

[0580] Recognition means (server)

[0581] The server receives the image data sent from the device and analyzes the item's identification information (name, type) and label information using an object recognition algorithm. Specifically, it uses a TensorFlow-based object recognition model. It also uses Tesseract as OCR technology to recognize the expiration date printed on the item's label.

[0582] Proposal method (server)

[0583] The server then suggests optimal ways to use the item (e.g., recipes) based on the recognized identification information and expiration date. The server also takes into account the user's food preferences and past history when selecting recipes. Based on the stored ingredient information, the server selects and lists the optimal recipes.

[0584] Transmission means (terminal)

[0585] The device sends the recognized item information to the server, packaged in JSON format, and transmitted via Wi-Fi.

[0586] Acquisition method (server)

[0587] The server retrieves sales information from nearby retailers from an external database, sorts the sales information, and identifies those relevant to the user. It then uses GeoAPI to obtain the user's location and sends an API request to the external database based on that location.

[0588] Notification method (server)

[0589] The server then notifies the user of the sale information and suggested recipes in real time via the smartphone app using Firebase Cloud Messaging (FCM).

[0590] Specific examples

[0591] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. It determines that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023, and sends this information to the server. The server then confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. Along with this suggestion, the user is also notified of a 20% off sale on tomatoes at a nearby supermarket. The user can check this information on their smartphone app and plan their cooking and shopping accordingly.

[0592] Example prompts for generative AI models

[0593] Here is an example prompt for the generative AI model to write specific steps for this system:

[0594] Please explain in detail the following system steps, including examples:

[0595] This system efficiently manages the contents of a food storage device (refrigerator) and provides users with information on optimal use and nearby special offers. It works by following these steps:

[0596] 1. Initiating a refrigerator scan: The device detects that the refrigerator door is closed, and the 360-degree camera activates to capture images of each shelf.

[0597] 2. Acquisition and temporary storage of image data: The image data captured by the camera is acquired and temporarily stored in local memory.

[0598] 3. Sending image data: The device sends the acquired image data to the server via Wi-Fi.

[0599] 4. Image data analysis: The server analyzes the received image data and extracts ingredient information.

[0600] 5. Storing and organizing ingredient information: The server stores and organizes ingredient information in a database.

[0601] 6. Optimal recipe suggestion: The server suggests optimal recipes to the user based on the stored ingredient information.

[0602] 7. Get nearby sales information: The server retrieves nearby sales information from an external database and selects the most relevant ones.

[0603] 8. User notification: The server pushes suggested recipes and special offers to the user's smartphone app.

[0604] For example, imagine you have milk, tomatoes, and carrots in your refrigerator. Your device scans them and sends that information to a server. The server sees that the tomatoes are nearing their expiration date and sends you a recipe for cream of tomato soup and a sale on tomatoes.

[0605] The above is a specific embodiment for carrying out the invention.

[0606] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0607] Step 1: Start the fridge scan

[0608] When the door sensor detects that the refrigerator door is closed, the device automatically enters scanning mode. The 360-degree camera starts up and takes pictures of each shelf in turn. The input is the sensor signal indicating that the door is closed, and the output is the image data captured by the camera.

[0609] Specific operation: After the door sensor detects the door closing, the camera waits 5 seconds and then starts shooting in all directions, scanning each shelf 10 degrees at a time, capturing a total of 36 image data.

[0610] Step 2: Acquire and temporarily save image data

[0611] The image data captured by the camera is acquired and temporarily stored in local memory. The input is image data taken by a 360-degree camera, and the output is a high-resolution JPEG image file stored in local memory.

[0612] How it works: Each scanned image is saved to local memory in real time and stored in a temporary storage area. The image is saved in JPEG format and a timestamp is added to each image.

[0613] Step 3: Sending image data

[0614] The device sends image data stored in its local memory to the server via Wi-Fi. The data is packaged in JSON format and sent as high-priority data. The input is the image data in the local memory, and the output is the JSON package sent to the server.

[0615] Specific behavior: The device checks for a Wi-Fi connection, sends data in 5MB chunks, and clears local memory after the transmission is complete.

[0616] Step 4: Analyzing the image data

[0617] The server analyzes the received image data and uses an object recognition algorithm to extract the identification information (name, type) of each ingredient. It also uses OCR technology to read the expiration date printed on the ingredient label. The input is a JSON package sent from the device, and the output is data containing the identification information and expiration date.

[0618] Specific operation: The server uses a TensorFlow-based object recognition model and only recognizes results with a confidence level of 0.9 or higher. OCR analysis is performed using the Tesseract library, and label data is extracted as text data.

[0619] Step 5: Save and organize ingredient information

[0620] The server stores the recognition results in a database and organizes the names, types, shelf life, and location information of ingredients by category. The input is data including identification information and shelf life, and the output is a database organized by category.

[0621] How it works: Recognition results are stored in a MySQL database and immediately indexed. Each item is categorized for efficient searching.

[0622] Step 6: Propose the best recipe

[0623] The server compares the stored ingredient information with the user's preferences and past history to suggest optimal recipes. The input is an organized database and the user's preferences and history data, and the output is a list of recommended recipes.

[0624] Specific operation: The server refers to the user profile and uses a matching algorithm to select three candidate recipes, generate a list of recommended recipes, and send it to the next notification step.

[0625] Step 7: Get local sales information

[0626] The server retrieves nearby sales information from an external retailer database based on the user's location information. The input is the user's location information and sale information from the external database, and the output is highly relevant sale information.

[0627] How it works: The server uses GeoAPI to obtain the user's location information and sends an API request to an external database based on that information. The obtained sale information is matched with food information, and the most relevant information is selected.

[0628] Step 8: Notify users

[0629] The server sends suggested recipes and sales information to the user's mobile device using push notifications. The input is a list of suggested recipes and related sales information, and the output is the notification sent to the user's mobile device.

[0630] How it works: The server uses Firebase Cloud Messaging (FCM) to send real-time notifications to users' smartphones, including detailed recipe instructions and special offers, designed to allow users to take immediate action.

[0631] (Application example 1)

[0632] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0633] Conventional food preservation devices have the problem that it is difficult to manage the inventory of food in the refrigerator and grasp the expiration date, resulting in a lot of food going to waste. Furthermore, when users shop in physical stores, they are unable to create an optimal purchasing plan that is linked to existing inventory, resulting in duplicate or unnecessary purchases. This leads to problems of wasting resources and increasing the burden on household finances.

[0634] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0635] In this invention, the server includes an imaging means for imaging items in the food preservation device, a recognition means for analyzing image data obtained by the imaging means to recognize the identification information and shelf life of the items, a suggestion means for suggesting optimal usage methods for the items based on the identification information and shelf life, a transmission means for transmitting the recognized item information to an external database, an acquisition means for acquiring sales information from retailers near the user from the external database, a notification means for notifying the user of the sales information acquired by the acquisition means, and a shopping support means for providing the recognized item information and sales information and suggesting an optimal purchasing plan when the user is shopping at a physical store. This allows the user to shop efficiently while referring to inventory information in the refrigerator in real time.

[0636] A "food preservation device" is a device used to maintain the quality of food and preserve it for a long period of time.

[0637] "Goods" refers to food and beverages stored in the refrigerator.

[0638] The "imaging means" is a device including a camera and a sensor for acquiring image data of an object.

[0639] "Image data" refers to visual information acquired by an imaging means expressed in digital form.

[0640] The "recognition means" is a device or system that analyzes the acquired image data and extracts the identification information and storage period of the item.

[0641] "Identification information" refers to information for identifying the name, type, characteristics, etc. of an item.

[0642] "Shelf life" is information that indicates the date by which an item can be consumed.

[0643] The "suggestion means" is a device or system that suggests to the user the most appropriate way to use the product based on the identification information and the shelf life.

[0644] The "transmitting means" is a device or system for transmitting the recognized product information to an external database.

[0645] "External database" refers to a database that stores and manages product information and special sale information and provides necessary information.

[0646] The "acquisition means" is a device or system for acquiring sale information of retailers in the user's neighborhood from an external database.

[0647] The "notification means" is a device or system for notifying the user of the acquired sale information and suggested usage methods.

[0648] A "shopping support means" is a device or system that provides recognized product information and sale information to a user when shopping at a physical store, and suggests an optimal purchasing plan.

[0649] "Brick and mortar store" refers to a store that sells products in person at a physical location.

[0650] The present invention is a system that efficiently manages the items in a food storage device and provides users with optimal usage instructions and nearby sales information based on the managed items. This system is composed of the following elements:

[0651] 1. Imaging means (terminal)

[0652] The device is installed inside the refrigerator and captures images of the items inside using a 360-degree camera. The image data captured by the camera is temporarily stored in local memory.

[0653] The specific hardware used is a 360-degree camera (e.g., Ricoh Theta) and a sensor module (e.g., Raspberry Pi).

[0654] 2. Recognition means (server)

[0655] The server receives the image data sent from the terminal and analyzes the identification information and storage period of the item.

[0656] Image recognition uses object recognition algorithms (e.g., YOLO) and optical character recognition (OCR) technology (e.g., Tesseract OCR) to identify the item's identity and shelf life.

[0657] 3. Proposed method (server)

[0658] Based on the recognized product information, the server suggests optimal usage methods, such as cooking recipes, to the user.

[0659] The system selects appropriate recipes based on the user's food preferences and past usage history.

[0660] 4. Transmission means (terminal)

[0661] The device packages the recognized item information in JSON format and sends it to the server via Wi-Fi.

[0662] 5. Acquisition method (server)

[0663] The server retrieves sales information from an external database for merchants in the user's vicinity.

[0664] The sale information is matched with information about items whose expiration dates are approaching for the user.

[0665] 6. Notification Method (Server)

[0666] The server then notifies the user of the acquired sale information and suggested recipes via push notification to their smartphone.

[0667] 7. Shopping Support Methods (User)

[0668] When users shop at a physical store, the smartphone app displays food items with upcoming expiration dates, inventory information, and special offers.

[0669] This allows users to shop efficiently and reduce waste.

[0670] Specific operation example

[0671] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. It determines that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023, and sends this information to the server. The server then confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. Along with this suggestion, the user is also notified of a 20% off sale on tomatoes at a nearby supermarket. The user can check this information on their smartphone app and plan their cooking and shopping accordingly.

[0672] Prompt Sentence Examples

[0673] "The system analyzes the image of food inventory in the refrigerator and identifies foods that are nearing their expiration date. While the user is shopping in a physical store, it takes into account the refrigerator's inventory and sales information and suggests the optimal shopping list in real time. It also includes recipes based on ingredients that are nearing their expiration date."

[0674] The above is a description of specific embodiments for carrying out the present invention.

[0675] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0676] Step 1:

[0677] The device starts scanning the inside of the refrigerator. When the refrigerator door is closed, the 360-degree camera automatically starts up and scans the interior. The image data acquired by the image capture is temporarily stored in local memory.

[0678] Input: Current state of the refrigerator

[0679] Data processing: 360-degree camera imaging

[0680] Output: 360-degree image data of the inside of the refrigerator

[0681] Step 2:

[0682] The device sends the scanned data to the server, and the image data stored in the local memory is packaged in JSON format and sent to the cloud server via Wi-Fi.

[0683] Input: 360-degree image data

[0684] Data processing: Packaging into JSON format

[0685] Output: Image data sent to the server

[0686] Step 3:

[0687] The server analyzes the received data. The cloud server analyzes the received image data and identifies the item's identity using an object recognition algorithm (e.g., YOLO). It also uses OCR technology (e.g., Tesseract OCR) to extract label information (name, expiration date), and stores this information in a database.

[0688] Input: Image data sent to the server

[0689] Data processing: object recognition, OCR analysis, saving to database

[0690] Output: Identification information and retention period

[0691] Step 4:

[0692] The server proposes the optimal way to use the product. Based on the recognized identification information and expiration date, the server proposes the optimal way to use the product, such as cooking recipes, to the user. The server selects recipes taking into consideration the user's food preferences and past history.

[0693] Input: Identification information, retention period, user preferences and history

[0694] Data processing: Recipe selection

[0695] Output: Suggested recipe information

[0696] Step 5:

[0697] The device receives the suggested recipe information from the server and notifies the user's smartphone. The suggested recipe information is sent to the user's smartphone via push notification.

[0698] Input: Suggested recipe information

[0699] Data processing: Notification to smartphone

[0700] Output: Recipe notification to smartphone

[0701] Step 6:

[0702] The server retrieves sale information from an external database, and based on the user's location information, retrieves sale information from nearby retailers and matches it with items with an approaching expiration date.

[0703] Input: User's location

[0704] Data processing: Acquiring and matching sale information

[0705] Output: Related deals

[0706] Step 7:

[0707] The server notifies the user of the special sale information it has acquired, and displays the information on the smartphone so that the user can refer to it when shopping at a physical store.

[0708] Input: Related sales information

[0709] Data processing: Notification to smartphone

[0710] Output: Special sale information notification to smartphone

[0711] Step 8:

[0712] Users can use a smartphone app to receive purchasing assistance in physical stores. The app displays items with upcoming expiration dates and sales information, and suggests optimal purchasing plans. This allows users to shop efficiently.

[0713] Input: Information notified to your smartphone

[0714] Data processing: Displaying expiration dates and special sale information, suggesting purchasing plans

[0715] Output: Purchase support information for the user

[0716] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0717] This invention is a system that efficiently manages items in a food storage device (e.g., a refrigerator) and provides users with optimal usage methods and nearby sale information that also take into account the user's emotions. This system includes three elements: a "terminal" that captures images of items in the refrigerator, a "server" that analyzes the image data and the user's emotions and makes suggestions based on the item's identification information, shelf life, and user emotions, and a "user" that notifies the user of the information.

[0718] Basic system configuration

[0719] 1. Imaging means (terminal)

[0720] The terminal is installed inside the refrigerator and uses a 360-degree camera to capture images of the items inside the refrigerator.

[0721] Image data of the item is acquired and temporarily stored in local memory.

[0722] 2. Recognition means (server)

[0723] It receives image data sent from the terminal and uses an object recognition algorithm to analyze the item's identification information (name, type) and label information.

[0724] OCR (optical character recognition) technology is used to recognize the expiration date printed on the product label.

[0725] 3. Proposed method (server)

[0726] Based on the recognized identity and shelf life, the system suggests the best way to use the item (e.g., a recipe).

[0727] The recipe selection incorporates an emotion engine that reflects the user's emotions, and by analyzing the user's current emotional state, it makes more personalized suggestions.

[0728] 4. Emotion engine (server)

[0729] It includes sensors that detect emotions from the user's voice input and facial expressions, and analyzes the information acquired by these sensors to identify the user's emotional state.

[0730] Based on the identified emotion information, a suggestion suited to the emotion of the user is made.

[0731] 5. Transmission means (terminal)

[0732] The recognized item information is transmitted from the terminal to the server.

[0733] The data is packaged in JSON format and sent over Wi-Fi.

[0734] 6. Acquisition method (server)

[0735] Get special offers from vendors near the user from an external database.

[0736] The acquired sale information is sorted and information relevant to the user is identified.

[0737] 7. Notification Method (Server)

[0738] The acquired sale information and suggested recipes are notified to the user's smartphone app.

[0739] Use push notifications to provide users with real-time information.

[0740] Specific processing of the program

[0741] Scan food in the refrigerator

[0742] The device automatically enters scanning mode when the refrigerator door is closed, activating its 360-degree camera to scan each shelf inside the refrigerator. The image data captured by the camera is analyzed using an object recognition algorithm to identify the location and appearance of each food item. OCR technology is used to read the food label information (name, expiration date). This information is stored in local memory and later sent to the server.

[0743] Scan data transmission and analysis

[0744] The device sends the collected information about ingredients to a server via Wi-Fi. The server analyzes the received data and stores it in a database. The names, expiration dates, and location information of ingredients are organized by category, allowing the user to understand the status of their refrigerator.

[0745] Optimal recipe suggestions

[0746] The server suggests optimal recipes to the user based on the stored ingredient information. Furthermore, it uses an emotion engine to select recipes that take into account the user's current emotional state. The server analyzes the user's voice input and facial recognition information to identify the user's emotional state. For example, if the user is feeling stressed, it suggests easy-to-make recipes or dishes that will help them relax.

[0747] Example: Emotion-based recipe suggestions

[0748] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. The data sent to the server includes the expiration date of the milk (May 1, 2023) and the expiration date of the tomatoes (April 28, 2023).

[0749] The server determines that the tomatoes are close to expiring and suggests a recipe for tomato cream soup. However, the emotion engine then detects that the user is feeling stressed from their voice and facial expressions. Based on this information, the server also suggests a recipe for a relaxing herbal soup that will help relieve stress.

[0750] Get information on nearby sales

[0751] The server retrieves sales information from nearby retailers from an external database based on the user's location information. It then sorts the retrieved information and identifies sales information relevant to the user. This sales information is also notified to the user's smartphone app.

[0752] Example: Notifications with sentiment and sales information

[0753] The smartphone app notifies users of a 20% off sale on tomatoes at a nearby supermarket along with recipes based on their emotions (e.g., herb soup). Because the app responds to the user's emotional state, it also provides suggestions for relaxation along with the recipe. Utilizing this information, users can plan their cooking and shopping accordingly.

[0754] The above is a description of a specific embodiment for carrying out the present invention. This system takes into account the user's emotions and makes more personalized suggestions, thereby improving the convenience of ingredient management and cooking.

[0755] The processing flow will be explained below.

[0756] Step 1:

[0757] The device automatically enters scanning mode when the refrigerator door is closed, activating the 360-degree camera and capturing images of each shelf inside the refrigerator.

[0758] Step 2:

[0759] The device temporarily stores image data captured by the camera in its local memory. The captured data includes all items in the refrigerator.

[0760] Step 3:

[0761] The device analyzes the image data and uses object recognition algorithms to identify the location and type of each item, thereby determining what is stored in the refrigerator.

[0762] Step 4:

[0763] The terminal uses OCR (optical character recognition) technology to read the label information (name, expiration date) of each item and stores this information in local memory.

[0764] Step 5:

[0765] The device sends the acquired ingredient information to a server via Wi-Fi, where the data is packaged in JSON format and encrypted.

[0766] Step 6:

[0767] The server analyzes the received data and stores it in a database. It organizes the names of ingredients, expiration dates, and location information by category, and keeps track of the status of the user's refrigerator.

[0768] Step 7:

[0769] The server periodically scans the database for ingredients that are nearing their expiration date and generates an alert for any ingredients that are nearing their expiration date.

[0770] Step 8:

[0771] The server activates the emotion engine and uses sensors to capture the user's voice input and facial expressions, thereby detecting the user's current emotional state.

[0772] Step 9:

[0773] The server selects recipe suggestions based on the user's emotional state. For example, if the user is feeling stressed, it will suggest recipes with a relaxing effect.

[0774] Step 10:

[0775] The server generates an optimal recipe list based on the stored ingredient information and emotion information, including suggestions based on ingredient usage and emotion.

[0776] Step 11:

[0777] The server notifies the user's smartphone app of the generated recipe list and alert information using the push notification function.

[0778] Step 12:

[0779] Users receive a notification via a smartphone app that displays a list of ingredients, expiration dates, and suggested recipes based on their emotions.

[0780] Step 13:

[0781] The server retrieves sales information from nearby retailers from an external database based on the user's location information, and obtains the latest sales data through the supermarket's API.

[0782] Step 14:

[0783] The server analyzes the sales information and identifies items that match the food information in the user's refrigerator. The sales information is organized into items relevant to the user.

[0784] Step 15:

[0785] The server sends the sale information to the user's smartphone app, where the user can check the sale information through notifications.

[0786] Step 16:

[0787] Users can check the sale information within the app, which includes store addresses, discount rates, product details, and more.

[0788] Step 17:

[0789] Users can add ingredients they need to a list and create a shopping list, which can be edited and saved within the app.

[0790] Examples:

[0791] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. The data sent to the server includes the expiration date of the milk (May 1, 2023) and the expiration date of the tomatoes (April 28, 2023). The server determines that the tomatoes are close to expiring and uses an emotion engine to detect the user's current emotional state. For example, if a user is feeling stressed, the device might suggest a recipe for a relaxing herbal soup and also notify them of a 20% off sale on tomatoes at a nearby supermarket.

[0792] The above is the specific processing flow of the system that combines the emotion engine. We have explained the detailed operation of each step.

[0793] Example 2

[0794] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0795] In recent years, managing items in food storage devices has become extremely complicated, leading to problems such as users often overlooking expiration dates and being unable to use food efficiently. Furthermore, there is a lack of recipe suggestions that take into account the user's emotions and stress levels, making it difficult to suggest the most suitable dishes for the user. Another problem is that users are unable to effectively utilize sales information from nearby retailers.

[0796] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an imaging means for capturing images of items in the food storage device, a recognition means for analyzing the image data captured by the imaging means to recognize the item's identification information and expiration date, a suggestion means for suggesting optimal usage methods for the items based on the identification information and expiration date, an emotion analysis means for analyzing the user's emotional state, a suggestion means for making suggestions appropriate to the user's emotions based on the emotion analysis means, a transmission means for transmitting the recognized item information to an external database, an acquisition means for acquiring sales information from retailers near the user from the external database, and a notification means for notifying the user of the sales information acquired by the acquisition means. This allows the user to efficiently manage food, avoid overlooking expiration dates, and receive optimal recipe suggestions based on their emotional state. It also allows the user to effectively utilize information about local sales.

[0797] A "food preservation device" is a device for preserving food at a constant temperature and humidity.

[0798] The "imaging means" is a means for taking a photograph of the items in the food storage device using a device such as a camera.

[0799] "Image data" refers to data of still images or moving images obtained by an imaging means.

[0800] The "recognition means" is a means for analyzing image data and identifying the identification information and storage period of an item.

[0801] "Identification information" is information for identifying the name and type of an item.

[0802] A "shelf life" is the date by which an item is properly stored.

[0803] The "suggestion means" is a means for providing the user with the optimum method of using the product (for example, a recipe) based on the identification information and the shelf life.

[0804] The "emotion analysis means" is a means for analyzing the user's emotional state through voice input or facial recognition.

[0805] The "transmitting means" is a means for transmitting the recognized product information to an external database.

[0806] The "acquisition means" is a means for acquiring sale information of retailers in the user's neighborhood from an external database.

[0807] The "notification means" is a means for notifying the user of the acquired sale information.

[0808] An "external database" is a database that can be accessed via the Internet and that stores various product information and special sale information.

[0809] "Special sale information" is information about discounts and campaigns being carried out by retailers in the user's neighborhood.

[0810] System basic configuration and operation

[0811] This invention is a system that efficiently manages items stored in a food storage device and provides users with information on optimal usage methods and nearby special sales, taking into account the user's emotions. This system includes three elements: a "terminal" that captures images of the items stored in the food storage device, a "server" that analyzes the captured image data and the user's emotions and makes suggestions based on the item's identification information, shelf life, and user emotions, and a "user" that notifies the user of the information.

[0812] Hardware and software used

[0813] Hardware: 360-degree camera installed inside the refrigerator, Wi-Fi module

[0814] Software: Object recognition algorithms, OCR technology, sentiment analysis engine, push notification system

[0815] Specific operation flow

[0816] Scan food in the refrigerator

[0817] The device automatically enters scanning mode when the refrigerator door is closed. It activates the 360-degree camera to scan each shelf in the refrigerator and capture images of the items in the food storage unit. The image data captured by the camera is analyzed using an object recognition algorithm to identify the location and appearance of each food item. OCR technology is used to read the food label information (name, expiration date). This information is temporarily stored in the device's local memory.

[0818] Scan data transmission and analysis

[0819] The device sends the collected food information to a server via Wi-Fi. The sent data is packaged in JSON format and includes information such as the food name, expiration date, and location. The server analyzes the received data, stores the food name, expiration date, and location information in a database, and organizes them by category to understand the status of the user's refrigerator.

[0820] User sentiment analysis

[0821] The server analyzes the user's voice input and facial recognition information and uses an emotion analysis engine to determine the user's emotional state: whether the user is stressed or relaxed.

[0822] Optimal recipe suggestions

[0823] The server then suggests optimal recipes based on the stored ingredient information and the user's emotional state, obtained from an emotion analysis engine. For example, if the expiration date of tomatoes is approaching, the server will suggest recipes that use tomatoes. If the user is looking to relax, the server will also suggest a recipe for a relaxing herbal soup.

[0824] Get information on nearby sales

[0825] The server retrieves sales information from nearby retailers from an external database based on the user's location information, and identifies the retrieved sales information as relevant to the user.

[0826] User Notification

[0827] The server notifies the user of the acquired sales information and suggested recipes via the smartphone app, using the push notification function to provide the information to the user in real time.

[0828] Specific examples

[0829] For example, if there is milk, tomatoes, and carrots in the refrigerator, the device scans them and recognizes the label information. The data sent to the server includes the fact that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023. The server confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. However, at this point, the emotion engine detects from the user's voice and facial expression that they are feeling stressed. Based on this information, the server also suggests a recipe for a relaxing herbal soup that will help relieve stress.

[0830] Prompt Sentence Examples

[0831] Create a program that scans ingredients in the refrigerator and obtains their names and expiration dates using object recognition and OCR technology. Also, incorporate logic to suggest appropriate recipes based on the user's emotional state. The ingredient information will be sent to a server via Wi-Fi, where it will be analyzed and stored in a database. Also implement a function to send push notifications of special offers and recipes to the user's smartphone app.

[0832] The above is a specific example of how to implement the invention. This system streamlines food ingredient management and enables personalized recommendations that take into account the user's emotional state. It also makes shopping more efficient by utilizing information on nearby sales.

[0833] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0834] Program processing flow

[0835] Step 1:

[0836] The device automatically enters scanning mode when the refrigerator door is closed. It then activates the 360-degree camera and scans each shelf inside the refrigerator. Specifically, the camera takes a full picture of the inside of the refrigerator and acquires the resulting image (input data). This is then input into an object recognition algorithm to identify the location and appearance of each ingredient (data processing). OCR technology is used to read the label information (name, expiration date) on the ingredients. This information is temporarily stored in local memory (output data).

[0837] Step 2:

[0838] The device sends the scanned data stored in its local memory to a server via Wi-Fi. Specifically, the scanned data is packaged in JSON format (input data). The data includes the name of the ingredient, its expiration date, and its location. This data is then sent to the server via a Wi-Fi module (output data).

[0839] Step 3:

[0840] The server receives the scanned data sent from the device and stores it in a database. Specifically, it analyzes the received JSON data (input data) and organizes the names of ingredients, expiration dates, and location information by category (data calculation). The organized data is then stored in a database (output data). This allows the user to understand the status of their refrigerator.

[0841] Step 4:

[0842] Users provide voice input and facial recognition information through a smartphone app. Specifically, the smartphone's microphone and camera capture the user's voice data and facial expression data (input data). This information is then sent to the server (output data).

[0843] Step 5:

[0844] The server analyzes the voice data and facial expression data and identifies the user's emotional state using an emotion analysis engine. Specifically, the analysis engine determines whether the user is feeling stressed based on the voice and facial expression (data calculation). The emotional state is saved as internal data (output data).

[0845] Step 6:

[0846] The server proposes optimal recipes to users based on the stored ingredient information and emotional state. Specifically, it refers to the emotional state and ingredient data (input data) and selects the optimal recipe based on an algorithm (data processing). The selected recipe is saved as output data.

[0847] Step 7:

[0848] The server retrieves sales information from nearby retailers from an external database based on the user's location information. Specifically, the server sends the location information (input data) as a query to the external database to retrieve sales information (data calculation). The retrieved sales information is then stored on the server (output data).

[0849] Step 8:

[0850] The server notifies the user's smartphone app of the acquired sales information and suggested recipes. Specifically, it integrates the acquired sales information and optimal recipes (input data) into a push notification message (data processing), and then pushes this message to the smartphone app (output data).

[0851] Through these steps, the system streamlines the management of ingredients in the user's refrigerator and provides personalized recipe suggestions and sale information based on the user's emotional state.

[0852] (Application example 2)

[0853] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0854] Conventional food storage device item management systems manage item identification information and expiration dates, and provide users with sale information and usage instructions. However, they do not provide optimal usage suggestions or warnings or advice that take into account the user's emotional state. As a result, they are unable to fully alleviate the user's psychological burden and stress. This limits the user experience and the system's usefulness.

[0855] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes emotion recognition means for detecting the emotional state of the user, and warning means for providing appropriate warnings and advice based on the emotional state detected by the emotion recognition means. This makes it possible to suggest cooking recipes according to the user's emotional state, and to provide warnings and advice in real time.

[0856] A "food preservation device" is a device for preserving food at low temperatures, generally referring to a refrigerator or freezer.

[0857] "Imaging means" refers to a device for imaging an item, and includes a camera and a sensor.

[0858] "Image data" refers to visual information acquired by imaging means such as a camera or sensor.

[0859] "Recognition means" refers to a system or algorithm for analyzing image data and recognizing the identification information and shelf life of an item.

[0860] "Identification information" is information for identifying an item, such as the name or type of the item.

[0861] "Shelf life" is information that indicates the period during which food or goods can maintain their quality.

[0862] The "suggestion means" is a system that suggests to the user the most appropriate way to use the item based on the identification information and the expiration date obtained by the recognition means.

[0863] The "transmission means" is a communication means for transmitting the recognized product information to an external database.

[0864] The "acquisition means" is a system that acquires sale information of retailers in the user's neighborhood from an external database.

[0865] The "notification means" is a means for notifying the user of the acquired sale information or proposals, and includes a terminal such as a smartphone.

[0866] The "emotion recognition means" is a system that analyzes the user's voice and facial expressions to detect their emotional state.

[0867] The "warning means" is a system that provides appropriate warnings and advice to the user based on the emotional state detected by the emotion recognition means.

[0868] The present invention is a system that efficiently manages items stored in a food storage device and provides optimal usage instructions and nearby sale information that take into account the user's feelings. Specific embodiments for implementing the present invention will be described below.

[0869] System configuration

[0870] This system consists of eight main parts: imaging means (terminal), recognition means (server), emotion recognition means (server), warning means (server), suggestion means (server), acquisition means (server), transmission means (terminal), and notification means (server).

[0871] 1. Imaging means (terminal):

[0872] The terminal captures images of items using cameras and sensors installed inside the food storage device (e.g., refrigerator). The image data acquired by this imaging means becomes the basic data for acquiring the item's identification information and shelf life.

[0873] 2. Recognition means (server):

[0874] The image data sent from the terminal is analyzed to recognize the item's identification information (name, type) and expiration date. Optical character recognition (OCR) technology is used here to read the label information attached to the item.

[0875] 3. Emotion recognition means (server):

[0876] The system analyzes voice input and facial expression data captured by a camera to detect the user's emotional state. This emotion recognition means allows the system to grasp the user's emotional state in real time.

[0877] 4. Alert Method (Server):

[0878] Based on the emotional state detected by the emotion recognition means, the system provides appropriate warnings and advice to the user. For example, if the user is feeling stressed or anxious, the system will provide advice on how to relax or call attention to the situation.

[0879] 5. Proposed method (server):

[0880] The system suggests optimal recipes based on the item's identification information and expiration date obtained through the recognition method. It also takes into account the user's current emotional state to provide more personalized suggestions.

[0881] 6. Acquisition method (server):

[0882] The service retrieves sales information from vendors in the user's neighborhood from an external database, allowing the user to obtain information on how to purchase fresh ingredients at lower prices.

[0883] 7. Transmission means (terminal):

[0884] The recognized item information is sent to a server, mainly via network communication such as Wi-Fi.

[0885] 8. Notification Method (Server):

[0886] The recipes generated by the suggestion means and the sale information collected by the acquisition means are notified to the user's device such as a smartphone, etc. This includes a function to send push notifications in real time.

[0887] Processing and its concrete examples

[0888] The server analyzes the image data sent from the device and extracts the item's identification information and expiration date. Technologies used include OpenCV and OCR. Next, an emotion recognition algorithm is used to analyze the user's emotions using their voice and facial expression data.

[0889] As a concrete example, if there is milk, tomatoes, and carrots in the refrigerator, the device will take a picture of these items and send the image data to the server. The server will then obtain the item's identification information (e.g., milk, tomatoes, carrots) based on the image data and recognize their respective expiration dates (e.g., May 1, 2023, April 28, 2023). Based on this information, the proposed method will provide the optimal recipe (e.g., tomato cream soup).

[0890] Additionally, if the emotion recognition means detects that the user is feeling stressed, it will also suggest relaxing recipes (e.g., herbal soups) that can help relieve stress.

[0891] Prompt Sentence Examples

[0892] "Develop a system that monitors the user's emotions in real time using a camera while they are using smart glasses, and provides advice and safety information based on their emotional state. If the user is in a 'stressed' state, notify them of nearby safety and alert information. Please also provide example program code."

[0893] The present invention provides more personalized information and enhances user convenience by making suggestions and warnings that take into account the user's emotional state.

[0894] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0895] Step 1:

[0896] Input: A user opens the refrigerator, stores items, and then closes the refrigerator door.

[0897] Processing: The device's camera automatically activates and takes a 360-degree image of the items in the refrigerator.

[0898] Output: Image data of items in the refrigerator.

[0899] How it works: A camera scans the inside of the refrigerator, capturing the location and appearance of each item.

[0900] Step 2:

[0901] Input: Image data of items in the refrigerator.

[0902] Processing: The device temporarily stores the acquired image data in its local memory.

[0903] Output: Image data of the item stored in local memory.

[0904] Operation: Stores image data in the device's storage.

[0905] Step 3:

[0906] Input: Image data of the item stored in local memory.

[0907] Processing: The device sends the image data to the server via Wi-Fi.

[0908] Output: Image data sent to the server.

[0909] How it works: The device uploads image data to a server via a Wi-Fi connection.

[0910] Step 4:

[0911] Input: Image data sent to the server.

[0912] Processing: The server analyzes the received image data and uses object recognition algorithms to extract the item's identity (name, type) and shelf life.

[0913] Output: Item identification and shelf life data.

[0914] How it works: The server uses object recognition algorithms and OCR technology to read the label information on the item.

[0915] Step 5:

[0916] Input: Item identification and shelf life data.

[0917] Processing: The server analyzes the user's voice input and facial expression data to recognize emotions.

[0918] Output: Emotion recognition result (e.g. stress, relaxation, etc.).

[0919] How it works: The server uses emotion recognition algorithms to identify the user's emotions from audio and image data.

[0920] Step 6:

[0921] Input: Item identification, shelf life data, and emotion recognition results.

[0922] Processing: Based on this data, the server proposes the optimal usage method (recipe).

[0923] Output: Suggested recipe information.

[0924] How it works: The server searches for suitable recipes from the database and generates suggestions taking into account emotion recognition results.

[0925] Step 7:

[0926] Input: The user's location.

[0927] Processing: The server retrieves sales information from vendors in the user's vicinity from an external database.

[0928] Output:Sale information.

[0929] How it works: The server uses an external API to retrieve special offers based on the user's location.

[0930] Step 8:

[0931] Input: Suggested recipe information and special offer information.

[0932] Processing: The server notifies the user's smartphone app of this information.

[0933] Output: A notification that appears on the user's smartphone.

[0934] How it works: The server uses push notifications to send information to users in real time.

[0935] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0936] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0937] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0938] [Third embodiment]

[0939] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0940] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0941] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0942] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0943] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0944] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0945] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0946] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0947] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0948] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0949] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0950] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0951] This invention is a system that efficiently manages the items stored in a food storage device (e.g., a refrigerator) and provides users with information on optimal usage methods and nearby special sales based on the managed items. This system includes three elements: a "terminal" that captures images of the items stored in the refrigerator, a "server" that analyzes the captured image data and identifies the items and their expiration dates, and a "user" that notifies the user based on the identified information.

[0952] Basic system configuration

[0953] 1. Imaging means (terminal)

[0954] The terminal is installed inside the refrigerator and uses a 360-degree camera to capture images of the items inside the refrigerator.

[0955] Image data of the item is acquired and temporarily stored in local memory.

[0956] 2. Recognition means (server)

[0957] It receives image data sent from the terminal and uses an object recognition algorithm to analyze the item's identification information (name, type) and label information.

[0958] Optical character recognition (OCR) technology is used to recognize the expiration date printed on the product label.

[0959] 3. Proposed method (server)

[0960] Based on the recognized identity and shelf life, the system suggests the best way to use the item (e.g., a recipe).

[0961] Recipe selection also takes into account the user's food preferences and past history.

[0962] 4. Transmission means (terminal)

[0963] The recognized item information is transmitted from the terminal to the server.

[0964] The data is packaged in JSON format and sent over Wi-Fi.

[0965] 5. Acquisition method (server)

[0966] Get special offers from vendors near the user from an external database.

[0967] The acquired sale information is sorted and information relevant to the user is identified.

[0968] 6. Notification Method (Server)

[0969] The acquired sale information and suggested recipes are notified to the user's smartphone app.

[0970] Use push notifications to provide users with real-time information.

[0971] Specific processing of the program

[0972] Scan food in the refrigerator

[0973] The device automatically enters scanning mode when the refrigerator door is closed, activating its 360-degree camera to scan each shelf inside the refrigerator. The image data captured by the camera is analyzed using an object recognition algorithm to identify the location and appearance of each food item. OCR technology is used to read the food label information (name, expiration date). This information is stored in local memory and later sent to the server.

[0974] Scan data transmission and analysis

[0975] The device sends the collected information about ingredients to a server via Wi-Fi. The server analyzes the received data and stores it in a database. The names, expiration dates, and location information of ingredients are organized by category, allowing the user to understand the status of their refrigerator.

[0976] Optimal recipe suggestions

[0977] The server then suggests optimal recipes to the user based on the stored ingredient information. The server then references the user's food preferences and past history to search for and generate a list of relevant recipes. This recipe list is then sent to the user's smartphone app via push notification.

[0978] Get information on nearby sales

[0979] The server retrieves sales information from nearby retailers from an external database based on the user's location information. The retrieved sales information is matched with the information about ingredients in the user's refrigerator to identify related sales information. This sales information is also notified to the user's smartphone app.

[0980] Specific examples

[0981] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. It determines that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023, and sends this information to the server. The server then confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. Along with this suggestion, the user is also notified of a 20% off sale on tomatoes at a nearby supermarket. The user can check this information on their smartphone app and plan their cooking and shopping accordingly.

[0982] The above is a description of specific embodiments for carrying out the present invention.

[0983] The processing flow will be explained below.

[0984] Step 1:

[0985] The device automatically enters scanning mode when the refrigerator door is closed, activating its built-in 360-degree camera to capture images of each shelf inside the refrigerator.

[0986] Step 2:

[0987] The device analyzes the image data captured by the camera using an object recognition algorithm, which identifies the location and type of each food item in the refrigerator.

[0988] Step 3:

[0989] The device uses OCR (optical character recognition) technology to read the label information (name, expiration date) of each ingredient, and this information is temporarily stored in the device's local memory.

[0990] Step 4:

[0991] The device sends the acquired information to a server via Wi-Fi. The data is packaged in JSON format and encrypted.

[0992] Step 5:

[0993] The server analyzes the received data and stores it in a database, organizing the names of ingredients, expiration dates, and location information by category.

[0994] Step 6:

[0995] The server periodically scans the database for ingredients that are nearing their expiration date and generates an alert for any ingredients that are nearing their expiration date.

[0996] Step 7:

[0997] The server refers to the user's preferences and past history, suggests recipes using ingredients that are close to their expiration date, searches for relevant recipes in the recipe database, and generates a list.

[0998] Step 8:

[0999] The server notifies the user of the generated recipe list and alert information via the user's smartphone app using the push notification function.

[1000] Step 9:

[1001] Users receive notifications on their smartphone app, which can include audio alerts and banners.

[1002] Step 10:

[1003] The user opens the app to check the ingredients in the refrigerator and see the suggested recipes. The app displays a list of ingredients, expiration dates, and suggested recipes.

[1004] Step 11:

[1005] The server retrieves sales information from nearby retailers from an external database based on the user's location information, and obtains the latest sales data through the supermarket's API.

[1006] Step 12:

[1007] The server analyzes the acquired sales information and identifies items that match the food information in the user's refrigerator. The identified sales information is organized and added to a recommendation list.

[1008] Step 13:

[1009] The server sends the recommendation list to the user's smartphone app, where the user can check the sale information through notifications.

[1010] Step 14:

[1011] Users can check the sale information within the app, which includes store addresses, discount rates, product details, and more.

[1012] Step 15:

[1013] Users can add ingredients they need to a list and create a shopping list, which can be edited and saved within the app.

[1014] The above is the processing flow of the specific program of the system. The specific operation has been explained in detail for each step.

[1015] Example 1

[1016] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1017] In modern households, food waste is a problem due to the inability to properly manage items stored in food storage devices. Other issues include overlooking items that are approaching their expiration date and taking time to find the perfect recipe. Furthermore, to efficiently do daily shopping, it is necessary to quickly obtain information about sales from nearby retailers, but current systems are not able to adequately handle this.

[1018] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1019] In this invention, the server includes an imaging means for capturing images of items in the food preservation device, a recognition means for analyzing the image data captured by the imaging means to recognize the identification information and shelf life of the items, a suggestion means for suggesting optimal usage methods for the items based on the identification information and shelf life, a transmission means for transmitting the recognized item information to an external database, a acquisition means for acquiring sales information from retailers near the user from the external database, a notification means for notifying the user of the sales information acquired by the acquisition means, a means for the imaging means to scan the interior of the refrigerator with a 360-degree camera triggered by a door sensor, and a means for the notification means to provide information to the user's mobile information terminal using a push notification function. This enables expiration date management of items in the food preservation device, suggestions for optimal usage methods, and efficient shopping.

[1020] A "food preservation device" is a device for preserving food at low temperatures, and mainly refers to a refrigerator or freezer.

[1021] "Imaging means" refers to a device for photographing items in the food storage device, and includes a camera and a sensor.

[1022] The "recognition means" refers to a technique or device for analyzing image data obtained by the imaging means and recognizing the identification information and shelf life of an item.

[1023] The "suggestion means" is a function or device for suggesting the most suitable method of use related to the item based on the identification information and the expiration date obtained by the recognition means.

[1024] The "transmitting means" refers to a function or device for transmitting the recognized product information to an external database.

[1025] The "acquisition means" refers to a function or device for acquiring sale information from an external database.

[1026] The "notification means" is a function or device for notifying the user of the acquired sale information.

[1027] A "door sensor" is a sensor that detects whether the refrigerator door is open or closed.

[1028] A "360-degree camera" is a camera that can capture images in all directions.

[1029] "Mobile information terminal" refers to a portable information processing device such as a mobile phone, smartphone, or tablet.

[1030] The "push notification function" is a function that delivers information from a server to a user's mobile information terminal in real time.

[1031] A "food management system" is a system that efficiently manages the items in a refrigerator and provides optimal usage methods and special sale information based on that information.

[1032] This system efficiently manages the items stored in a food storage device and provides users with information on optimal usage methods and nearby special sales based on that information. This system includes three elements: a terminal that captures images of the items stored in the refrigerator, a server that analyzes the captured image data and identifies the items and their expiration dates, and a user interface that notifies the user based on the identified information.

[1033] Specifically, the following hardware and software are used:

[1034] Imaging means (terminal)

[1035] The device is installed inside the refrigerator and captures images of the items inside using a 360-degree camera. Image data of the items is acquired and temporarily stored in local memory. When the door sensor detects that the refrigerator door is closed, the 360-degree camera is activated and begins scanning.

[1036] Recognition means (server)

[1037] The server receives the image data sent from the device and analyzes the item's identification information (name, type) and label information using an object recognition algorithm. Specifically, it uses a TensorFlow-based object recognition model. It also uses Tesseract as OCR technology to recognize the expiration date printed on the item's label.

[1038] Proposal method (server)

[1039] The server then suggests optimal ways to use the item (e.g., recipes) based on the recognized identification information and expiration date. The server also takes into account the user's food preferences and past history when selecting recipes. Based on the stored ingredient information, the server selects and lists the optimal recipes.

[1040] Transmission means (terminal)

[1041] The device sends the recognized item information to the server, packaged in JSON format, and transmitted via Wi-Fi.

[1042] Acquisition method (server)

[1043] The server retrieves sales information from nearby retailers from an external database, sorts the sales information, and identifies those relevant to the user. It then uses GeoAPI to obtain the user's location and sends an API request to the external database based on that location.

[1044] Notification method (server)

[1045] The server then notifies the user of the sale information and suggested recipes in real time via the smartphone app using Firebase Cloud Messaging (FCM).

[1046] Specific examples

[1047] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. It determines that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023, and sends this information to the server. The server then confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. Along with this suggestion, the user is also notified of a 20% off sale on tomatoes at a nearby supermarket. The user can check this information on their smartphone app and plan their cooking and shopping accordingly.

[1048] Example prompts for generative AI models

[1049] Here is an example prompt for the generative AI model to write specific steps for this system:

[1050] Please explain in detail the following system steps, including examples:

[1051] This system efficiently manages the contents of a food storage device (refrigerator) and provides users with information on optimal use and nearby special offers. It works by following these steps:

[1052] 1. Initiating a refrigerator scan: The device detects that the refrigerator door is closed, and the 360-degree camera activates to capture images of each shelf.

[1053] 2. Acquisition and temporary storage of image data: The image data captured by the camera is acquired and temporarily stored in local memory.

[1054] 3. Sending image data: The device sends the acquired image data to the server via Wi-Fi.

[1055] 4. Image data analysis: The server analyzes the received image data and extracts ingredient information.

[1056] 5. Storing and organizing ingredient information: The server stores and organizes ingredient information in a database.

[1057] 6. Optimal recipe suggestion: The server suggests optimal recipes to the user based on the stored ingredient information.

[1058] 7. Get nearby sales information: The server retrieves nearby sales information from an external database and selects the most relevant ones.

[1059] 8. User notification: The server pushes suggested recipes and special offers to the user's smartphone app.

[1060] For example, imagine you have milk, tomatoes, and carrots in your refrigerator. Your device scans them and sends that information to a server. The server sees that the tomatoes are nearing their expiration date and sends you a recipe for cream of tomato soup and a sale on tomatoes.

[1061] The above is a specific embodiment for carrying out the invention.

[1062] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1063] Step 1: Start the fridge scan

[1064] When the door sensor detects that the refrigerator door is closed, the device automatically enters scanning mode. The 360-degree camera starts up and takes pictures of each shelf in turn. The input is the sensor signal indicating that the door is closed, and the output is the image data captured by the camera.

[1065] Specific operation: After the door sensor detects the door closing, the camera waits 5 seconds and then starts shooting in all directions, scanning each shelf 10 degrees at a time, capturing a total of 36 image data.

[1066] Step 2: Acquire and temporarily save image data

[1067] The image data captured by the camera is acquired and temporarily stored in local memory. The input is image data taken by a 360-degree camera, and the output is a high-resolution JPEG image file stored in local memory.

[1068] How it works: Each scanned image is saved to local memory in real time and stored in a temporary storage area. The image is saved in JPEG format and a timestamp is added to each image.

[1069] Step 3: Sending image data

[1070] The device sends image data stored in its local memory to the server via Wi-Fi. The data is packaged in JSON format and sent as high-priority data. The input is the image data in the local memory, and the output is the JSON package sent to the server.

[1071] Specific behavior: The device checks for a Wi-Fi connection, sends data in 5MB chunks, and clears local memory after the transmission is complete.

[1072] Step 4: Analyzing the image data

[1073] The server analyzes the received image data and uses an object recognition algorithm to extract the identification information (name, type) of each ingredient. It also uses OCR technology to read the expiration date printed on the ingredient label. The input is a JSON package sent from the device, and the output is data containing the identification information and expiration date.

[1074] Specific operation: The server uses a TensorFlow-based object recognition model and only recognizes results with a confidence level of 0.9 or higher. OCR analysis is performed using the Tesseract library, and label data is extracted as text data.

[1075] Step 5: Save and organize ingredient information

[1076] The server stores the recognition results in a database and organizes the names, types, shelf life, and location information of ingredients by category. The input is data including identification information and shelf life, and the output is a database organized by category.

[1077] How it works: Recognition results are stored in a MySQL database and immediately indexed. Each item is categorized for efficient searching.

[1078] Step 6: Propose the best recipe

[1079] The server compares the stored ingredient information with the user's preferences and past history to suggest optimal recipes. The input is an organized database and the user's preferences and history data, and the output is a list of recommended recipes.

[1080] Specific operation: The server refers to the user profile and uses a matching algorithm to select three candidate recipes, generate a list of recommended recipes, and send it to the next notification step.

[1081] Step 7: Get local sales information

[1082] The server retrieves nearby sales information from an external retailer database based on the user's location information. The input is the user's location information and sale information from the external database, and the output is highly relevant sale information.

[1083] How it works: The server uses GeoAPI to obtain the user's location information and sends an API request to an external database based on that information. The obtained sale information is matched with food information, and the most relevant information is selected.

[1084] Step 8: Notify users

[1085] The server sends suggested recipes and sales information to the user's mobile device using push notifications. The input is a list of suggested recipes and related sales information, and the output is the notification sent to the user's mobile device.

[1086] How it works: The server uses Firebase Cloud Messaging (FCM) to send real-time notifications to users' smartphones, including detailed recipe instructions and special offers, designed to allow users to take immediate action.

[1087] (Application example 1)

[1088] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1089] Conventional food preservation devices have the problem that it is difficult to manage the inventory of food in the refrigerator and grasp the expiration date, resulting in a lot of food going to waste. Furthermore, when users shop in physical stores, they are unable to create an optimal purchasing plan that is linked to existing inventory, resulting in duplicate or unnecessary purchases. This leads to problems of wasting resources and increasing the burden on household finances.

[1090] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1091] In this invention, the server includes an imaging means for imaging items in the food preservation device, a recognition means for analyzing image data obtained by the imaging means to recognize the identification information and shelf life of the items, a suggestion means for suggesting optimal usage methods for the items based on the identification information and shelf life, a transmission means for transmitting the recognized item information to an external database, an acquisition means for acquiring sales information from retailers near the user from the external database, a notification means for notifying the user of the sales information acquired by the acquisition means, and a shopping support means for providing the recognized item information and sales information and suggesting an optimal purchasing plan when the user is shopping at a physical store. This allows the user to shop efficiently while referring to inventory information in the refrigerator in real time.

[1092] A "food preservation device" is a device used to maintain the quality of food and preserve it for a long period of time.

[1093] "Goods" refers to food and beverages stored in the refrigerator.

[1094] The "imaging means" is a device including a camera and a sensor for acquiring image data of an object.

[1095] "Image data" refers to visual information acquired by an imaging means expressed in digital form.

[1096] The "recognition means" is a device or system that analyzes the acquired image data and extracts the identification information and storage period of the item.

[1097] "Identification information" refers to information for identifying the name, type, characteristics, etc. of an item.

[1098] "Shelf life" is information that indicates the date by which an item can be consumed.

[1099] The "suggestion means" is a device or system that suggests to the user the most appropriate way to use the product based on the identification information and the shelf life.

[1100] The "transmitting means" is a device or system for transmitting the recognized product information to an external database.

[1101] "External database" refers to a database that stores and manages product information and special sale information and provides necessary information.

[1102] The "acquisition means" is a device or system for acquiring sale information of retailers in the user's neighborhood from an external database.

[1103] The "notification means" is a device or system for notifying the user of the acquired sale information and suggested usage methods.

[1104] A "shopping support means" is a device or system that provides recognized product information and sale information to a user when shopping at a physical store, and suggests an optimal purchasing plan.

[1105] "Brick and mortar store" refers to a store that sells products in person at a physical location.

[1106] The present invention is a system that efficiently manages the items in a food storage device and provides users with optimal usage instructions and nearby sales information based on the managed items. This system is composed of the following elements:

[1107] 1. Imaging means (terminal)

[1108] The device is installed inside the refrigerator and captures images of the items inside using a 360-degree camera. The image data captured by the camera is temporarily stored in local memory.

[1109] The specific hardware used is a 360-degree camera (e.g., Ricoh Theta) and a sensor module (e.g., Raspberry Pi).

[1110] 2. Recognition means (server)

[1111] The server receives the image data sent from the terminal and analyzes the identification information and storage period of the item.

[1112] Image recognition uses object recognition algorithms (e.g., YOLO) and optical character recognition (OCR) technology (e.g., Tesseract OCR) to identify the item's identity and shelf life.

[1113] 3. Proposed method (server)

[1114] Based on the recognized product information, the server suggests optimal usage methods, such as cooking recipes, to the user.

[1115] The system selects appropriate recipes based on the user's food preferences and past usage history.

[1116] 4. Transmission means (terminal)

[1117] The device packages the recognized item information in JSON format and sends it to the server via Wi-Fi.

[1118] 5. Acquisition method (server)

[1119] The server retrieves sales information from an external database for merchants in the user's vicinity.

[1120] The sale information is matched with information about items whose expiration dates are approaching for the user.

[1121] 6. Notification Method (Server)

[1122] The server then notifies the user of the acquired sale information and suggested recipes via push notification to their smartphone.

[1123] 7. Shopping Support Methods (User)

[1124] When users shop at a physical store, the smartphone app displays food items with upcoming expiration dates, inventory information, and special offers.

[1125] This allows users to shop efficiently and reduce waste.

[1126] Specific operation example

[1127] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. It determines that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023, and sends this information to the server. The server then confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. Along with this suggestion, the user is also notified of a 20% off sale on tomatoes at a nearby supermarket. The user can check this information on their smartphone app and plan their cooking and shopping accordingly.

[1128] Prompt Sentence Examples

[1129] "The system analyzes the image of food inventory in the refrigerator and identifies foods that are nearing their expiration date. While the user is shopping in a physical store, it takes into account the refrigerator's inventory and sales information and suggests the optimal shopping list in real time. It also includes recipes based on ingredients that are nearing their expiration date."

[1130] The above is a description of specific embodiments for carrying out the present invention.

[1131] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1132] Step 1:

[1133] The device starts scanning the inside of the refrigerator. When the refrigerator door is closed, the 360-degree camera automatically starts up and scans the interior. The image data acquired by the image capture is temporarily stored in local memory.

[1134] Input: Current state of the refrigerator

[1135] Data processing: 360-degree camera imaging

[1136] Output: 360-degree image data of the inside of the refrigerator

[1137] Step 2:

[1138] The device sends the scanned data to the server, and the image data stored in the local memory is packaged in JSON format and sent to the cloud server via Wi-Fi.

[1139] Input: 360-degree image data

[1140] Data processing: Packaging into JSON format

[1141] Output: Image data sent to the server

[1142] Step 3:

[1143] The server analyzes the received data. The cloud server analyzes the received image data and identifies the item's identity using an object recognition algorithm (e.g., YOLO). It also uses OCR technology (e.g., Tesseract OCR) to extract label information (name, expiration date), and stores this information in a database.

[1144] Input: Image data sent to the server

[1145] Data processing: object recognition, OCR analysis, saving to database

[1146] Output: Identification information and retention period

[1147] Step 4:

[1148] The server proposes the optimal way to use the product. Based on the recognized identification information and expiration date, the server proposes the optimal way to use the product, such as cooking recipes, to the user. The server selects recipes taking into consideration the user's food preferences and past history.

[1149] Input: Identification information, retention period, user preferences and history

[1150] Data processing: Recipe selection

[1151] Output: Suggested recipe information

[1152] Step 5:

[1153] The device receives the suggested recipe information from the server and notifies the user's smartphone. The suggested recipe information is sent to the user's smartphone via push notification.

[1154] Input: Suggested recipe information

[1155] Data processing: Notification to smartphone

[1156] Output: Recipe notification to smartphone

[1157] Step 6:

[1158] The server retrieves sale information from an external database, and based on the user's location information, retrieves sale information from nearby retailers and matches it with items with an approaching expiration date.

[1159] Input: User's location

[1160] Data processing: Acquiring and matching sale information

[1161] Output: Related deals

[1162] Step 7:

[1163] The server notifies the user of the special sale information it has acquired, and displays the information on the smartphone so that the user can refer to it when shopping at a physical store.

[1164] Input: Related sales information

[1165] Data processing: Notification to smartphone

[1166] Output: Special sale information notification to smartphone

[1167] Step 8:

[1168] Users can use a smartphone app to receive purchasing assistance in physical stores. The app displays items with upcoming expiration dates and sales information, and suggests optimal purchasing plans. This allows users to shop efficiently.

[1169] Input: Information notified to your smartphone

[1170] Data processing: Displaying expiration dates and special sale information, suggesting purchasing plans

[1171] Output: Purchase support information for the user

[1172] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1173] This invention is a system that efficiently manages items in a food storage device (e.g., a refrigerator) and provides users with optimal usage methods and nearby sale information that also take into account the user's emotions. This system includes three elements: a "terminal" that captures images of items in the refrigerator, a "server" that analyzes the image data and the user's emotions and makes suggestions based on the item's identification information, shelf life, and user emotions, and a "user" that notifies the user of the information.

[1174] Basic system configuration

[1175] 1. Imaging means (terminal)

[1176] The terminal is installed inside the refrigerator and uses a 360-degree camera to capture images of the items inside the refrigerator.

[1177] Image data of the item is acquired and temporarily stored in local memory.

[1178] 2. Recognition means (server)

[1179] It receives image data sent from the terminal and uses an object recognition algorithm to analyze the item's identification information (name, type) and label information.

[1180] OCR (optical character recognition) technology is used to recognize the expiration date printed on the product label.

[1181] 3. Proposed method (server)

[1182] Based on the recognized identity and shelf life, the system suggests the best way to use the item (e.g., a recipe).

[1183] The recipe selection incorporates an emotion engine that reflects the user's emotions, and by analyzing the user's current emotional state, it makes more personalized suggestions.

[1184] 4. Emotion engine (server)

[1185] It includes sensors that detect emotions from the user's voice input and facial expressions, and analyzes the information acquired by these sensors to identify the user's emotional state.

[1186] Based on the identified emotion information, a suggestion suited to the emotion of the user is made.

[1187] 5. Transmission means (terminal)

[1188] The recognized item information is transmitted from the terminal to the server.

[1189] The data is packaged in JSON format and sent over Wi-Fi.

[1190] 6. Acquisition method (server)

[1191] Get special offers from vendors near the user from an external database.

[1192] The acquired sale information is sorted and information relevant to the user is identified.

[1193] 7. Notification Method (Server)

[1194] The acquired sale information and suggested recipes are notified to the user's smartphone app.

[1195] Use push notifications to provide users with real-time information.

[1196] Specific processing of the program

[1197] Scan food in the refrigerator

[1198] The device automatically enters scanning mode when the refrigerator door is closed, activating its 360-degree camera to scan each shelf inside the refrigerator. The image data captured by the camera is analyzed using an object recognition algorithm to identify the location and appearance of each food item. OCR technology is used to read the food label information (name, expiration date). This information is stored in local memory and later sent to the server.

[1199] Scan data transmission and analysis

[1200] The device sends the collected information about ingredients to a server via Wi-Fi. The server analyzes the received data and stores it in a database. The names, expiration dates, and location information of ingredients are organized by category, allowing the user to understand the status of their refrigerator.

[1201] Optimal recipe suggestions

[1202] The server suggests optimal recipes to the user based on the stored ingredient information. Furthermore, it uses an emotion engine to select recipes that take into account the user's current emotional state. The server analyzes the user's voice input and facial recognition information to identify the user's emotional state. For example, if the user is feeling stressed, it suggests easy-to-make recipes or dishes that will help them relax.

[1203] Example: Emotion-based recipe suggestions

[1204] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. The data sent to the server includes the expiration date of the milk (May 1, 2023) and the expiration date of the tomatoes (April 28, 2023).

[1205] The server determines that the tomatoes are close to expiring and suggests a recipe for tomato cream soup. However, the emotion engine then detects that the user is feeling stressed from their voice and facial expressions. Based on this information, the server also suggests a recipe for a relaxing herbal soup that will help relieve stress.

[1206] Get information on nearby sales

[1207] The server retrieves sales information from nearby retailers from an external database based on the user's location information. It then sorts the retrieved information and identifies sales information relevant to the user. This sales information is also notified to the user's smartphone app.

[1208] Example: Notifications with sentiment and sales information

[1209] The smartphone app notifies users of a 20% off sale on tomatoes at a nearby supermarket along with recipes based on their emotions (e.g., herb soup). Because the app responds to the user's emotional state, it also provides suggestions for relaxation along with the recipe. Utilizing this information, users can plan their cooking and shopping accordingly.

[1210] The above is a description of a specific embodiment for carrying out the present invention. This system takes into account the user's emotions and makes more personalized suggestions, thereby improving the convenience of ingredient management and cooking.

[1211] The processing flow will be explained below.

[1212] Step 1:

[1213] The device automatically enters scanning mode when the refrigerator door is closed, activating the 360-degree camera and capturing images of each shelf inside the refrigerator.

[1214] Step 2:

[1215] The device temporarily stores image data captured by the camera in its local memory. The captured data includes all items in the refrigerator.

[1216] Step 3:

[1217] The device analyzes the image data and uses object recognition algorithms to identify the location and type of each item, thereby determining what is stored in the refrigerator.

[1218] Step 4:

[1219] The terminal uses OCR (optical character recognition) technology to read the label information (name, expiration date) of each item and stores this information in local memory.

[1220] Step 5:

[1221] The device sends the acquired ingredient information to a server via Wi-Fi, where the data is packaged in JSON format and encrypted.

[1222] Step 6:

[1223] The server analyzes the received data and stores it in a database. It organizes the names of ingredients, expiration dates, and location information by category, and keeps track of the status of the user's refrigerator.

[1224] Step 7:

[1225] The server periodically scans the database for ingredients that are nearing their expiration date and generates an alert for any ingredients that are nearing their expiration date.

[1226] Step 8:

[1227] The server activates the emotion engine and uses sensors to capture the user's voice input and facial expressions, thereby detecting the user's current emotional state.

[1228] Step 9:

[1229] The server selects recipe suggestions based on the user's emotional state. For example, if the user is feeling stressed, it will suggest recipes with a relaxing effect.

[1230] Step 10:

[1231] The server generates an optimal recipe list based on the stored ingredient information and emotion information, including suggestions based on ingredient usage and emotion.

[1232] Step 11:

[1233] The server notifies the user's smartphone app of the generated recipe list and alert information using the push notification function.

[1234] Step 12:

[1235] Users receive a notification via a smartphone app that displays a list of ingredients, expiration dates, and suggested recipes based on their emotions.

[1236] Step 13:

[1237] The server retrieves sales information from nearby retailers from an external database based on the user's location information, and obtains the latest sales data through the supermarket's API.

[1238] Step 14:

[1239] The server analyzes the sales information and identifies items that match the food information in the user's refrigerator. The sales information is organized into items relevant to the user.

[1240] Step 15:

[1241] The server sends the sale information to the user's smartphone app, where the user can check the sale information through notifications.

[1242] Step 16:

[1243] Users can check the sale information within the app, which includes store addresses, discount rates, product details, and more.

[1244] Step 17:

[1245] Users can add ingredients they need to a list and create a shopping list, which can be edited and saved within the app.

[1246] Examples:

[1247] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. The data sent to the server includes the expiration date of the milk (May 1, 2023) and the expiration date of the tomatoes (April 28, 2023). The server determines that the tomatoes are close to expiring and uses an emotion engine to detect the user's current emotional state. For example, if a user is feeling stressed, the device might suggest a recipe for a relaxing herbal soup and also notify them of a 20% off sale on tomatoes at a nearby supermarket.

[1248] The above is the specific processing flow of the system that combines the emotion engine. We have explained the detailed operation of each step.

[1249] Example 2

[1250] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1251] In recent years, managing items in food storage devices has become extremely complicated, leading to problems such as users often overlooking expiration dates and being unable to use food efficiently. Furthermore, there is a lack of recipe suggestions that take into account the user's emotions and stress levels, making it difficult to suggest the most suitable dishes for the user. Another problem is that users are unable to effectively utilize sales information from nearby retailers.

[1252] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an imaging means for capturing images of items in the food storage device, a recognition means for analyzing the image data captured by the imaging means to recognize the item's identification information and expiration date, a suggestion means for suggesting optimal usage methods for the items based on the identification information and expiration date, an emotion analysis means for analyzing the user's emotional state, a suggestion means for making suggestions appropriate to the user's emotions based on the emotion analysis means, a transmission means for transmitting the recognized item information to an external database, an acquisition means for acquiring sales information from retailers near the user from the external database, and a notification means for notifying the user of the sales information acquired by the acquisition means. This allows the user to efficiently manage food, avoid overlooking expiration dates, and receive optimal recipe suggestions based on their emotional state. It also allows the user to effectively utilize information about local sales.

[1253] A "food preservation device" is a device for preserving food at a constant temperature and humidity.

[1254] The "imaging means" is a means for taking a photograph of the items in the food storage device using a device such as a camera.

[1255] "Image data" refers to data of still images or moving images obtained by an imaging means.

[1256] The "recognition means" is a means for analyzing image data and identifying the identification information and storage period of an item.

[1257] "Identification information" is information for identifying the name and type of an item.

[1258] A "shelf life" is the date by which an item is properly stored.

[1259] The "suggestion means" is a means for providing the user with the optimum method of using the product (for example, a recipe) based on the identification information and the shelf life.

[1260] The "emotion analysis means" is a means for analyzing the user's emotional state through voice input or facial recognition.

[1261] The "transmitting means" is a means for transmitting the recognized product information to an external database.

[1262] The "acquisition means" is a means for acquiring sale information of retailers in the user's neighborhood from an external database.

[1263] The "notification means" is a means for notifying the user of the acquired sale information.

[1264] An "external database" is a database that can be accessed via the Internet and that stores various product information and special sale information.

[1265] "Special sale information" is information about discounts and campaigns being carried out by retailers in the user's neighborhood.

[1266] System basic configuration and operation

[1267] This invention is a system that efficiently manages items stored in a food storage device and provides users with information on optimal usage methods and nearby special sales, taking into account the user's emotions. This system includes three elements: a "terminal" that captures images of the items stored in the food storage device, a "server" that analyzes the captured image data and the user's emotions and makes suggestions based on the item's identification information, shelf life, and user emotions, and a "user" that notifies the user of the information.

[1268] Hardware and software used

[1269] Hardware: 360-degree camera installed inside the refrigerator, Wi-Fi module

[1270] Software: Object recognition algorithms, OCR technology, sentiment analysis engine, push notification system

[1271] Specific operation flow

[1272] Scan food in the refrigerator

[1273] The device automatically enters scanning mode when the refrigerator door is closed. It activates the 360-degree camera to scan each shelf in the refrigerator and capture images of the items in the food storage unit. The image data captured by the camera is analyzed using an object recognition algorithm to identify the location and appearance of each food item. OCR technology is used to read the food label information (name, expiration date). This information is temporarily stored in the device's local memory.

[1274] Scan data transmission and analysis

[1275] The device sends the collected food information to a server via Wi-Fi. The sent data is packaged in JSON format and includes information such as the food name, expiration date, and location. The server analyzes the received data, stores the food name, expiration date, and location information in a database, and organizes them by category to understand the status of the user's refrigerator.

[1276] User sentiment analysis

[1277] The server analyzes the user's voice input and facial recognition information and uses an emotion analysis engine to determine the user's emotional state: whether the user is stressed or relaxed.

[1278] Optimal recipe suggestions

[1279] The server then suggests optimal recipes based on the stored ingredient information and the user's emotional state, obtained from an emotion analysis engine. For example, if the expiration date of tomatoes is approaching, the server will suggest recipes that use tomatoes. If the user is looking to relax, the server will also suggest a recipe for a relaxing herbal soup.

[1280] Get information on nearby sales

[1281] The server retrieves sales information from nearby retailers from an external database based on the user's location information, and identifies the retrieved sales information as relevant to the user.

[1282] User Notification

[1283] The server notifies the user of the acquired sales information and suggested recipes via the smartphone app, using the push notification function to provide the information to the user in real time.

[1284] Specific examples

[1285] For example, if there is milk, tomatoes, and carrots in the refrigerator, the device scans them and recognizes the label information. The data sent to the server includes the fact that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023. The server confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. However, at this point, the emotion engine detects from the user's voice and facial expression that they are feeling stressed. Based on this information, the server also suggests a recipe for a relaxing herbal soup that will help relieve stress.

[1286] Prompt Sentence Examples

[1287] Create a program that scans ingredients in the refrigerator and obtains their names and expiration dates using object recognition and OCR technology. Also, incorporate logic to suggest appropriate recipes based on the user's emotional state. The ingredient information will be sent to a server via Wi-Fi, where it will be analyzed and stored in a database. Also implement a function to send push notifications of special offers and recipes to the user's smartphone app.

[1288] The above is a specific example of how to implement the invention. This system streamlines food ingredient management and enables personalized recommendations that take into account the user's emotional state. It also makes shopping more efficient by utilizing information on nearby sales.

[1289] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1290] Program processing flow

[1291] Step 1:

[1292] The device automatically enters scanning mode when the refrigerator door is closed. It then activates the 360-degree camera and scans each shelf inside the refrigerator. Specifically, the camera takes a full picture of the inside of the refrigerator and acquires the resulting image (input data). This is then input into an object recognition algorithm to identify the location and appearance of each ingredient (data processing). OCR technology is used to read the label information (name, expiration date) on the ingredients. This information is temporarily stored in local memory (output data).

[1293] Step 2:

[1294] The device sends the scanned data stored in its local memory to a server via Wi-Fi. Specifically, the scanned data is packaged in JSON format (input data). The data includes the name of the ingredient, its expiration date, and its location. This data is then sent to the server via a Wi-Fi module (output data).

[1295] Step 3:

[1296] The server receives the scanned data sent from the device and stores it in a database. Specifically, it analyzes the received JSON data (input data) and organizes the names of ingredients, expiration dates, and location information by category (data calculation). The organized data is then stored in a database (output data). This allows the user to understand the status of their refrigerator.

[1297] Step 4:

[1298] Users provide voice input and facial recognition information through a smartphone app. Specifically, the smartphone's microphone and camera capture the user's voice data and facial expression data (input data). This information is then sent to the server (output data).

[1299] Step 5:

[1300] The server analyzes the voice data and facial expression data and identifies the user's emotional state using an emotion analysis engine. Specifically, the analysis engine determines whether the user is feeling stressed based on the voice and facial expression (data calculation). The emotional state is saved as internal data (output data).

[1301] Step 6:

[1302] The server proposes optimal recipes to users based on the stored ingredient information and emotional state. Specifically, it refers to the emotional state and ingredient data (input data) and selects the optimal recipe based on an algorithm (data processing). The selected recipe is saved as output data.

[1303] Step 7:

[1304] The server retrieves sales information from nearby retailers from an external database based on the user's location information. Specifically, the server sends the location information (input data) as a query to the external database to retrieve sales information (data calculation). The retrieved sales information is then stored on the server (output data).

[1305] Step 8:

[1306] The server notifies the user's smartphone app of the acquired sales information and suggested recipes. Specifically, it integrates the acquired sales information and optimal recipes (input data) into a push notification message (data processing), and then pushes this message to the smartphone app (output data).

[1307] Through these steps, the system streamlines the management of ingredients in the user's refrigerator and provides personalized recipe suggestions and sale information based on the user's emotional state.

[1308] (Application example 2)

[1309] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1310] Conventional food storage device item management systems manage item identification information and expiration dates, and provide users with sale information and usage instructions. However, they do not provide optimal usage suggestions or warnings or advice that take into account the user's emotional state. As a result, they are unable to fully alleviate the user's psychological burden and stress. This limits the user experience and the system's usefulness.

[1311] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes emotion recognition means for detecting the emotional state of the user, and warning means for providing appropriate warnings and advice based on the emotional state detected by the emotion recognition means. This makes it possible to suggest cooking recipes according to the user's emotional state, and to provide warnings and advice in real time.

[1312] A "food preservation device" is a device for preserving food at low temperatures, generally referring to a refrigerator or freezer.

[1313] "Imaging means" refers to a device for imaging an item, and includes a camera and a sensor.

[1314] "Image data" refers to visual information acquired by imaging means such as a camera or sensor.

[1315] "Recognition means" refers to a system or algorithm for analyzing image data and recognizing the identification information and shelf life of an item.

[1316] "Identification information" is information for identifying an item, such as the name or type of the item.

[1317] "Shelf life" is information that indicates the period during which food or goods can maintain their quality.

[1318] The "suggestion means" is a system that suggests to the user the most appropriate way to use the item based on the identification information and the expiration date obtained by the recognition means.

[1319] The "transmission means" is a communication means for transmitting the recognized product information to an external database.

[1320] The "acquisition means" is a system that acquires sale information of retailers in the user's neighborhood from an external database.

[1321] The "notification means" is a means for notifying the user of the acquired sale information or proposals, and includes a terminal such as a smartphone.

[1322] The "emotion recognition means" is a system that analyzes the user's voice and facial expressions to detect their emotional state.

[1323] The "warning means" is a system that provides appropriate warnings and advice to the user based on the emotional state detected by the emotion recognition means.

[1324] The present invention is a system that efficiently manages items stored in a food storage device and provides optimal usage instructions and nearby sale information that take into account the user's feelings. Specific embodiments for implementing the present invention will be described below.

[1325] System configuration

[1326] This system consists of eight main parts: imaging means (terminal), recognition means (server), emotion recognition means (server), warning means (server), suggestion means (server), acquisition means (server), transmission means (terminal), and notification means (server).

[1327] 1. Imaging means (terminal):

[1328] The terminal captures images of items using cameras and sensors installed inside the food storage device (e.g., refrigerator). The image data acquired by this imaging means becomes the basic data for acquiring the item's identification information and shelf life.

[1329] 2. Recognition means (server):

[1330] The image data sent from the terminal is analyzed to recognize the item's identification information (name, type) and expiration date. Optical character recognition (OCR) technology is used here to read the label information attached to the item.

[1331] 3. Emotion recognition means (server):

[1332] The system analyzes voice input and facial expression data captured by a camera to detect the user's emotional state. This emotion recognition means allows the system to grasp the user's emotional state in real time.

[1333] 4. Alert Method (Server):

[1334] Based on the emotional state detected by the emotion recognition means, the system provides appropriate warnings and advice to the user. For example, if the user is feeling stressed or anxious, the system will provide advice on how to relax or call attention to the situation.

[1335] 5. Proposed method (server):

[1336] The system suggests optimal recipes based on the item's identification information and expiration date obtained through the recognition method. It also takes into account the user's current emotional state to provide more personalized suggestions.

[1337] 6. Acquisition method (server):

[1338] The service retrieves sales information from vendors in the user's neighborhood from an external database, allowing the user to obtain information on how to purchase fresh ingredients at lower prices.

[1339] 7. Transmission means (terminal):

[1340] The recognized item information is sent to a server, mainly via network communication such as Wi-Fi.

[1341] 8. Notification Method (Server):

[1342] The recipes generated by the suggestion means and the sale information collected by the acquisition means are notified to the user's device such as a smartphone, etc. This includes a function to send push notifications in real time.

[1343] Processing and its concrete examples

[1344] The server analyzes the image data sent from the device and extracts the item's identification information and expiration date. Technologies used include OpenCV and OCR. Next, an emotion recognition algorithm is used to analyze the user's emotions using their voice and facial expression data.

[1345] As a concrete example, if there is milk, tomatoes, and carrots in the refrigerator, the device will take a picture of these items and send the image data to the server. The server will then obtain the item's identification information (e.g., milk, tomatoes, carrots) based on the image data and recognize their respective expiration dates (e.g., May 1, 2023, April 28, 2023). Based on this information, the proposed method will provide the optimal recipe (e.g., tomato cream soup).

[1346] Additionally, if the emotion recognition means detects that the user is feeling stressed, it will also suggest relaxing recipes (e.g., herbal soups) that can help relieve stress.

[1347] Prompt Sentence Examples

[1348] "Develop a system that monitors the user's emotions in real time using a camera while they are using smart glasses, and provides advice and safety information based on their emotional state. If the user is in a 'stressed' state, notify them of nearby safety and alert information. Please also provide example program code."

[1349] The present invention provides more personalized information and enhances user convenience by making suggestions and warnings that take into account the user's emotional state.

[1350] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1351] Step 1:

[1352] Input: A user opens the refrigerator, stores items, and then closes the refrigerator door.

[1353] Processing: The device's camera automatically activates and takes a 360-degree image of the items in the refrigerator.

[1354] Output: Image data of items in the refrigerator.

[1355] How it works: A camera scans the inside of the refrigerator, capturing the location and appearance of each item.

[1356] Step 2:

[1357] Input: Image data of items in the refrigerator.

[1358] Processing: The device temporarily stores the acquired image data in its local memory.

[1359] Output: Image data of the item stored in local memory.

[1360] Operation: Stores image data in the device's storage.

[1361] Step 3:

[1362] Input: Image data of the item stored in local memory.

[1363] Processing: The device sends the image data to the server via Wi-Fi.

[1364] Output: Image data sent to the server.

[1365] How it works: The device uploads image data to a server via a Wi-Fi connection.

[1366] Step 4:

[1367] Input: Image data sent to the server.

[1368] Processing: The server analyzes the received image data and uses object recognition algorithms to extract the item's identity (name, type) and shelf life.

[1369] Output: Item identification and shelf life data.

[1370] How it works: The server uses object recognition algorithms and OCR technology to read the label information on the item.

[1371] Step 5:

[1372] Input: Item identification and shelf life data.

[1373] Processing: The server analyzes the user's voice input and facial expression data to recognize emotions.

[1374] Output: Emotion recognition result (e.g. stress, relaxation, etc.).

[1375] How it works: The server uses emotion recognition algorithms to identify the user's emotions from audio and image data.

[1376] Step 6:

[1377] Input: Item identification, shelf life data, and emotion recognition results.

[1378] Processing: Based on this data, the server proposes the optimal usage method (recipe).

[1379] Output: Suggested recipe information.

[1380] How it works: The server searches for suitable recipes from the database and generates suggestions taking into account emotion recognition results.

[1381] Step 7:

[1382] Input: The user's location.

[1383] Processing: The server retrieves sales information from vendors in the user's vicinity from an external database.

[1384] Output:Sale information.

[1385] How it works: The server uses an external API to retrieve special offers based on the user's location.

[1386] Step 8:

[1387] Input: Suggested recipe information and special offer information.

[1388] Processing: The server notifies the user's smartphone app of this information.

[1389] Output: A notification that appears on the user's smartphone.

[1390] How it works: The server uses push notifications to send information to users in real time.

[1391] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1392] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1393] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1394] [Fourth embodiment]

[1395] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1396] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1397] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1398] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1399] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1400] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1401] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1402] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1403] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1404] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1405] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1406] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1407] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1408] This invention is a system that efficiently manages the items stored in a food storage device (e.g., a refrigerator) and provides users with information on optimal usage methods and nearby special sales based on the managed items. This system includes three elements: a "terminal" that captures images of the items stored in the refrigerator, a "server" that analyzes the captured image data and identifies the items and their expiration dates, and a "user" that notifies the user based on the identified information.

[1409] Basic system configuration

[1410] 1. Imaging means (terminal)

[1411] The terminal is installed inside the refrigerator and uses a 360-degree camera to capture images of the items inside the refrigerator.

[1412] Image data of the item is acquired and temporarily stored in local memory.

[1413] 2. Recognition means (server)

[1414] It receives image data sent from the terminal and uses an object recognition algorithm to analyze the item's identification information (name, type) and label information.

[1415] Optical character recognition (OCR) technology is used to recognize the expiration date printed on the product label.

[1416] 3. Proposed method (server)

[1417] Based on the recognized identity and shelf life, the system suggests the best way to use the item (e.g., a recipe).

[1418] Recipe selection also takes into account the user's food preferences and past history.

[1419] 4. Transmission means (terminal)

[1420] The recognized item information is transmitted from the terminal to the server.

[1421] The data is packaged in JSON format and sent over Wi-Fi.

[1422] 5. Acquisition method (server)

[1423] Get special offers from vendors near the user from an external database.

[1424] The acquired sale information is sorted and information relevant to the user is identified.

[1425] 6. Notification Method (Server)

[1426] The acquired sale information and suggested recipes are notified to the user's smartphone app.

[1427] Use push notifications to provide users with real-time information.

[1428] Specific processing of the program

[1429] Scan food in the refrigerator

[1430] The device automatically enters scanning mode when the refrigerator door is closed, activating its 360-degree camera to scan each shelf inside the refrigerator. The image data captured by the camera is analyzed using an object recognition algorithm to identify the location and appearance of each food item. OCR technology is used to read the food label information (name, expiration date). This information is stored in local memory and later sent to the server.

[1431] Scan data transmission and analysis

[1432] The device sends the collected information about ingredients to a server via Wi-Fi. The server analyzes the received data and stores it in a database. The names, expiration dates, and location information of ingredients are organized by category, allowing the user to understand the status of their refrigerator.

[1433] Optimal recipe suggestions

[1434] The server then suggests optimal recipes to the user based on the stored ingredient information. The server then references the user's food preferences and past history to search for and generate a list of relevant recipes. This recipe list is then sent to the user's smartphone app via push notification.

[1435] Get information on nearby sales

[1436] The server retrieves sales information from nearby retailers from an external database based on the user's location information. The retrieved sales information is matched with the information about ingredients in the user's refrigerator to identify related sales information. This sales information is also notified to the user's smartphone app.

[1437] Specific examples

[1438] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. It determines that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023, and sends this information to the server. The server then confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. Along with this suggestion, the user is also notified of a 20% off sale on tomatoes at a nearby supermarket. The user can check this information on their smartphone app and plan their cooking and shopping accordingly.

[1439] The above is a description of specific embodiments for carrying out the present invention.

[1440] The processing flow will be explained below.

[1441] Step 1:

[1442] The device automatically enters scanning mode when the refrigerator door is closed, activating its built-in 360-degree camera to capture images of each shelf inside the refrigerator.

[1443] Step 2:

[1444] The device analyzes the image data captured by the camera using an object recognition algorithm, which identifies the location and type of each food item in the refrigerator.

[1445] Step 3:

[1446] The device uses OCR (optical character recognition) technology to read the label information (name, expiration date) of each ingredient, and this information is temporarily stored in the device's local memory.

[1447] Step 4:

[1448] The device sends the acquired information to a server via Wi-Fi. The data is packaged in JSON format and encrypted.

[1449] Step 5:

[1450] The server analyzes the received data and stores it in a database, organizing the names of ingredients, expiration dates, and location information by category.

[1451] Step 6:

[1452] The server periodically scans the database for ingredients that are nearing their expiration date and generates an alert for any ingredients that are nearing their expiration date.

[1453] Step 7:

[1454] The server refers to the user's preferences and past history, suggests recipes using ingredients that are close to their expiration date, searches for relevant recipes in the recipe database, and generates a list.

[1455] Step 8:

[1456] The server notifies the user of the generated recipe list and alert information via the user's smartphone app using the push notification function.

[1457] Step 9:

[1458] Users receive notifications on their smartphone app, which can include audio alerts and banners.

[1459] Step 10:

[1460] The user opens the app to check the ingredients in the refrigerator and see the suggested recipes. The app displays a list of ingredients, expiration dates, and suggested recipes.

[1461] Step 11:

[1462] The server retrieves sales information from nearby retailers from an external database based on the user's location information, and obtains the latest sales data through the supermarket's API.

[1463] Step 12:

[1464] The server analyzes the acquired sales information and identifies items that match the food information in the user's refrigerator. The identified sales information is organized and added to a recommendation list.

[1465] Step 13:

[1466] The server sends the recommendation list to the user's smartphone app, where the user can check the sale information through notifications.

[1467] Step 14:

[1468] Users can check the sale information within the app, which includes store addresses, discount rates, product details, and more.

[1469] Step 15:

[1470] Users can add ingredients they need to a list and create a shopping list, which can be edited and saved within the app.

[1471] The above is the processing flow of the specific program of the system. The specific operation has been explained in detail for each step.

[1472] Example 1

[1473] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1474] In modern households, food waste is a problem due to the inability to properly manage items stored in food storage devices. Other issues include overlooking items that are approaching their expiration date and taking time to find the perfect recipe. Furthermore, to efficiently do daily shopping, it is necessary to quickly obtain information about sales from nearby retailers, but current systems are not able to adequately handle this.

[1475] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1476] In this invention, the server includes an imaging means for capturing images of items in the food preservation device, a recognition means for analyzing the image data captured by the imaging means to recognize the identification information and shelf life of the items, a suggestion means for suggesting optimal usage methods for the items based on the identification information and shelf life, a transmission means for transmitting the recognized item information to an external database, a acquisition means for acquiring sales information from retailers near the user from the external database, a notification means for notifying the user of the sales information acquired by the acquisition means, a means for the imaging means to scan the interior of the refrigerator with a 360-degree camera triggered by a door sensor, and a means for the notification means to provide information to the user's mobile information terminal using a push notification function. This enables expiration date management of items in the food preservation device, suggestions for optimal usage methods, and efficient shopping.

[1477] A "food preservation device" is a device for preserving food at low temperatures, and mainly refers to a refrigerator or freezer.

[1478] "Imaging means" refers to a device for photographing items in the food storage device, and includes a camera and a sensor.

[1479] The "recognition means" refers to a technique or device for analyzing image data obtained by the imaging means and recognizing the identification information and shelf life of an item.

[1480] The "suggestion means" is a function or device for suggesting the most suitable method of use related to the item based on the identification information and the expiration date obtained by the recognition means.

[1481] The "transmitting means" refers to a function or device for transmitting the recognized product information to an external database.

[1482] The "acquisition means" refers to a function or device for acquiring sale information from an external database.

[1483] The "notification means" is a function or device for notifying the user of the acquired sale information.

[1484] A "door sensor" is a sensor that detects whether the refrigerator door is open or closed.

[1485] A "360-degree camera" is a camera that can capture images in all directions.

[1486] "Mobile information terminal" refers to a portable information processing device such as a mobile phone, smartphone, or tablet.

[1487] The "push notification function" is a function that delivers information from a server to a user's mobile information terminal in real time.

[1488] A "food management system" is a system that efficiently manages the items in a refrigerator and provides optimal usage methods and special sale information based on that information.

[1489] This system efficiently manages the items stored in a food storage device and provides users with information on optimal usage methods and nearby special sales based on that information. This system includes three elements: a terminal that captures images of the items stored in the refrigerator, a server that analyzes the captured image data and identifies the items and their expiration dates, and a user interface that notifies the user based on the identified information.

[1490] Specifically, the following hardware and software are used:

[1491] Imaging means (terminal)

[1492] The device is installed inside the refrigerator and captures images of the items inside using a 360-degree camera. Image data of the items is acquired and temporarily stored in local memory. When the door sensor detects that the refrigerator door is closed, the 360-degree camera is activated and begins scanning.

[1493] Recognition means (server)

[1494] The server receives the image data sent from the device and analyzes the item's identification information (name, type) and label information using an object recognition algorithm. Specifically, it uses a TensorFlow-based object recognition model. It also uses Tesseract as OCR technology to recognize the expiration date printed on the item's label.

[1495] Proposal method (server)

[1496] The server then suggests optimal ways to use the item (e.g., recipes) based on the recognized identification information and expiration date. The server also takes into account the user's food preferences and past history when selecting recipes. Based on the stored ingredient information, the server selects and lists the optimal recipes.

[1497] Transmission means (terminal)

[1498] The device sends the recognized item information to the server, packaged in JSON format, and transmitted via Wi-Fi.

[1499] Acquisition method (server)

[1500] The server retrieves sales information from nearby retailers from an external database, sorts the sales information, and identifies those relevant to the user. It then uses GeoAPI to obtain the user's location and sends an API request to the external database based on that location.

[1501] Notification method (server)

[1502] The server then notifies the user of the sale information and suggested recipes in real time via the smartphone app using Firebase Cloud Messaging (FCM).

[1503] Specific examples

[1504] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. It determines that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023, and sends this information to the server. The server then confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. Along with this suggestion, the user is also notified of a 20% off sale on tomatoes at a nearby supermarket. The user can check this information on their smartphone app and plan their cooking and shopping accordingly.

[1505] Example prompts for generative AI models

[1506] Here is an example prompt for the generative AI model to write specific steps for this system:

[1507] Please explain in detail the following system steps, including examples:

[1508] This system efficiently manages the contents of a food storage device (refrigerator) and provides users with information on optimal use and nearby special offers. It works by following these steps:

[1509] 1. Initiating a refrigerator scan: The device detects that the refrigerator door is closed, and the 360-degree camera activates to capture images of each shelf.

[1510] 2. Acquisition and temporary storage of image data: The image data captured by the camera is acquired and temporarily stored in local memory.

[1511] 3. Sending image data: The device sends the acquired image data to the server via Wi-Fi.

[1512] 4. Image data analysis: The server analyzes the received image data and extracts ingredient information.

[1513] 5. Storing and organizing ingredient information: The server stores and organizes ingredient information in a database.

[1514] 6. Optimal recipe suggestion: The server suggests optimal recipes to the user based on the stored ingredient information.

[1515] 7. Get nearby sales information: The server retrieves nearby sales information from an external database and selects the most relevant ones.

[1516] 8. User notification: The server pushes suggested recipes and special offers to the user's smartphone app.

[1517] For example, imagine you have milk, tomatoes, and carrots in your refrigerator. Your device scans them and sends that information to a server. The server sees that the tomatoes are nearing their expiration date and sends you a recipe for cream of tomato soup and a sale on tomatoes.

[1518] The above is a specific embodiment for carrying out the invention.

[1519] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1520] Step 1: Start the fridge scan

[1521] When the door sensor detects that the refrigerator door is closed, the device automatically enters scanning mode. The 360-degree camera starts up and takes pictures of each shelf in turn. The input is the sensor signal indicating that the door is closed, and the output is the image data captured by the camera.

[1522] Specific operation: After the door sensor detects the door closing, the camera waits 5 seconds and then starts shooting in all directions, scanning each shelf 10 degrees at a time, capturing a total of 36 image data.

[1523] Step 2: Acquire and temporarily save image data

[1524] The image data captured by the camera is acquired and temporarily stored in local memory. The input is image data taken by a 360-degree camera, and the output is a high-resolution JPEG image file stored in local memory.

[1525] How it works: Each scanned image is saved to local memory in real time and stored in a temporary storage area. The image is saved in JPEG format and a timestamp is added to each image.

[1526] Step 3: Sending image data

[1527] The device sends image data stored in its local memory to the server via Wi-Fi. The data is packaged in JSON format and sent as high-priority data. The input is the image data in the local memory, and the output is the JSON package sent to the server.

[1528] Specific behavior: The device checks for a Wi-Fi connection, sends data in 5MB chunks, and clears local memory after the transmission is complete.

[1529] Step 4: Analyzing the image data

[1530] The server analyzes the received image data and uses an object recognition algorithm to extract the identification information (name, type) of each ingredient. It also uses OCR technology to read the expiration date printed on the ingredient label. The input is a JSON package sent from the device, and the output is data containing the identification information and expiration date.

[1531] Specific operation: The server uses a TensorFlow-based object recognition model and only recognizes results with a confidence level of 0.9 or higher. OCR analysis is performed using the Tesseract library, and label data is extracted as text data.

[1532] Step 5: Save and organize ingredient information

[1533] The server stores the recognition results in a database and organizes the names, types, shelf life, and location information of ingredients by category. The input is data including identification information and shelf life, and the output is a database organized by category.

[1534] How it works: Recognition results are stored in a MySQL database and immediately indexed. Each item is categorized for efficient searching.

[1535] Step 6: Propose the best recipe

[1536] The server compares the stored ingredient information with the user's preferences and past history to suggest optimal recipes. The input is an organized database and the user's preferences and history data, and the output is a list of recommended recipes.

[1537] Specific operation: The server refers to the user profile and uses a matching algorithm to select three candidate recipes, generate a list of recommended recipes, and send it to the next notification step.

[1538] Step 7: Get local sales information

[1539] The server retrieves nearby sales information from an external retailer database based on the user's location information. The input is the user's location information and sale information from the external database, and the output is highly relevant sale information.

[1540] How it works: The server uses GeoAPI to obtain the user's location information and sends an API request to an external database based on that information. The obtained sale information is matched with food information, and the most relevant information is selected.

[1541] Step 8: Notify users

[1542] The server sends suggested recipes and sales information to the user's mobile device using push notifications. The input is a list of suggested recipes and related sales information, and the output is the notification sent to the user's mobile device.

[1543] How it works: The server uses Firebase Cloud Messaging (FCM) to send real-time notifications to users' smartphones, including detailed recipe instructions and special offers, designed to allow users to take immediate action.

[1544] (Application example 1)

[1545] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1546] Conventional food preservation devices have the problem that it is difficult to manage the inventory of food in the refrigerator and grasp the expiration date, resulting in a lot of food going to waste. Furthermore, when users shop in physical stores, they are unable to create an optimal purchasing plan that is linked to existing inventory, resulting in duplicate or unnecessary purchases. This leads to problems of wasting resources and increasing the burden on household finances.

[1547] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1548] In this invention, the server includes an imaging means for imaging items in the food preservation device, a recognition means for analyzing image data obtained by the imaging means to recognize the identification information and shelf life of the items, a suggestion means for suggesting optimal usage methods for the items based on the identification information and shelf life, a transmission means for transmitting the recognized item information to an external database, an acquisition means for acquiring sales information from retailers near the user from the external database, a notification means for notifying the user of the sales information acquired by the acquisition means, and a shopping support means for providing the recognized item information and sales information and suggesting an optimal purchasing plan when the user is shopping at a physical store. This allows the user to shop efficiently while referring to inventory information in the refrigerator in real time.

[1549] A "food preservation device" is a device used to maintain the quality of food and preserve it for a long period of time.

[1550] "Goods" refers to food and beverages stored in the refrigerator.

[1551] The "imaging means" is a device including a camera and a sensor for acquiring image data of an object.

[1552] "Image data" refers to visual information acquired by an imaging means expressed in digital form.

[1553] The "recognition means" is a device or system that analyzes the acquired image data and extracts the identification information and storage period of the item.

[1554] "Identification information" refers to information for identifying the name, type, characteristics, etc. of an item.

[1555] "Shelf life" is information that indicates the date by which an item can be consumed.

[1556] The "suggestion means" is a device or system that suggests to the user the most appropriate way to use the product based on the identification information and the shelf life.

[1557] The "transmitting means" is a device or system for transmitting the recognized product information to an external database.

[1558] "External database" refers to a database that stores and manages product information and special sale information and provides necessary information.

[1559] The "acquisition means" is a device or system for acquiring sale information of retailers in the user's neighborhood from an external database.

[1560] The "notification means" is a device or system for notifying the user of the acquired sale information and suggested usage methods.

[1561] A "shopping support means" is a device or system that provides recognized product information and sale information to a user when shopping at a physical store, and suggests an optimal purchasing plan.

[1562] "Brick and mortar store" refers to a store that sells products in person at a physical location.

[1563] The present invention is a system that efficiently manages the items in a food storage device and provides users with optimal usage instructions and nearby sales information based on the managed items. This system is composed of the following elements:

[1564] 1. Imaging means (terminal)

[1565] The device is installed inside the refrigerator and captures images of the items inside using a 360-degree camera. The image data captured by the camera is temporarily stored in local memory.

[1566] The specific hardware used is a 360-degree camera (e.g., Ricoh Theta) and a sensor module (e.g., Raspberry Pi).

[1567] 2. Recognition means (server)

[1568] The server receives the image data sent from the terminal and analyzes the identification information and storage period of the item.

[1569] Image recognition uses object recognition algorithms (e.g., YOLO) and optical character recognition (OCR) technology (e.g., Tesseract OCR) to identify the item's identity and shelf life.

[1570] 3. Proposed method (server)

[1571] Based on the recognized product information, the server suggests optimal usage methods, such as cooking recipes, to the user.

[1572] The system selects appropriate recipes based on the user's food preferences and past usage history.

[1573] 4. Transmission means (terminal)

[1574] The device packages the recognized item information in JSON format and sends it to the server via Wi-Fi.

[1575] 5. Acquisition method (server)

[1576] The server retrieves sales information from an external database for merchants in the user's vicinity.

[1577] The sale information is matched with information about items whose expiration dates are approaching for the user.

[1578] 6. Notification Method (Server)

[1579] The server then notifies the user of the acquired sale information and suggested recipes via push notification to their smartphone.

[1580] 7. Shopping Support Methods (User)

[1581] When users shop at a physical store, the smartphone app displays food items with upcoming expiration dates, inventory information, and special offers.

[1582] This allows users to shop efficiently and reduce waste.

[1583] Specific operation example

[1584] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. It determines that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023, and sends this information to the server. The server then confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. Along with this suggestion, the user is also notified of a 20% off sale on tomatoes at a nearby supermarket. The user can check this information on their smartphone app and plan their cooking and shopping accordingly.

[1585] Prompt Sentence Examples

[1586] "The system analyzes the image of food inventory in the refrigerator and identifies foods that are nearing their expiration date. While the user is shopping in a physical store, it takes into account the refrigerator's inventory and sales information and suggests the optimal shopping list in real time. It also includes recipes based on ingredients that are nearing their expiration date."

[1587] The above is a description of specific embodiments for carrying out the present invention.

[1588] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1589] Step 1:

[1590] The device starts scanning the inside of the refrigerator. When the refrigerator door is closed, the 360-degree camera automatically starts up and scans the interior. The image data acquired by the image capture is temporarily stored in local memory.

[1591] Input: Current state of the refrigerator

[1592] Data processing: 360-degree camera imaging

[1593] Output: 360-degree image data of the inside of the refrigerator

[1594] Step 2:

[1595] The device sends the scanned data to the server, and the image data stored in the local memory is packaged in JSON format and sent to the cloud server via Wi-Fi.

[1596] Input: 360-degree image data

[1597] Data processing: Packaging into JSON format

[1598] Output: Image data sent to the server

[1599] Step 3:

[1600] The server analyzes the received data. The cloud server analyzes the received image data and identifies the item's identity using an object recognition algorithm (e.g., YOLO). It also uses OCR technology (e.g., Tesseract OCR) to extract label information (name, expiration date), and stores this information in a database.

[1601] Input: Image data sent to the server

[1602] Data processing: object recognition, OCR analysis, saving to database

[1603] Output: Identification information and retention period

[1604] Step 4:

[1605] The server proposes the optimal way to use the product. Based on the recognized identification information and expiration date, the server proposes the optimal way to use the product, such as cooking recipes, to the user. The server selects recipes taking into consideration the user's food preferences and past history.

[1606] Input: Identification information, retention period, user preferences and history

[1607] Data processing: Recipe selection

[1608] Output: Suggested recipe information

[1609] Step 5:

[1610] The device receives the suggested recipe information from the server and notifies the user's smartphone. The suggested recipe information is sent to the user's smartphone via push notification.

[1611] Input: Suggested recipe information

[1612] Data processing: Notification to smartphone

[1613] Output: Recipe notification to smartphone

[1614] Step 6:

[1615] The server retrieves sale information from an external database, and based on the user's location information, retrieves sale information from nearby retailers and matches it with items with an approaching expiration date.

[1616] Input: User's location

[1617] Data processing: Acquiring and matching sale information

[1618] Output: Related deals

[1619] Step 7:

[1620] The server notifies the user of the special sale information it has acquired, and displays the information on the smartphone so that the user can refer to it when shopping at a physical store.

[1621] Input: Related sales information

[1622] Data processing: Notification to smartphone

[1623] Output: Special sale information notification to smartphone

[1624] Step 8:

[1625] Users can use a smartphone app to receive purchasing assistance in physical stores. The app displays items with upcoming expiration dates and sales information, and suggests optimal purchasing plans. This allows users to shop efficiently.

[1626] Input: Information notified to your smartphone

[1627] Data processing: Displaying expiration dates and special sale information, suggesting purchasing plans

[1628] Output: Purchase support information for the user

[1629] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1630] This invention is a system that efficiently manages items in a food storage device (e.g., a refrigerator) and provides users with optimal usage methods and nearby sale information that also take into account the user's emotions. This system includes three elements: a "terminal" that captures images of items in the refrigerator, a "server" that analyzes the image data and the user's emotions and makes suggestions based on the item's identification information, shelf life, and user emotions, and a "user" that notifies the user of the information.

[1631] Basic system configuration

[1632] 1. Imaging means (terminal)

[1633] The terminal is installed inside the refrigerator and uses a 360-degree camera to capture images of the items inside the refrigerator.

[1634] Image data of the item is acquired and temporarily stored in local memory.

[1635] 2. Recognition means (server)

[1636] It receives image data sent from the terminal and uses an object recognition algorithm to analyze the item's identification information (name, type) and label information.

[1637] OCR (optical character recognition) technology is used to recognize the expiration date printed on the product label.

[1638] 3. Proposed method (server)

[1639] Based on the recognized identity and shelf life, the system suggests the best way to use the item (e.g., a recipe).

[1640] The recipe selection incorporates an emotion engine that reflects the user's emotions, and by analyzing the user's current emotional state, it makes more personalized suggestions.

[1641] 4. Emotion engine (server)

[1642] It includes sensors that detect emotions from the user's voice input and facial expressions, and analyzes the information acquired by these sensors to identify the user's emotional state.

[1643] Based on the identified emotion information, a suggestion suited to the emotion of the user is made.

[1644] 5. Transmission means (terminal)

[1645] The recognized item information is transmitted from the terminal to the server.

[1646] The data is packaged in JSON format and sent over Wi-Fi.

[1647] 6. Acquisition method (server)

[1648] Get special offers from vendors near the user from an external database.

[1649] The acquired sale information is sorted and information relevant to the user is identified.

[1650] 7. Notification Method (Server)

[1651] The acquired sale information and suggested recipes are notified to the user's smartphone app.

[1652] Use push notifications to provide users with real-time information.

[1653] Specific processing of the program

[1654] Scan food in the refrigerator

[1655] The device automatically enters scanning mode when the refrigerator door is closed, activating its 360-degree camera to scan each shelf inside the refrigerator. The image data captured by the camera is analyzed using an object recognition algorithm to identify the location and appearance of each food item. OCR technology is used to read the food label information (name, expiration date). This information is stored in local memory and later sent to the server.

[1656] Scan data transmission and analysis

[1657] The device sends the collected information about ingredients to a server via Wi-Fi. The server analyzes the received data and stores it in a database. The names, expiration dates, and location information of ingredients are organized by category, allowing the user to understand the status of their refrigerator.

[1658] Optimal recipe suggestions

[1659] The server suggests optimal recipes to the user based on the stored ingredient information. Furthermore, it uses an emotion engine to select recipes that take into account the user's current emotional state. The server analyzes the user's voice input and facial recognition information to identify the user's emotional state. For example, if the user is feeling stressed, it suggests easy-to-make recipes or dishes that will help them relax.

[1660] Example: Emotion-based recipe suggestions

[1661] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. The data sent to the server includes the expiration date of the milk (May 1, 2023) and the expiration date of the tomatoes (April 28, 2023).

[1662] The server determines that the tomatoes are close to expiring and suggests a recipe for tomato cream soup. However, the emotion engine then detects that the user is feeling stressed from their voice and facial expressions. Based on this information, the server also suggests a recipe for a relaxing herbal soup that will help relieve stress.

[1663] Get information on nearby sales

[1664] The server retrieves sales information from nearby retailers from an external database based on the user's location information. It then sorts the retrieved information and identifies sales information relevant to the user. This sales information is also notified to the user's smartphone app.

[1665] Example: Notifications with sentiment and sales information

[1666] The smartphone app notifies users of a 20% off sale on tomatoes at a nearby supermarket along with recipes based on their emotions (e.g., herb soup). Because the app responds to the user's emotional state, it also provides suggestions for relaxation along with the recipe. Utilizing this information, users can plan their cooking and shopping accordingly.

[1667] The above is a description of a specific embodiment for carrying out the present invention. This system takes into account the user's emotions and makes more personalized suggestions, thereby improving the convenience of ingredient management and cooking.

[1668] The processing flow will be explained below.

[1669] Step 1:

[1670] The device automatically enters scanning mode when the refrigerator door is closed, activating the 360-degree camera and capturing images of each shelf inside the refrigerator.

[1671] Step 2:

[1672] The device temporarily stores image data captured by the camera in its local memory. The captured data includes all items in the refrigerator.

[1673] Step 3:

[1674] The device analyzes the image data and uses object recognition algorithms to identify the location and type of each item, thereby determining what is stored in the refrigerator.

[1675] Step 4:

[1676] The terminal uses OCR (optical character recognition) technology to read the label information (name, expiration date) of each item and stores this information in local memory.

[1677] Step 5:

[1678] The device sends the acquired ingredient information to a server via Wi-Fi, where the data is packaged in JSON format and encrypted.

[1679] Step 6:

[1680] The server analyzes the received data and stores it in a database. It organizes the names of ingredients, expiration dates, and location information by category, and keeps track of the status of the user's refrigerator.

[1681] Step 7:

[1682] The server periodically scans the database for ingredients that are nearing their expiration date and generates an alert for any ingredients that are nearing their expiration date.

[1683] Step 8:

[1684] The server activates the emotion engine and uses sensors to capture the user's voice input and facial expressions, thereby detecting the user's current emotional state.

[1685] Step 9:

[1686] The server selects recipe suggestions based on the user's emotional state. For example, if the user is feeling stressed, it will suggest recipes with a relaxing effect.

[1687] Step 10:

[1688] The server generates an optimal recipe list based on the stored ingredient information and emotion information, including suggestions based on ingredient usage and emotion.

[1689] Step 11:

[1690] The server notifies the user's smartphone app of the generated recipe list and alert information using the push notification function.

[1691] Step 12:

[1692] Users receive a notification via a smartphone app that displays a list of ingredients, expiration dates, and suggested recipes based on their emotions.

[1693] Step 13:

[1694] The server retrieves sales information from nearby retailers from an external database based on the user's location information, and obtains the latest sales data through the supermarket's API.

[1695] Step 14:

[1696] The server analyzes the sales information and identifies items that match the food information in the user's refrigerator. The sales information is organized into items relevant to the user.

[1697] Step 15:

[1698] The server sends the sale information to the user's smartphone app, where the user can check the sale information through notifications.

[1699] Step 16:

[1700] Users can check the sale information within the app, which includes store addresses, discount rates, product details, and more.

[1701] Step 17:

[1702] Users can add ingredients they need to a list and create a shopping list, which can be edited and saved within the app.

[1703] Examples:

[1704] For example, if a user has milk, tomatoes, and carrots in their refrigerator, the device scans them and recognizes the label information. The data sent to the server includes the expiration date of the milk (May 1, 2023) and the expiration date of the tomatoes (April 28, 2023). The server determines that the tomatoes are close to expiring and uses an emotion engine to detect the user's current emotional state. For example, if a user is feeling stressed, the device might suggest a recipe for a relaxing herbal soup and also notify them of a 20% off sale on tomatoes at a nearby supermarket.

[1705] The above is the specific processing flow of the system that combines the emotion engine. We have explained the detailed operation of each step.

[1706] Example 2

[1707] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1708] In recent years, managing items in food storage devices has become extremely complicated, leading to problems such as users often overlooking expiration dates and being unable to use food efficiently. Furthermore, there is a lack of recipe suggestions that take into account the user's emotions and stress levels, making it difficult to suggest the most suitable dishes for the user. Another problem is that users are unable to effectively utilize sales information from nearby retailers.

[1709] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes an imaging means for capturing images of items in the food storage device, a recognition means for analyzing the image data captured by the imaging means to recognize the item's identification information and expiration date, a suggestion means for suggesting optimal usage methods for the items based on the identification information and expiration date, an emotion analysis means for analyzing the user's emotional state, a suggestion means for making suggestions appropriate to the user's emotions based on the emotion analysis means, a transmission means for transmitting the recognized item information to an external database, an acquisition means for acquiring sales information from retailers near the user from the external database, and a notification means for notifying the user of the sales information acquired by the acquisition means. This allows the user to efficiently manage food, avoid overlooking expiration dates, and receive optimal recipe suggestions based on their emotional state. It also allows the user to effectively utilize information about local sales.

[1710] A "food preservation device" is a device for preserving food at a constant temperature and humidity.

[1711] The "imaging means" is a means for taking a photograph of the items in the food storage device using a device such as a camera.

[1712] "Image data" refers to data of still images or moving images obtained by an imaging means.

[1713] The "recognition means" is a means for analyzing image data and identifying the identification information and storage period of an item.

[1714] "Identification information" is information for identifying the name and type of an item.

[1715] A "shelf life" is the date by which an item is properly stored.

[1716] The "suggestion means" is a means for providing the user with the optimum method of using the product (for example, a recipe) based on the identification information and the shelf life.

[1717] The "emotion analysis means" is a means for analyzing the user's emotional state through voice input or facial recognition.

[1718] The "transmitting means" is a means for transmitting the recognized product information to an external database.

[1719] The "acquisition means" is a means for acquiring sale information of retailers in the user's neighborhood from an external database.

[1720] The "notification means" is a means for notifying the user of the acquired sale information.

[1721] An "external database" is a database that can be accessed via the Internet and that stores various product information and special sale information.

[1722] "Special sale information" is information about discounts and campaigns being carried out by retailers in the user's neighborhood.

[1723] System basic configuration and operation

[1724] This invention is a system that efficiently manages items stored in a food storage device and provides users with information on optimal usage methods and nearby special sales, taking into account the user's emotions. This system includes three elements: a "terminal" that captures images of the items stored in the food storage device, a "server" that analyzes the captured image data and the user's emotions and makes suggestions based on the item's identification information, shelf life, and user emotions, and a "user" that notifies the user of the information.

[1725] Hardware and software used

[1726] Hardware: 360-degree camera installed inside the refrigerator, Wi-Fi module

[1727] Software: Object recognition algorithms, OCR technology, sentiment analysis engine, push notification system

[1728] Specific operation flow

[1729] Scan food in the refrigerator

[1730] The device automatically enters scanning mode when the refrigerator door is closed. It activates the 360-degree camera to scan each shelf in the refrigerator and capture images of the items in the food storage unit. The image data captured by the camera is analyzed using an object recognition algorithm to identify the location and appearance of each food item. OCR technology is used to read the food label information (name, expiration date). This information is temporarily stored in the device's local memory.

[1731] Scan data transmission and analysis

[1732] The device sends the collected food information to a server via Wi-Fi. The sent data is packaged in JSON format and includes information such as the food name, expiration date, and location. The server analyzes the received data, stores the food name, expiration date, and location information in a database, and organizes them by category to understand the status of the user's refrigerator.

[1733] User sentiment analysis

[1734] The server analyzes the user's voice input and facial recognition information and uses an emotion analysis engine to determine the user's emotional state: whether the user is stressed or relaxed.

[1735] Optimal recipe suggestions

[1736] The server then suggests optimal recipes based on the stored ingredient information and the user's emotional state, obtained from an emotion analysis engine. For example, if the expiration date of tomatoes is approaching, the server will suggest recipes that use tomatoes. If the user is looking to relax, the server will also suggest a recipe for a relaxing herbal soup.

[1737] Get information on nearby sales

[1738] The server retrieves sales information from nearby retailers from an external database based on the user's location information, and identifies the retrieved sales information as relevant to the user.

[1739] User Notification

[1740] The server notifies the user of the acquired sales information and suggested recipes via the smartphone app, using the push notification function to provide the information to the user in real time.

[1741] Specific examples

[1742] For example, if there is milk, tomatoes, and carrots in the refrigerator, the device scans them and recognizes the label information. The data sent to the server includes the fact that the milk's expiration date is May 1, 2023, and the tomatoes' expiration date is April 28, 2023. The server confirms that the tomatoes' expiration date is approaching and suggests a recipe for tomato cream soup. However, at this point, the emotion engine detects from the user's voice and facial expression that they are feeling stressed. Based on this information, the server also suggests a recipe for a relaxing herbal soup that will help relieve stress.

[1743] Prompt Sentence Examples

[1744] Create a program that scans ingredients in the refrigerator and obtains their names and expiration dates using object recognition and OCR technology. Also, incorporate logic to suggest appropriate recipes based on the user's emotional state. The ingredient information will be sent to a server via Wi-Fi, where it will be analyzed and stored in a database. Also implement a function to send push notifications of special offers and recipes to the user's smartphone app.

[1745] The above is a specific example of how to implement the invention. This system streamlines food ingredient management and enables personalized recommendations that take into account the user's emotional state. It also makes shopping more efficient by utilizing information on nearby sales.

[1746] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1747] Program processing flow

[1748] Step 1:

[1749] The device automatically enters scanning mode when the refrigerator door is closed. It then activates the 360-degree camera and scans each shelf inside the refrigerator. Specifically, the camera takes a full picture of the inside of the refrigerator and acquires the resulting image (input data). This is then input into an object recognition algorithm to identify the location and appearance of each ingredient (data processing). OCR technology is used to read the label information (name, expiration date) on the ingredients. This information is temporarily stored in local memory (output data).

[1750] Step 2:

[1751] The device sends the scanned data stored in its local memory to a server via Wi-Fi. Specifically, the scanned data is packaged in JSON format (input data). The data includes the name of the ingredient, its expiration date, and its location. This data is then sent to the server via a Wi-Fi module (output data).

[1752] Step 3:

[1753] The server receives the scanned data sent from the device and stores it in a database. Specifically, it analyzes the received JSON data (input data) and organizes the names of ingredients, expiration dates, and location information by category (data calculation). The organized data is then stored in a database (output data). This allows the user to understand the status of their refrigerator.

[1754] Step 4:

[1755] Users provide voice input and facial recognition information through a smartphone app. Specifically, the smartphone's microphone and camera capture the user's voice data and facial expression data (input data). This information is then sent to the server (output data).

[1756] Step 5:

[1757] The server analyzes the voice data and facial expression data and identifies the user's emotional state using an emotion analysis engine. Specifically, the analysis engine determines whether the user is feeling stressed based on the voice and facial expression (data calculation). The emotional state is saved as internal data (output data).

[1758] Step 6:

[1759] The server proposes optimal recipes to users based on the stored ingredient information and emotional state. Specifically, it refers to the emotional state and ingredient data (input data) and selects the optimal recipe based on an algorithm (data processing). The selected recipe is saved as output data.

[1760] Step 7:

[1761] The server retrieves sales information from nearby retailers from an external database based on the user's location information. Specifically, the server sends the location information (input data) as a query to the external database to retrieve sales information (data calculation). The retrieved sales information is then stored on the server (output data).

[1762] Step 8:

[1763] The server notifies the user's smartphone app of the acquired sales information and suggested recipes. Specifically, it integrates the acquired sales information and optimal recipes (input data) into a push notification message (data processing), and then pushes this message to the smartphone app (output data).

[1764] Through these steps, the system streamlines the management of ingredients in the user's refrigerator and provides personalized recipe suggestions and sale information based on the user's emotional state.

[1765] (Application example 2)

[1766] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1767] Conventional food storage device item management systems manage item identification information and expiration dates, and provide users with sale information and usage instructions. However, they do not provide optimal usage suggestions or warnings or advice that take into account the user's emotional state. As a result, they are unable to fully alleviate the user's psychological burden and stress. This limits the user experience and the system's usefulness.

[1768] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes emotion recognition means for detecting the emotional state of the user, and warning means for providing appropriate warnings and advice based on the emotional state detected by the emotion recognition means. This makes it possible to suggest cooking recipes according to the user's emotional state, and to provide warnings and advice in real time.

[1769] A "food preservation device" is a device for preserving food at low temperatures, generally referring to a refrigerator or freezer.

[1770] "Imaging means" refers to a device for imaging an item, and includes a camera and a sensor.

[1771] "Image data" refers to visual information acquired by imaging means such as a camera or sensor.

[1772] "Recognition means" refers to a system or algorithm for analyzing image data and recognizing the identification information and shelf life of an item.

[1773] "Identification information" is information for identifying an item, such as the name or type of the item.

[1774] "Shelf life" is information that indicates the period during which food or goods can maintain their quality.

[1775] The "suggestion means" is a system that suggests to the user the most appropriate way to use the item based on the identification information and the expiration date obtained by the recognition means.

[1776] The "transmission means" is a communication means for transmitting the recognized product information to an external database.

[1777] The "acquisition means" is a system that acquires sale information of retailers in the user's neighborhood from an external database.

[1778] The "notification means" is a means for notifying the user of the acquired sale information or proposals, and includes a terminal such as a smartphone.

[1779] The "emotion recognition means" is a system that analyzes the user's voice and facial expressions to detect their emotional state.

[1780] The "warning means" is a system that provides appropriate warnings and advice to the user based on the emotional state detected by the emotion recognition means.

[1781] The present invention is a system that efficiently manages items stored in a food storage device and provides optimal usage instructions and nearby sale information that take into account the user's feelings. Specific embodiments for implementing the present invention will be described below.

[1782] System configuration

[1783] This system consists of eight main parts: imaging means (terminal), recognition means (server), emotion recognition means (server), warning means (server), suggestion means (server), acquisition means (server), transmission means (terminal), and notification means (server).

[1784] 1. Imaging means (terminal):

[1785] The terminal captures images of items using cameras and sensors installed inside the food storage device (e.g., refrigerator). The image data acquired by this imaging means becomes the basic data for acquiring the item's identification information and shelf life.

[1786] 2. Recognition means (server):

[1787] The image data sent from the terminal is analyzed to recognize the item's identification information (name, type) and expiration date. Optical character recognition (OCR) technology is used here to read the label information attached to the item.

[1788] 3. Emotion recognition means (server):

[1789] The system analyzes voice input and facial expression data captured by a camera to detect the user's emotional state. This emotion recognition means allows the system to grasp the user's emotional state in real time.

[1790] 4. Alert Method (Server):

[1791] Based on the emotional state detected by the emotion recognition means, the system provides appropriate warnings and advice to the user. For example, if the user is feeling stressed or anxious, the system will provide advice on how to relax or call attention to the situation.

[1792] 5. Proposed method (server):

[1793] The system suggests optimal recipes based on the item's identification information and expiration date obtained through the recognition method. It also takes into account the user's current emotional state to provide more personalized suggestions.

[1794] 6. Acquisition method (server):

[1795] The service retrieves sales information from vendors in the user's neighborhood from an external database, allowing the user to obtain information on how to purchase fresh ingredients at lower prices.

[1796] 7. Transmission means (terminal):

[1797] The recognized item information is sent to a server, mainly via network communication such as Wi-Fi.

[1798] 8. Notification Method (Server):

[1799] The recipes generated by the suggestion means and the sale information collected by the acquisition means are notified to the user's device such as a smartphone, etc. This includes a function to send push notifications in real time.

[1800] Processing and its concrete examples

[1801] The server analyzes the image data sent from the device and extracts the item's identification information and expiration date. Technologies used include OpenCV and OCR. Next, an emotion recognition algorithm is used to analyze the user's emotions using their voice and facial expression data.

[1802] As a concrete example, if there is milk, tomatoes, and carrots in the refrigerator, the device will take a picture of these items and send the image data to the server. The server will then obtain the item's identification information (e.g., milk, tomatoes, carrots) based on the image data and recognize their respective expiration dates (e.g., May 1, 2023, April 28, 2023). Based on this information, the proposed method will provide the optimal recipe (e.g., tomato cream soup).

[1803] Additionally, if the emotion recognition means detects that the user is feeling stressed, it will also suggest relaxing recipes (e.g., herbal soups) that can help relieve stress.

[1804] Prompt Sentence Examples

[1805] "Develop a system that monitors the user's emotions in real time using a camera while they are using smart glasses, and provides advice and safety information based on their emotional state. If the user is in a 'stressed' state, notify them of nearby safety and alert information. Please also provide example program code."

[1806] The present invention provides more personalized information and enhances user convenience by making suggestions and warnings that take into account the user's emotional state.

[1807] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1808] Step 1:

[1809] Input: A user opens the refrigerator, stores items, and then closes the refrigerator door.

[1810] Processing: The device's camera automatically activates and takes a 360-degree image of the items in the refrigerator.

[1811] Output: Image data of items in the refrigerator.

[1812] How it works: A camera scans the inside of the refrigerator, capturing the location and appearance of each item.

[1813] Step 2:

[1814] Input: Image data of items in the refrigerator.

[1815] Processing: The device temporarily stores the acquired image data in its local memory.

[1816] Output: Image data of the item stored in local memory.

[1817] Operation: Stores image data in the device's storage.

[1818] Step 3:

[1819] Input: Image data of the item stored in local memory.

[1820] Processing: The device sends the image data to the server via Wi-Fi.

[1821] Output: Image data sent to the server.

[1822] How it works: The device uploads image data to a server via a Wi-Fi connection.

[1823] Step 4:

[1824] Input: Image data sent to the server.

[1825] Processing: The server analyzes the received image data and uses object recognition algorithms to extract the item's identity (name, type) and shelf life.

[1826] Output: Item identification and shelf life data.

[1827] How it works: The server uses object recognition algorithms and OCR technology to read the label information on the item.

[1828] Step 5:

[1829] Input: Item identification and shelf life data.

[1830] Processing: The server analyzes the user's voice input and facial expression data to recognize emotions.

[1831] Output: Emotion recognition result (e.g. stress, relaxation, etc.).

[1832] How it works: The server uses emotion recognition algorithms to identify the user's emotions from audio and image data.

[1833] Step 6:

[1834] Input: Item identification, shelf life data, and emotion recognition results.

[1835] Processing: Based on this data, the server proposes the optimal usage method (recipe).

[1836] Output: Suggested recipe information.

[1837] How it works: The server searches for suitable recipes from the database and generates suggestions taking into account emotion recognition results.

[1838] Step 7:

[1839] Input: The user's location.

[1840] Processing: The server retrieves sales information from vendors in the user's vicinity from an external database.

[1841] Output:Sale information.

[1842] How it works: The server uses an external API to retrieve special offers based on the user's location.

[1843] Step 8:

[1844] Input: Suggested recipe information and special offer information.

[1845] Processing: The server notifies the user's smartphone app of this information.

[1846] Output: A notification that appears on the user's smartphone.

[1847] How it works: The server uses push notifications to send information to users in real time.

[1848] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1849] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1850] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1851] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1852] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1853] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1854] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1855] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1856] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1857] The emo...

Claims

1. imaging means for imaging items within the food storage device; a recognition means for analyzing image data obtained by the imaging means and recognizing identification information and a shelf life of the item; a suggestion means for suggesting an optimal use method related to the item based on the identification information and the shelf life; a transmitting means for transmitting the recognized product information to an external database; an acquisition means for acquiring sale information of retailers located near the user from the external database; a notification means for notifying a user of the sale information acquired by the acquisition means; A system including:

2. 2. The system according to claim 1, wherein the suggestion means suggests a cooking recipe based on the identification information and expiration date of the item.

3. 2. The system of claim 1, wherein the recognition means uses optical character recognition (OCR) technology to read label information on the item.

Images