system

Abstract

Provide a system. [Solution] A means for acquiring an image of the inside of a refrigerator; means for processing the image and identifying the food product; means for storing information of the identified food products in a database; means for notifying a user terminal of information in the database; means for analyzing a question from a user using natural language processing and generating a response based on information in the database; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, managing food in the refrigerator is becoming increasingly important. However, manually managing food inventory and expiration dates in the refrigerator is cumbersome and inefficient. This increases the risk of wasting food or discarding it after the expiration date, especially for people who lead busy lives. To address these issues, there is a demand for a system that can check the status of the refrigerator in real time and manage it efficiently. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for acquiring images of the inside of a refrigerator, a means for processing the images to identify food items, a means for saving information about the identified food items in a database, a means for notifying a user terminal of the information in the database, and a means for analyzing questions from users using natural language processing and generating responses based on the information in the database. This automates refrigerator inventory management and food expiration date management, allowing users to use food more efficiently. Furthermore, the system further enhances convenience by including a means for periodically reacquiring images of the inside of the refrigerator and updating them to the latest version, and a means for sending notifications to the user terminal based on expiration dates.

[0006] The "means for acquiring images inside the refrigerator" refers to a mechanism that captures video and photos from cameras and sensors installed inside the refrigerator and transmits that data to the system.

[0007] The "means for processing the images and identifying the food items" refers to a function that uses an image recognition algorithm on the acquired images to identify each food item in the refrigerator and determine its labeling and quantity.

[0008] The "means for storing information about the identified food in a database" refers to a mechanism for recording information about the identified food, such as the type, quantity, and expiration date, in a database so that it can be referenced later.

[0009] "Means for notifying the user terminal of the information in the database" is a function that sends and displays the food information stored in the refrigerator on the user's terminal such as a smartphone or tablet.

[0010] "Means for analyzing questions from users using natural language processing and generating responses based on information in the database" refers to a mechanism that analyzes questions posed by users via voice or text, generates appropriate answers based on information in the database, and provides them to the user.

[0011] The "means for periodically reacquiring images of the inside of the refrigerator and updating them to the latest state" is a function for capturing new images of the inside of the refrigerator at regular intervals and updating existing data to the latest information.

[0012] The "means for sending a notification to a user terminal based on the expiration date of the food" is a function that monitors the expiration date of food and sends an alert or notification to the user terminal when the expiration date approaches. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] This system manages the contents of a refrigerator in real time by capturing images of the inside of the refrigerator, processing the images to identify food items, storing the information in a database, and notifying the user as needed. It also has the ability to generate answers to user questions using natural language processing.

[0035] Specific Embodiments of the System

[0036] 1. Installing and starting the camera

[0037] The server installs multiple cameras inside the refrigerator and periodically activates these cameras to capture images inside the refrigerator.

[0038] For example, a dedicated camera is installed in the vegetable compartment, and every time the refrigerator door is closed, the camera activates and captures an image of the entire vegetable compartment. The server receives this image in real time.

[0039] 2. Image Preprocessing and Analysis

[0040] The server pre-processes the received images, which includes noise reduction, color correction, etc.

[0041] The pre-processed images are then fed into an object recognition algorithm to identify the food.

[0042] For example, cabbage and tomatoes are identified from the captured image, and the server identifies the type and quantity of these foods.

[0043] 3. Data storage

[0044] The server stores the identified food information in a database, compares it with existing data, and updates it if there are any changes.

[0045] For example, if a new cabbage is added, it will be saved in the database as "1 cabbage." If a cabbage is already registered, the quantity will be updated.

[0046] 4. Notice to Users

[0047] The server sends the latest information about the contents of the refrigerator to the terminal, allowing the user to check the status of the refrigerator in real time via a smartphone or tablet.

[0048] For example, the terminal will display, "There is one cabbage and two tomatoes in the refrigerator." Furthermore, if the expiration date approaches, the server will send a notification to the terminal, warning the user, "The expiration date of the cabbage is tomorrow."

[0049] 5. Answering questions using natural language processing

[0050] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server.

[0051] The server analyzes the question, checks its database and generates an appropriate answer.

[0052] For example, if a user asks, "How many tomatoes are there?", the server generates the answer "There are two tomatoes" based on the information in the database and notifies the user via the terminal.

[0053] This system allows for efficient management of refrigerator contents and reduces food waste. Furthermore, users can check the status of their refrigerator anytime, anywhere, making it easier to plan their shopping and cooking. It also contributes to reducing food waste by managing expiration dates.

[0054] The processing flow will be explained below.

[0055] Step 1:

[0056] The server periodically activates a camera installed inside the refrigerator to capture images of the inside of the refrigerator. For example, the camera is set to activate every time the refrigerator door is closed.

[0057] Step 2:

[0058] The server receives the captured image. Since the received image may be imperfect as it is, it performs preprocessing such as noise removal and color correction. For example, if the image contains noise, a filter is applied to remove the noise.

[0059] Step 3:

[0060] The server then inputs the pre-processed images into an object recognition algorithm, which uses advanced object recognition techniques such as YOLO (You Only Look Once) and Faster R-CNN to identify the food items in the refrigerator.

[0061] Step 4:

[0062] The server identifies the type and quantity of food based on the results of object recognition. For example, if one cabbage and two tomatoes are identified from the image, the server extracts the information about each item as data.

[0063] Step 5:

[0064] The server stores the extracted food information in a database, compares it with existing data, and reflects any changes in stock or the addition of new food items. For example, if cabbage is already registered in the database, the quantity is updated.

[0065] Step 6:

[0066] The server periodically sends updated information about the contents of the refrigerator to the device, which then displays this information so the user can check the status of the refrigerator. For example, a notification might appear on the device saying, "There is one cabbage and two tomatoes in the refrigerator."

[0067] Step 7:

[0068] The server monitors the expiration dates of food products and sends notifications to the device when the expiration date is approaching. For example, it sends a warning to the device saying, "The expiration date of the cabbage is tomorrow."

[0069] Step 8:

[0070] The user asks a question about the status of the refrigerator via voice or text, for example, "How many tomatoes are there in the refrigerator?"

[0071] Step 9:

[0072] The terminal receives the query and sends the request to the server.

[0073] Step 10:

[0074] The server analyzes the question using natural language processing and queries the database to generate an appropriate answer, such as "There are two tomatoes."

[0075] Step 11:

[0076] The terminal will notify the user of the generated answer, for example, "There are two tomatoes" will be displayed on the terminal.

[0077] Example 1

[0078] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0079] In conventional refrigerator management systems, managing food inventory and expiration dates is often done manually, requiring users to frequently check the contents of the refrigerator. There is also a high risk of incorrect information being entered, which can lead to discrepancies between actual inventory and records. Furthermore, there is an insufficient system for users to quickly respond to questions about the food in the refrigerator.

[0080] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0081] In this invention, the server includes means for acquiring images of the inside of the refrigerator, means for preprocessing the images, means for analyzing the preprocessed images to identify food items, means for saving information about the identified food items in a database, means for notifying a user terminal of information in the database, and means for analyzing questions from users using natural language processing and generating responses based on information in the database. This allows for accurate real-time management of food items in the refrigerator and enables users to easily check the inventory status and expiration dates in the refrigerator. Furthermore, a question-answering function using natural language processing improves user convenience.

[0082] "Means for acquiring images inside the refrigerator" refers to a function that periodically takes images of the food and containers inside the refrigerator using a photographic device such as a camera installed inside the refrigerator and acquires them as digital data.

[0083] "Preprocessing" refers to a series of processes that involve processing the acquired image data, such as noise removal and color correction, to improve the quality of the image.

[0084] "Analysis" or "image analysis" refers to the use of algorithms and artificial intelligence to analyze pre-processed image data and recognize and identify objects (in this case, food) within the image.

[0085] "Means for identifying food" refers to the function of using image analysis to identify individual food items in the refrigerator and clarify their type and quantity.

[0086] "Means for storing in a database" refers to the technology for recording and storing information on identified food (such as type, quantity, location, etc.) in a database as digital data.

[0087] "Means for notifying user devices" refers to the function of analyzing information stored in the database and sending necessary notifications in real time to the user's electronic device (such as a smartphone or tablet).

[0088] "Natural language processing" refers to technology that understands, analyzes, and responds appropriately to natural human language. In this context, it includes the ability for a system to generate appropriate answers to user inquiries.

[0089] "Means for generating a response" refers to technology that uses natural language processing to understand a user's question, constructs an appropriate response based on information in a database, and notifies the user's device of this response.

[0090] "Means for sending notifications based on expiration dates" refers to a function that monitors expiration date information for each food item stored in the database and automatically sends notifications to users when the expiration date approaches.

[0091] The present invention relates to a system for managing food in a refrigerator in real time. The program processing of this system is specifically explained below. Specific examples of the system's hardware and software include cameras, servers, user terminals, and artificial intelligence models.

[0092] 1. Installing and starting the camera

[0093] The server installs high-resolution cameras at key locations inside the refrigerator. Each time the refrigerator door is closed, the cameras are automatically activated and capture images of the interior. A typical example of such cameras is a general-purpose digital camera.

[0094] Example: When the refrigerator door is closed, the server sets the camera to immediately take pictures of the vegetable compartment and main storage area, allowing the overall status of the refrigerator to be grasped in real time.

[0095] 2. Image Preprocessing

[0096] The server preprocesses the acquired image data using libraries such as OpenCV. This preprocessing includes noise reduction and color correction, improving the quality of the images and the accuracy of subsequent analysis.

[0097] Example: The server applies noise filtering and adjusts the brightness and contrast of the image to clarify fuzzy areas.

[0098] 3. Image Analysis

[0099] The pre-processed image data is then fed into an object recognition algorithm such as YOLO (You Only Look Once) to identify the food items. The server then analyzes the data and identifies the type and quantity of food items in the refrigerator.

[0100] Example: For example, the server uses an object recognition algorithm to identify one cabbage and two tomatoes in an image.

[0101] 4. Data storage

[0102] The server stores the information about the identified food in a database using MySQL (registered trademark) or PostgreSQL. It compares the information with existing data and immediately updates any changes.

[0103] Example: When a new cabbage is added, the server stores it in the database as "1 cabbage." If a cabbage is already registered, the server updates the quantity.

[0104] 5. Notice to Users

[0105] The server sends the latest information about the contents of the refrigerator to the user's device. Users can check the status of the refrigerator in real time via their smartphone or tablet. The server also notifies users when the expiration date of each food item is approaching.

[0106] Example: The user device displays "There is one cabbage and two tomatoes in the refrigerator." If the expiration date is one day away, the device displays a notification saying "The cabbage's expiration date is tomorrow."

[0107] 6. Answering questions using natural language processing

[0108] When a user asks a question about the status of the refrigerator by voice or text, the user device sends the question to the server, which analyzes the question, checks the database, and generates an appropriate answer.

[0109] Example: When a user asks, "How many tomatoes are there?", the server generates the answer "There are two tomatoes" based on the information in the database and notifies the user via their device.

[0110] Prompt Sentence Examples

[0111] 1. "Please tell me how many tomatoes are in the refrigerator."

[0112] 2. "When is the expiration date for cabbage?"

[0113] This system allows users to accurately manage the contents of their refrigerator in real time, greatly improving user convenience. It allows users to efficiently grasp food inventory and expiration dates, reducing food waste and making shopping and cooking planning easier. Furthermore, a question-and-answer function using natural language processing allows users to quickly and accurately obtain information about what's inside their refrigerator.

[0114] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0115] Step 1:

[0116] Camera installation and startup

[0117] The server installs a high-resolution camera inside the refrigerator, which automatically activates every time the door is closed.

[0118] Input: Refrigerator door closing trigger signal

[0119] Data processing and calculation: Camera activation and image capture

[0120] Output: High-resolution image data of the inside of the refrigerator

[0121] Specific operation: When the refrigerator door is closed, the camera automatically starts up, takes a picture of the inside of the refrigerator, and sends it to the server.

[0122] Step 2:

[0123] Image preprocessing

[0124] The server performs pre-processing such as noise removal and color correction on the received image data.

[0125] Input: High-resolution image data of the inside of the refrigerator

[0126] Data processing and calculation: noise filtering, color correction

[0127] Output: Preprocessed image data

[0128] What it does: The server uses libraries such as OpenCV to remove noise from the image and balance the colors to produce a clear image.

[0129] Step 3:

[0130] Image analysis

[0131] The server inputs the preprocessed image data into an object recognition algorithm (e.g., YOLO) to identify the food item.

[0132] Input: Preprocessed image data

[0133] Data processing and calculation: Analysis using object recognition algorithms

[0134] Output: Identified food type and quantity data

[0135] How it works: The analysis algorithm scans the image and identifies foods such as cabbage and tomatoes, and then identifies the type and quantity of each food item and stores it as data.

[0136] Step 4:

[0137] Data storage

[0138] The server stores the identified food information in a database.

[0139] Input: Identified food type and quantity data

[0140] Data processing and calculation: writing information to the database, updating existing data

[0141] Output: A database containing the latest food information

[0142] Specific operation: The server uses MySQL or PostgreSQL to record food information in a database and compare it with existing data to update it.

[0143] Step 5:

[0144] User Notification

[0145] The server notifies the user terminal of the latest refrigerator information.

[0146] Input: A database containing the latest food information

[0147] Data processing and calculation: Generates notification messages and sends them to user terminals

[0148] Output: Latest refrigerator information displayed on the user's device

[0149] Specific operation: The server retrieves the latest information from the database and displays the status of the refrigerator on the user's device. For example, it displays "There is one cabbage and two tomatoes in the refrigerator."

[0150] Step 6:

[0151] Answering questions using natural language processing

[0152] The user asks questions about the status of the refrigerator by voice or text, and the device sends the questions to the server, which analyzes the questions and generates appropriate answers.

[0153] Input: User question (voice or text)

[0154] Data processing and calculation: Question analysis using natural language processing, database matching, and response generation

[0155] Output: Response message to be sent to the user

[0156] Specific operation: When a user asks "How many tomatoes are there?", the server analyzes the question, retrieves information from the database, and sends the answer "There are two tomatoes" to the user's terminal.

[0157] (Application example 1)

[0158] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0159] In modern households, managing food in the refrigerator is extremely time-consuming, and food waste due to expired food and unnecessary duplicate purchases is a major problem. It is also difficult to keep track of the refrigerator's contents while out or cooking, making it difficult to efficiently plan shopping and meals. Furthermore, there is a lack of systems that can quickly and accurately respond to users' questions about the refrigerator's contents, and there is a need to solve this problem.

[0160] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0161] In this invention, the server includes means for acquiring images of the inside of the refrigerator, means for processing the images and identifying foods, means for storing information about the identified foods in a database, means for notifying a user terminal of the information in the database, means for analyzing questions from users using natural language processing and generating responses based on the information in the database, means for users to check the information in the database in real time through an application installed on a smart device, and means for notifying a user terminal of foods that are approaching their expiration date. This allows for efficient management of foods in the refrigerator, reduces food waste, and enables users to keep track of the status of their refrigerator anytime, anywhere.

[0162] The "means for acquiring images inside the refrigerator" is a mechanism for capturing images of the inside of the refrigerator using a camera installed inside the refrigerator and acquiring the image data.

[0163] The "means for processing the image and identifying the food" refers to an algorithm or program that analyzes the acquired image data and identifies the various foods present in the refrigerator.

[0164] The "means for storing information on the identified food in a database" is a mechanism for recording and managing data such as the type and quantity of food identified through analysis in a database.

[0165] The "means for notifying the user terminal of the information in the database" is a mechanism for transmitting and displaying the food information stored in the database to a terminal such as a smartphone or tablet owned by the user.

[0166] "Means for analyzing questions from users using natural language processing and generating responses based on information in the database" refers to a mechanism that uses natural language processing technology to analyze voice or text questions from users, compares them with information in the database, and generates appropriate responses.

[0167] "A means by which users can check the information in the database in real time through an application installed on a smart device" refers to a mechanism that allows users to view the latest food information in the database in real time using a dedicated application installed on a smart device such as a smartphone.

[0168] The "means for notifying a user terminal of food products approaching their expiration date" is a mechanism that sends a notification to a user terminal about food products that are approaching their expiration date based on food information in the database.

[0169] The present invention provides a system that efficiently manages food in a refrigerator, reduces food waste, and allows a user to grasp the status of the refrigerator anytime, anywhere.

[0170] A system for implementing the present invention comprises the following components:

[0171] 1. Image acquisition method:

[0172] There are multiple cameras installed inside the refrigerator, and these cameras periodically capture images of the inside of the refrigerator. For example, every time the refrigerator door is closed, a camera is automatically activated and takes a picture of the inside of the refrigerator.

[0173] 2. Imaging and Food Identification Methods:

[0174] The captured images are first preprocessed with noise removal and color correction, then an object recognition algorithm (e.g., a deep learning model using Keras) is used to identify food items. For example, cabbage and tomatoes are identified, and the quantity of each is also determined.

[0175] 3. Database storage method:

[0176] Information on the identified food (type, quantity, etc.) is stored in a database such as SQLite. If an update is required, this information is compared with existing data and updated to the latest version.

[0177] 4. User terminal notification means:

[0178] The identified food information is sent in real time to the user's device, such as a smartphone or tablet. The user can check the status of the refrigerator through a dedicated application. For example, information such as "There is one cabbage and two tomatoes in the refrigerator" is displayed.

[0179] 5. Natural Language Processing Tools:

[0180] Voice and text questions from users are sent to the server via their device. The questions are analyzed using natural language processing technology and compared with information in a database to generate an appropriate answer. For example, in response to the question "How many tomatoes are there?", an answer such as "There are two tomatoes" is generated and notified to the user's device.

[0181] 6. Best before date notification method:

[0182] When the expiration date of an identified food item is approaching, the server sends a notification to the user's terminal. This notification allows the user to take action early on regarding the food item that is approaching its expiration date. For example, the user may receive a notification saying, "The expiration date of the cabbage is tomorrow."

[0183] This allows users to effectively manage the food in their refrigerator and reduce food waste. It also allows users to check the status of their refrigerator in real time while they are out or shopping, helping them avoid unnecessary purchases. Furthermore, users can instantly ask questions about the status of their refrigerator and get answers, enabling more efficient meal preparation.

[0184] Example 1:

[0185] When you open the smartphone app and tap the "Show latest refrigerator status" button, the app requests the latest food information from the server and displays it on the screen. You can instantly see information such as "There is one cabbage and two tomatoes in the refrigerator."

[0186] Example 2:

[0187] When the expiration date approaches, a notification will be sent to the smartphone, with the message "The cabbage's expiration date is tomorrow," allowing the user to use the cabbage sooner.

[0188] Example prompt for a generative AI model:

[0189] A user asks: "How many tomatoes are in the fridge right now?"

[0190] Prompt for the AI ​​model: "The user wants to know how many tomatoes are in the refrigerator. Please check your database and generate the correct answer."

[0191] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0192] Step 1:

[0193] Capture images of the inside of the refrigerator

[0194] The server activates a camera installed inside the refrigerator to capture an image of the inside of the refrigerator.

[0195] Input: Latest status of the refrigerator

[0196] Output: Image data of the inside of the refrigerator (e.g., JPEG image file)

[0197] Step 2:

[0198] Image preprocessing

[0199] The server performs preprocessing such as noise removal and color correction on the acquired image data.

[0200] Input: Raw captured image data

[0201] Output: Pre-processed image data (e.g., noise-removed and color-optimized images)

[0202] Step 3:

[0203] Food Identification

[0204] The server inputs the preprocessed image data into a food identification algorithm to identify the food items, for example, using a deep learning model (e.g., using Keras) to identify each food item in the refrigerator.

[0205] Input: Preprocessed image data

[0206] Output: A list of identified foods (e.g., {'cabbage': 1, 'tomato': 2})

[0207] Step 4:

[0208] Saving to a database

[0209] The server stores the identified food information (type, quantity) in a database such as SQLite, compares it with existing data, and updates it as necessary.

[0210] Input: List of identified foods

[0211] Output: Latest food information registered in the database

[0212] Step 5:

[0213] Notification to user device

[0214] The server sends the latest food information stored in the database to the user's smartphone or tablet, where the user can check this information in real time through a dedicated application.

[0215] Input: Latest food information from the database

[0216] Output: Food information displayed on the user's device (application)

[0217] Step 6:

[0218] Answering questions using natural language processing

[0219] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server. The server uses natural language processing technology to analyze the question, compares it with information in a database, generates an appropriate answer, and sends it to the device. For example, if a user asks, "How many tomatoes are there?", the server generates the answer, "There are two tomatoes."

[0220] Input: User question (voice or text)

[0221] Output: Answer based on the database (e.g., there are two tomatoes)

[0222] Step 7:

[0223] Best before date notification

[0224] The server uses the food information in the database to identify foods that are approaching their expiration date and sends a notification to the user's smartphone, such as "The expiration date for the cabbage is tomorrow."

[0225] Input: Food information and its expiration date in the database

[0226] Output: Expiration warning notification (e.g., the expiration date for cabbage is tomorrow)

[0227] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0228] This invention combines an emotion engine with a system for managing the contents of a refrigerator in real time. It captures images of the inside of the refrigerator, processes them to identify food items, stores the information in a database, and notifies the user as needed. It also has the ability to generate answers to user questions using natural language processing, and to analyze user emotions to customize notifications and responses.

[0229] Specific Embodiments of the System

[0230] 1. Installing and starting the camera

[0231] The server installs multiple cameras inside the refrigerator and periodically activates these cameras to capture images inside the refrigerator.

[0232] For example, a dedicated camera is installed in the vegetable compartment, and every time the refrigerator door is closed, the camera activates and captures an image of the entire vegetable compartment. The server receives this image in real time.

[0233] 2. Image Preprocessing and Analysis

[0234] The server pre-processes the received images, which includes noise reduction, color correction, etc.

[0235] The pre-processed images are then fed into an object recognition algorithm to identify the food.

[0236] For example, cabbage and tomatoes are identified from the captured image, and the server identifies the type and quantity of these foods.

[0237] 3. Data storage

[0238] The server stores the identified food information in a database, compares it with existing data, and updates it if there are any changes.

[0239] For example, if a new cabbage is added, it will be saved in the database as "1 cabbage." If a cabbage is already registered, the quantity will be updated.

[0240] 4. Notice to Users

[0241] The server sends the latest information about the contents of the refrigerator to the terminal, allowing the user to check the status of the refrigerator in real time via a smartphone or tablet.

[0242] For example, the terminal will display, "There is one cabbage and two tomatoes in the refrigerator." Furthermore, if the expiration date approaches, the server will send a notification to the terminal, warning the user, "The expiration date of the cabbage is tomorrow."

[0243] 5. Answering questions using natural language processing

[0244] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server.

[0245] The server analyzes the question, checks its database and generates an appropriate answer.

[0246] For example, if a user asks, "How many tomatoes are there?", the server generates an answer based on the information in the database: "There are two tomatoes," and notifies the user via the terminal.

[0247] 6. Implementing the Emotion Engine

[0248] The server is equipped with an emotion engine that recognizes emotions from the user's voice, text, facial expressions, etc. This allows it to analyze the user's emotions and customize corresponding notifications and responses.

[0249] For example, if the user is in a hurry, the system will calmly display a notification saying, "The cabbage's expiration date is approaching, so please use it up quickly."

[0250] 7. Customize notifications based on emotions

[0251] The server customizes the notification content and response based on the user's emotional information obtained by the emotion engine.

[0252] For example, if a user looks anxious, a notification offering assistance such as "I have two tomatoes, can I help you?" can be sent.

[0253] This system not only allows for efficient management of the contents of the refrigerator, but also takes into account the user's emotions, providing a more friendly user experience. By recognizing emotions and optimizing notifications and responses, we can expect to reduce user stress and improve the efficiency of food management.

[0254] The processing flow will be explained below.

[0255] Step 1:

[0256] The server periodically activates the camera installed inside the refrigerator to capture images of the inside of the refrigerator. For example, every time the refrigerator door is closed, the camera is automatically activated to capture a full image of the inside.

[0257] Step 2:

[0258] The server receives the captured image, which may be incomplete as it is, so it undergoes pre-processing such as noise reduction and color correction.

[0259] Step 3:

[0260] The server then inputs the preprocessed images into an object recognition algorithm, specifically using object recognition techniques such as YOLO (You Only Look Once) and Faster R-CNN to identify the food in the image.

[0261] Step 4:

[0262] The server identifies the type and quantity of food based on the results of object recognition. For example, if one cabbage and two tomatoes are identified from the image, the server extracts the information about each item as data.

[0263] Step 5:

[0264] The server stores the extracted food information in a database, compares it with existing data, and reflects any changes in stock or the addition of new food items. For example, if cabbage is already registered in the database, the quantity is updated.

[0265] Step 6:

[0266] The server periodically sends updated information about the contents of the refrigerator to the terminal, allowing the user to check the status of the refrigerator in real time via a smartphone or tablet.

[0267] Step 7:

[0268] The server monitors the expiration dates of food products and sends notifications to the device when the expiration date is approaching. For example, it sends a warning to the device saying, "The expiration date of the cabbage is tomorrow."

[0269] Step 8:

[0270] The user asks a question via voice or text about what's going on in the refrigerator, for example, "How many tomatoes are there in the refrigerator?"

[0271] Step 9:

[0272] The terminal receives the user's query and sends the request to the server.

[0273] Step 10:

[0274] The server analyzes the question using natural language processing and queries the database to generate an appropriate answer, such as "There are two tomatoes."

[0275] Step 11:

[0276] The terminal notifies the user of the generated answer, for example by displaying the message "There are two tomatoes" on the terminal.

[0277] Step 12:

[0278] The server analyzes the user's emotions using an emotion engine that recognizes emotions from the user's voice, text, facial expressions, etc. For example, if the user's tone of voice sounds impatient, the server recognizes that emotion.

[0279] Step 13:

[0280] The server customizes the notification content and response based on the user's emotional information obtained by the emotion engine. For example, if the user is feeling anxious, it generates a notification offering support, such as "I have two tomatoes. Can I help you?"

[0281] Step 14:

[0282] The device will display customized notifications to the user, for example, if the user is feeling busy, it will calmly display a notification saying, "The cabbage is nearing its expiration date, please use it up soon."

[0283] In this way, this system not only improves the efficiency of inventory management in the refrigerator, but also provides a more user-friendly interface by providing notifications and responses based on emotions.

[0284] Example 2

[0285] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0286] Conventional refrigerator management systems have the drawback of requiring a lot of time and effort to understand the status of food in the refrigerator, resulting in low user convenience. Furthermore, they lack the ability to provide appropriate notifications and responses that reflect the user's emotions and state of mind, making it difficult to improve the user experience. Furthermore, they lack the functionality to manage food expiration dates or provide specific advice, which can easily lead to food waste.

[0287] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0288] In this invention, the server includes means for capturing images of the inside of the refrigerator using a camera, means for preprocessing the images and identifying food items using an object recognition algorithm, means for saving information about the identified food items in a database, means for notifying a user terminal of the information in the database, means for analyzing questions from users using natural language processing and generating responses based on the information in the database, means for analyzing the user's voice, text, and facial expressions to recognize emotions, and means for customizing notification content and responses based on the recognized emotions. This makes food management in the refrigerator more efficient and enables appropriate responses based on the user's emotions, which is expected to improve the user experience and reduce food waste.

[0289] The "camera" is a photographing device installed to capture images inside the refrigerator.

[0290] "Preprocessing" refers to processes such as noise removal and color correction that are performed to make the acquired image easier to analyze.

[0291] An "object recognition algorithm" is a computational method for identifying food in an image and determining its type and quantity.

[0292] "Database" means a collection of digital data for storing and managing information about identified foods.

[0293] A "user terminal" is a device that a user uses to check information about the contents of the refrigerator and to communicate with the system.

[0294] "Natural language processing" is an artificial intelligence technology that analyzes questions from users and generates appropriate responses.

[0295] An "emotion engine" is a system that analyzes a user's voice, text, and facial expressions to recognize emotions.

[0296] "Customizing notification content and responses" refers to the process of adjusting the content of notifications and responses based on the user's emotions.

[0297] This invention relates to a system for efficiently managing food in a refrigerator, aiming to improve user convenience and reduce stress. The system consists of a camera installed in the refrigerator, a server, a user terminal, an object recognition algorithm, a database, natural language processing, and an emotion engine.

[0298] Hardware and Software Configuration

[0299] camera

[0300] There are multiple high-resolution cameras installed inside the refrigerator. These cameras (e.g., high-resolution cameras) have the ability to capture an image every time the refrigerator door is closed.

[0301] server

[0302] The server receives the image data sent from the camera and begins processing. First, the image is pre-processed. Specifically, camera noise is removed and color correction is performed. After this pre-processing is complete, the image is input into an object recognition algorithm (e.g., YOLO) to identify the type and quantity of food.

[0303] Database

[0304] The server stores the identified food information in a database (e.g., MySQL). When a new food is added, the information is updated in the database and existing food information is updated.

[0305] User terminal

[0306] The user terminal is a device such as a smartphone or tablet that allows the user to check the information inside the refrigerator. The latest food information is sent from the server to the terminal, allowing the user to check the status of the refrigerator in real time. For example, information such as "There is one cabbage and two tomatoes in the refrigerator" is displayed.

[0307] Natural Language Processing

[0308] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server. The server uses natural language processing (e.g., BERT) to analyze the question and compares it with information in the database to generate an appropriate answer. For example, the question "How many tomatoes are there?" will generate an answer such as "There are two tomatoes."

[0309] Emotion Engine

[0310] The server is equipped with an emotion engine (e.g., Affectiva SDK) that analyzes the user's voice, text, and facial expressions to recognize their emotions. This allows it to analyze the user's emotions and customize corresponding notifications and responses. For example, if the user is feeling anxious, the server may notify them that "The expiration date of the cabbage is approaching, so please use it up quickly."

[0311] Customize notifications based on emotions

[0312] The server customizes the notification content and response based on the user's emotional information obtained by the emotion engine. For example, if the user looks anxious, a notification offering support such as "There are two tomatoes. Can I help you?" will be sent.

[0313] Examples of concrete examples and prompts

[0314] For example, whenever a user opens or closes the refrigerator, the camera captures an image and the server updates the food information based on this. For example, when a user asks, "How many tomatoes are there?", the server responds via the device with, "There are two tomatoes."

[0315] Example prompt sentence:

[0316] "Capture an image of the inside of your refrigerator, identify the food in the crisper, and store it in a database."

[0317] This system allows users to manage the food in their refrigerator efficiently and without waste, and also provides a pleasant user experience by receiving appropriate notifications and responses based on their emotions.

[0318] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0319] Step 1:

[0320] Camera installation and image acquisition

[0321] The server installs multiple high-resolution cameras inside the refrigerator. Each time the refrigerator door is closed, these cameras are activated and capture images of the inside of the refrigerator. Specifically, the cameras are automatically activated when the refrigerator door sensor detects that the door is closing, and images of multiple locations inside the refrigerator are taken.

[0322] Input: Actual view inside the refrigerator, door sensor signal

[0323] Output: High-resolution image of the inside of the refrigerator

[0324] Step 2:

[0325] Image preprocessing

[0326] The server receives the captured images and performs pre-processing, such as noise reduction and color correction to improve image quality, resulting in image data ready to be input into object recognition algorithms.

[0327] Input: High-resolution image

[0328] Output: Preprocessed image

[0329] Step 3:

[0330] object recognition

[0331] The server inputs the preprocessed image into an object recognition algorithm (e.g., YOLO). The algorithm identifies the type and quantity of food and identifies the results. Specifically, the YOLO algorithm analyzes each pixel in the image and identifies specific objects (e.g., cabbage, tomato).

[0332] Input: Preprocessed image

[0333] Output: Identified food types and quantities

[0334] Step 4:

[0335] Data storage

[0336] The server stores the identified food information in a database (e.g., MySQL). When a new food is added, the information is saved in the database and existing food information is updated as necessary. Specifically, the new and existing data are compared and the differences are updated.

[0337] Input: Type and quantity of food identified

[0338] Output: Updated database contents

[0339] Step 5:

[0340] Notification to user device

[0341] The server sends the latest food information to the user's device (smartphone or tablet), allowing the user to check the status of the refrigerator in real time. For example, a notification might say, "There is one cabbage and two tomatoes in the refrigerator."

[0342] Input: Updated database contents

[0343] Output: Food information displayed on the user's device

[0344] Step 6:

[0345] Answering questions using natural language processing

[0346] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server. The server uses natural language processing (e.g., BERT) to analyze the question, compares it with information in the database, generates an appropriate answer, and notifies the user via the device. For example, the question "How many tomatoes are there?" is answered with "There are two tomatoes."

[0347] Input: User question (voice or text)

[0348] Output: Response to user terminal

[0349] Step 7:

[0350] Implementing the Emotion Engine

[0351] The server is equipped with an emotion engine (e.g., Affectiva SDK) that analyzes the user's voice, text, and facial expressions to recognize their emotions. This allows it to analyze the user's emotions and generate appropriate notifications and responses. For example, if the user is feeling impatient, the server will notify them by saying, "The cabbage's expiration date is approaching, so please use it up quickly."

[0352] Input: User voice, text, facial expressions

[0353] Output: Notification based on user sentiment

[0354] Step 8:

[0355] Customize notifications based on emotions

[0356] The server customizes the notification content and response based on the user's emotional information obtained from the emotion engine. For example, if the user looks anxious, the server sends a notification offering support such as, "There are two tomatoes. Can I help you?"

[0357] Input: User's emotional information

[0358] Output: Customized notification content

[0359] (Application example 2)

[0360] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0361] Conventional refrigerator management systems and store management systems only had the function of managing inventory information and were unable to respond or notify users based on their emotions. Furthermore, when it came to in-store product management, it was difficult to check inventory in real time, making it impossible to respond quickly to customer needs. This resulted in a poor user experience and a decline in the efficiency of inventory management and customer service.

[0362] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring images of the inside of the refrigerator and the inside of the store, means for processing the images and identifying food and products, and means for analyzing customer emotions and customizing notifications and responses. This enables appropriate responses according to user emotions and real-time management of inventory in the refrigerator and the store.

[0363] A "server" is a device that provides computing resources and storage, analyzes and stores various types of data, and manages communications.

[0364] The "means for acquiring images inside the refrigerator" is a technology for capturing images using a camera installed inside the refrigerator and acquiring them as data.

[0365] The "means for processing the image and identifying the food" refers to a technology for preprocessing the acquired image and identifying the food using an object recognition algorithm.

[0366] The "means for storing information on the identified food in a database" refers to a technique for recording data on the type and quantity of the identified food in a database.

[0367] "Means for notifying the user terminal of the information in the database" refers to technology that sends food information and update information stored in the database to the user's terminal such as a smartphone or tablet.

[0368] "Means for analyzing questions from users using natural language processing and generating responses based on information in the database" refers to technology that analyzes voice or text questions uttered by users and generates appropriate answers based on information in the database.

[0369] The "means for acquiring images within the store" refers to a technology for capturing images using cameras installed within the store and acquiring them as data.

[0370] The "means for processing the image and identifying the product" refers to a technology for preprocessing images acquired in a store and identifying the product using an object recognition algorithm.

[0371] "Means for analyzing customer emotions and customizing notifications and responses" refers to technology that reads emotions from a customer's facial expressions, voice, text, etc., and uses that information to customize the content of notifications and responses.

[0372] "Means for periodically reacquiring images of the inside of the refrigerator and the inside of the store and updating them to the latest state" refers to a technology that periodically operates the camera to acquire the latest images and keep the contents of the database up to date.

[0373] "Means for sending notifications to a user terminal based on the expiration dates of the food and products" refers to a technology that sends warnings and notifications to a user terminal about products that are approaching their expiration date based on expiration date information of stored food and products.

[0374] The embodiments for carrying out the present invention are as follows.

[0375] 1. System Overview

[0376] This system improves the efficiency of inventory management in refrigerators and physical stores, and provides responses based on the user's emotions. The system mainly consists of a server, a terminal, a camera, an emotion analysis engine, an object recognition algorithm, and a database.

[0377] 2. Hardware and Software

[0378] Cameras: Installed in refrigerators and in stores. Example: Logitech C920 webcam

[0379] Server: Responsible for data processing and storage. Example: AWS EC2 instance

[0380] Terminal: The device on which the user receives information. Examples: smartphone, tablet

[0381] software:

[0382] OpenCV: Image capture and preprocessing library

[0383] Object Recognition Algorithm: Identifying food and goods using YOLOv3 and ResNet

[0384] Database: Manage inventory data with MySQL or Firebase

[0385] Sentiment analysis engine: Uses Microsoft® Azure® Emotion API

[0386] Natural Language Processing: Question answering using GPT-3 (registered trademark)

[0387] 3. System Operation

[0388] 1. Image Acquisition

[0389] The server periodically activates the cameras installed in the refrigerator and in the store to capture the latest images. For example, the server can activate the camera and capture an image every time the refrigerator door is closed.

[0390] 2. Image Preprocessing and Analysis

[0391] The server preprocesses the acquired images using OpenCV. Specifically, it performs noise removal and color correction. The preprocessed images are input into an object recognition algorithm, and the server identifies food and products. For example, it identifies cabbages and tomatoes from an image of a vegetable drawer and determines their quantities.

[0392] 3. Data storage

[0393] The server stores the identified food and product information in a database. If there are any changes compared to existing data, the information is updated. For example, if a new item, "1 cabbage," is added to the database, the information is updated.

[0394] 4. Notice to Users

[0395] The server sends the latest information about what's in the refrigerator and in the store to the terminal. Users can check inventory status in real time via their smartphone or tablet. For example, the terminal will display information like, "There is one cabbage and two tomatoes in the refrigerator." In addition, it will send a warning if the expiration date is approaching.

[0396] 5. Answering questions using natural language processing

[0397] When a user asks a question about the inventory in the refrigerator or store by voice or text, the device sends the question to the server. The server uses GPT-3 to analyze the question, compares it with the database, and generates an appropriate answer. For example, if a user asks, "How many tomatoes are there?", the device will respond, "There are two tomatoes."

[0398] 6. Implementing the Emotion Engine

[0399] The server is equipped with an emotion analysis engine that recognizes emotions from the user's voice, text, facial expressions, etc. This allows it to analyze the user's emotions and customize corresponding notifications and responses.

[0400] 7. Customize notifications based on emotions

[0401] The server customizes notifications and responses based on the user's emotional information obtained from the emotion analysis engine. For example, if the user is in a hurry, the server will calmly display a notification such as, "The expiration date of the cabbage is approaching. Please use it up quickly." If the user is anxious, the server will send a notification offering support, such as, "You have two tomatoes. Is there anything I can help you with?"

[0402] 8. Examples of prompt sentences

[0403] "Tell me the stock situation"

[0404] "Do you have product A?"

[0405] Specific use cases

[0406] When a user asks "Do you have any cabbages?" on their smartphone, the server checks the database and responds, "We have one cabbage." If the user looks anxious, the server responds with a gentler response, such as, "We have two tomatoes. May I help you?"

[0407] Such a system enables flexible responses that adapt to the user's emotions, and allows for appropriate management of inventory status in refrigerators and stores.

[0408] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0409] Step 1:

[0410] The server activates the cameras installed in the refrigerator and in the store to capture images. Based on the request, the server obtains image data from the cameras and preprocesses it. Preprocessing includes noise reduction and color correction. The input is the raw image captured by the camera, and the output is the preprocessed image.

[0411] Step 2:

[0412] The server inputs the preprocessed images into an object recognition algorithm (e.g., YOLOv3 or ResNet) to identify food and store items. The input is the preprocessed image, and the output is the type and quantity of identified food or items. Specifically, the algorithm identifies objects in the image and returns their labels and locations.

[0413] Step 3:

[0414] The server stores the identified food and product information (type, quantity) in a database. It compares the data with existing data and updates it if there are any changes. The input is the identified food or product data, and the output is the updated database entry. For example, information such as "two tomatoes were added" is recorded in the database.

[0415] Step 4:

[0416] The server sends the latest information about what's in the refrigerator and the store to the user terminal. Here, it retrieves the latest information from the database and generates a message to send to the user terminal. The input is inventory information retrieved from the database, and the output is a notification message to be sent to the user terminal. For example, the message might say, "There is one cabbage and two tomatoes in the refrigerator."

[0417] Step 5:

[0418] The user asks a question about the inventory in the refrigerator or in the store by voice or text. The terminal sends the question to the server. The input to the terminal is the user's question, and the output is the question data to the server. For example, the terminal sends a user's question such as "How many tomatoes are there?" to the server.

[0419] Step 6:

[0420] The server uses GPT-3 to analyze the question, compare it with the database, and generate an appropriate answer. The input is the user's question data received from the device, and the output is the generated answer. Specifically, it uses natural language processing to understand the intent of the question and constructs an answer based on related information in the database.

[0421] Step 7:

[0422] The server uses an emotion analysis engine to analyze the user's emotions and recognizes emotions from voice, text, and facial expressions. The input is the user's voice data, text data, and image data, and the output is analyzed emotional information. Specifically, the emotion analysis engine analyzes this data and determines whether the user is anxious or impatient.

[0423] Step 8:

[0424] The server customizes notifications and responses based on the user's emotional information obtained by the emotion analysis engine. The input is the user's emotional information and the generated response information, and the output is a customized notification message. For example, if the user is in a hurry, the server sends a notification such as, "Remain calm. The expiration date of the cabbage is approaching. Please use it up quickly."

[0425] Step 9:

[0426] The server sends a customized notification to the user terminal. The input is a customized notification message, and the output is a notification displayed on the user terminal, allowing the user to receive accurate information.

[0427] Through the above steps, a system is realized in which the server, terminals, and users work together to manage inventory and handle customer service.

[0428] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0429] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0430] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0431] [Second embodiment]

[0432] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0433] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0434] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0435] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0436] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0437] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0438] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0439] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0440] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0441] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0442] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0443] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0444] This system manages the contents of a refrigerator in real time by capturing images of the inside of the refrigerator, processing the images to identify food items, storing the information in a database, and notifying the user as needed. It also has the ability to generate answers to user questions using natural language processing.

[0445] Specific Embodiments of the System

[0446] 1. Installing and starting the camera

[0447] The server installs multiple cameras inside the refrigerator and periodically activates these cameras to capture images inside the refrigerator.

[0448] For example, a dedicated camera is installed in the vegetable compartment, and every time the refrigerator door is closed, the camera activates and captures an image of the entire vegetable compartment. The server receives this image in real time.

[0449] 2. Image Preprocessing and Analysis

[0450] The server pre-processes the received images, which includes noise reduction, color correction, etc.

[0451] The pre-processed images are then fed into an object recognition algorithm to identify the food.

[0452] For example, cabbage and tomatoes are identified from the captured image, and the server identifies the type and quantity of these foods.

[0453] 3. Data storage

[0454] The server stores the identified food information in a database, compares it with existing data, and updates it if there are any changes.

[0455] For example, if a new cabbage is added, it will be saved in the database as "1 cabbage." If a cabbage is already registered, the quantity will be updated.

[0456] 4. Notice to Users

[0457] The server sends the latest information about the contents of the refrigerator to the terminal, allowing the user to check the status of the refrigerator in real time via a smartphone or tablet.

[0458] For example, the terminal will display, "There is one cabbage and two tomatoes in the refrigerator." Furthermore, if the expiration date approaches, the server will send a notification to the terminal, warning the user, "The expiration date of the cabbage is tomorrow."

[0459] 5. Answering questions using natural language processing

[0460] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server.

[0461] The server analyzes the question, checks its database and generates an appropriate answer.

[0462] For example, if a user asks, "How many tomatoes are there?", the server generates the answer "There are two tomatoes" based on the information in the database and notifies the user via the terminal.

[0463] This system allows for efficient management of refrigerator contents and reduces food waste. Furthermore, users can check the status of their refrigerator anytime, anywhere, making it easier to plan their shopping and cooking. It also contributes to reducing food waste by managing expiration dates.

[0464] The processing flow will be explained below.

[0465] Step 1:

[0466] The server periodically activates a camera installed inside the refrigerator to capture images of the inside of the refrigerator. For example, the camera is set to activate every time the refrigerator door is closed.

[0467] Step 2:

[0468] The server receives the captured image. Since the received image may be imperfect as it is, it performs preprocessing such as noise removal and color correction. For example, if the image contains noise, a filter is applied to remove the noise.

[0469] Step 3:

[0470] The server then inputs the pre-processed images into an object recognition algorithm, which uses advanced object recognition techniques such as YOLO (You Only Look Once) and Faster R-CNN to identify the food items in the refrigerator.

[0471] Step 4:

[0472] The server identifies the type and quantity of food based on the results of object recognition. For example, if one cabbage and two tomatoes are identified from the image, the server extracts the information about each item as data.

[0473] Step 5:

[0474] The server stores the extracted food information in a database, compares it with existing data, and reflects any changes in stock or the addition of new food items. For example, if cabbage is already registered in the database, the quantity is updated.

[0475] Step 6:

[0476] The server periodically sends updated information about the contents of the refrigerator to the device, which then displays this information so the user can check the status of the refrigerator. For example, a notification might appear on the device saying, "There is one cabbage and two tomatoes in the refrigerator."

[0477] Step 7:

[0478] The server monitors the expiration dates of food products and sends notifications to the device when the expiration date is approaching. For example, it sends a warning to the device saying, "The expiration date of the cabbage is tomorrow."

[0479] Step 8:

[0480] The user asks a question about the status of the refrigerator via voice or text, for example, "How many tomatoes are there in the refrigerator?"

[0481] Step 9:

[0482] The terminal receives the query and sends the request to the server.

[0483] Step 10:

[0484] The server analyzes the question using natural language processing and queries the database to generate an appropriate answer, such as "There are two tomatoes."

[0485] Step 11:

[0486] The terminal will notify the user of the generated answer, for example, "There are two tomatoes" will be displayed on the terminal.

[0487] Example 1

[0488] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0489] In conventional refrigerator management systems, managing food inventory and expiration dates is often done manually, requiring users to frequently check the contents of the refrigerator. There is also a high risk of incorrect information being entered, which can lead to discrepancies between actual inventory and records. Furthermore, there is an insufficient system for users to quickly respond to questions about the food in the refrigerator.

[0490] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0491] In this invention, the server includes means for acquiring images of the inside of the refrigerator, means for preprocessing the images, means for analyzing the preprocessed images to identify food items, means for saving information about the identified food items in a database, means for notifying a user terminal of information in the database, and means for analyzing questions from users using natural language processing and generating responses based on information in the database. This allows for accurate real-time management of food items in the refrigerator and enables users to easily check the inventory status and expiration dates in the refrigerator. Furthermore, a question-answering function using natural language processing improves user convenience.

[0492] "Means for acquiring images inside the refrigerator" refers to a function that periodically takes images of the food and containers inside the refrigerator using a photographic device such as a camera installed inside the refrigerator and acquires them as digital data.

[0493] "Preprocessing" refers to a series of processes that involve processing the acquired image data, such as noise removal and color correction, to improve the quality of the image.

[0494] "Analysis" or "image analysis" refers to the use of algorithms and artificial intelligence to analyze pre-processed image data and recognize and identify objects (in this case, food) within the image.

[0495] "Means for identifying food" refers to the function of using image analysis to identify individual food items in the refrigerator and clarify their type and quantity.

[0496] "Means for storing in a database" refers to the technology for recording and storing information on identified food (such as type, quantity, location, etc.) in a database as digital data.

[0497] "Means for notifying user devices" refers to the function of analyzing information stored in the database and sending necessary notifications in real time to the user's electronic device (such as a smartphone or tablet).

[0498] "Natural language processing" refers to technology that understands, analyzes, and responds appropriately to natural human language. In this context, it includes the ability for a system to generate appropriate answers to user inquiries.

[0499] "Means for generating a response" refers to technology that uses natural language processing to understand a user's question, constructs an appropriate response based on information in a database, and notifies the user's device of this response.

[0500] "Means for sending notifications based on expiration dates" refers to a function that monitors expiration date information for each food item stored in the database and automatically sends notifications to users when the expiration date approaches.

[0501] The present invention relates to a system for managing food in a refrigerator in real time. The program processing of this system is specifically explained below. Specific examples of the system's hardware and software include cameras, servers, user terminals, and artificial intelligence models.

[0502] 1. Installing and starting the camera

[0503] The server installs high-resolution cameras at key locations inside the refrigerator. Each time the refrigerator door is closed, the cameras are automatically activated and capture images of the interior. A typical example of such cameras is a general-purpose digital camera.

[0504] Example: When the refrigerator door is closed, the server sets the camera to immediately take pictures of the vegetable compartment and main storage area, allowing the overall status of the refrigerator to be grasped in real time.

[0505] 2. Image Preprocessing

[0506] The server preprocesses the acquired image data using libraries such as OpenCV. This preprocessing includes noise reduction and color correction, improving the quality of the images and the accuracy of subsequent analysis.

[0507] Example: The server applies noise filtering and adjusts the brightness and contrast of the image to clarify fuzzy areas.

[0508] 3. Image Analysis

[0509] The pre-processed image data is then fed into an object recognition algorithm such as YOLO (You Only Look Once) to identify the food items. The server then analyzes the data and identifies the type and quantity of food items in the refrigerator.

[0510] Example: For example, the server uses an object recognition algorithm to identify one cabbage and two tomatoes in an image.

[0511] 4. Data storage

[0512] The server stores the information of the identified food in a database using MySQL or PostgreSQL, comparing it with existing data and updating it immediately if there are any changes.

[0513] Example: When a new cabbage is added, the server stores it in the database as "1 cabbage." If a cabbage is already registered, the server updates the quantity.

[0514] 5. Notice to Users

[0515] The server sends the latest information about the contents of the refrigerator to the user's device. Users can check the status of the refrigerator in real time via their smartphone or tablet. The server also notifies users when the expiration date of each food item is approaching.

[0516] Example: The user device displays "There is one cabbage and two tomatoes in the refrigerator." If the expiration date is one day away, the device displays a notification saying "The cabbage's expiration date is tomorrow."

[0517] 6. Answering questions using natural language processing

[0518] When a user asks a question about the status of the refrigerator by voice or text, the user device sends the question to the server, which analyzes the question, checks the database, and generates an appropriate answer.

[0519] Example: When a user asks, "How many tomatoes are there?", the server generates the answer "There are two tomatoes" based on the information in the database and notifies the user via their device.

[0520] Prompt Sentence Examples

[0521] 1. "Please tell me how many tomatoes are in the refrigerator."

[0522] 2. "When is the expiration date for cabbage?"

[0523] This system allows users to accurately manage the contents of their refrigerator in real time, greatly improving user convenience. It allows users to efficiently grasp food inventory and expiration dates, reducing food waste and making shopping and cooking planning easier. Furthermore, a question-and-answer function using natural language processing allows users to quickly and accurately obtain information about what's inside their refrigerator.

[0524] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0525] Step 1:

[0526] Camera installation and startup

[0527] The server installs a high-resolution camera inside the refrigerator, which automatically activates every time the door is closed.

[0528] Input: Refrigerator door closing trigger signal

[0529] Data processing and calculation: Camera activation and image capture

[0530] Output: High-resolution image data of the inside of the refrigerator

[0531] Specific operation: When the refrigerator door is closed, the camera automatically starts up, takes a picture of the inside of the refrigerator, and sends it to the server.

[0532] Step 2:

[0533] Image preprocessing

[0534] The server performs pre-processing such as noise removal and color correction on the received image data.

[0535] Input: High-resolution image data of the inside of the refrigerator

[0536] Data processing and calculation: noise filtering, color correction

[0537] Output: Preprocessed image data

[0538] What it does: The server uses libraries such as OpenCV to remove noise from the image and balance the colors to produce a clear image.

[0539] Step 3:

[0540] Image analysis

[0541] The server inputs the preprocessed image data into an object recognition algorithm (e.g., YOLO) to identify the food item.

[0542] Input: Preprocessed image data

[0543] Data processing and calculation: Analysis using object recognition algorithms

[0544] Output: Identified food type and quantity data

[0545] How it works: The analysis algorithm scans the image and identifies foods such as cabbage and tomatoes, and then identifies the type and quantity of each food item and stores it as data.

[0546] Step 4:

[0547] Data storage

[0548] The server stores the identified food information in a database.

[0549] Input: Identified food type and quantity data

[0550] Data processing and calculation: writing information to the database, updating existing data

[0551] Output: A database containing the latest food information

[0552] Specific operation: The server uses MySQL or PostgreSQL to record food information in a database and compare it with existing data to update it.

[0553] Step 5:

[0554] User Notification

[0555] The server notifies the user terminal of the latest refrigerator information.

[0556] Input: A database containing the latest food information

[0557] Data processing and calculation: Generates notification messages and sends them to user terminals

[0558] Output: Latest refrigerator information displayed on the user's device

[0559] Specific operation: The server retrieves the latest information from the database and displays the status of the refrigerator on the user's device. For example, it displays "There is one cabbage and two tomatoes in the refrigerator."

[0560] Step 6:

[0561] Answering questions using natural language processing

[0562] The user asks questions about the status of the refrigerator by voice or text, and the device sends the questions to the server, which analyzes the questions and generates appropriate answers.

[0563] Input: User question (voice or text)

[0564] Data processing and calculation: Question analysis using natural language processing, database matching, and response generation

[0565] Output: Response message to be sent to the user

[0566] Specific operation: When a user asks "How many tomatoes are there?", the server analyzes the question, retrieves information from the database, and sends the answer "There are two tomatoes" to the user's terminal.

[0567] (Application example 1)

[0568] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0569] In modern households, managing food in the refrigerator is extremely time-consuming, and food waste due to expired food and unnecessary duplicate purchases is a major problem. It is also difficult to keep track of the refrigerator's contents while out or cooking, making it difficult to efficiently plan shopping and meals. Furthermore, there is a lack of systems that can quickly and accurately respond to users' questions about the refrigerator's contents, and there is a need to solve this problem.

[0570] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0571] In this invention, the server includes means for acquiring images of the inside of the refrigerator, means for processing the images and identifying foods, means for storing information about the identified foods in a database, means for notifying a user terminal of the information in the database, means for analyzing questions from users using natural language processing and generating responses based on the information in the database, means for users to check the information in the database in real time through an application installed on a smart device, and means for notifying a user terminal of foods that are approaching their expiration date. This allows for efficient management of foods in the refrigerator, reduces food waste, and enables users to keep track of the status of their refrigerator anytime, anywhere.

[0572] The "means for acquiring images inside the refrigerator" is a mechanism for capturing images of the inside of the refrigerator using a camera installed inside the refrigerator and acquiring the image data.

[0573] The "means for processing the image and identifying the food" refers to an algorithm or program that analyzes the acquired image data and identifies the various foods present in the refrigerator.

[0574] The "means for storing information on the identified food in a database" is a mechanism for recording and managing data such as the type and quantity of food identified through analysis in a database.

[0575] The "means for notifying the user terminal of the information in the database" is a mechanism for transmitting and displaying the food information stored in the database to a terminal such as a smartphone or tablet owned by the user.

[0576] "Means for analyzing questions from users using natural language processing and generating responses based on information in the database" refers to a mechanism that uses natural language processing technology to analyze voice or text questions from users, compares them with information in the database, and generates appropriate responses.

[0577] "A means by which users can check the information in the database in real time through an application installed on a smart device" refers to a mechanism that allows users to view the latest food information in the database in real time using a dedicated application installed on a smart device such as a smartphone.

[0578] The "means for notifying a user terminal of food products approaching their expiration date" is a mechanism that sends a notification to a user terminal about food products that are approaching their expiration date based on food information in the database.

[0579] The present invention provides a system that efficiently manages food in a refrigerator, reduces food waste, and allows a user to grasp the status of the refrigerator anytime, anywhere.

[0580] A system for implementing the present invention comprises the following components:

[0581] 1. Image acquisition method:

[0582] There are multiple cameras installed inside the refrigerator, and these cameras periodically capture images of the inside of the refrigerator. For example, every time the refrigerator door is closed, a camera is automatically activated and takes a picture of the inside of the refrigerator.

[0583] 2. Imaging and Food Identification Methods:

[0584] The captured images are first preprocessed with noise removal and color correction, then an object recognition algorithm (e.g., a deep learning model using Keras) is used to identify food items. For example, cabbage and tomatoes are identified, and the quantity of each is also determined.

[0585] 3. Database storage method:

[0586] Information on the identified food (type, quantity, etc.) is stored in a database such as SQLite. If an update is required, this information is compared with existing data and updated to the latest version.

[0587] 4. User terminal notification means:

[0588] The identified food information is sent in real time to the user's device, such as a smartphone or tablet. The user can check the status of the refrigerator through a dedicated application. For example, information such as "There is one cabbage and two tomatoes in the refrigerator" is displayed.

[0589] 5. Natural Language Processing Tools:

[0590] Voice and text questions from users are sent to the server via their device. The questions are analyzed using natural language processing technology and compared with information in a database to generate an appropriate answer. For example, in response to the question "How many tomatoes are there?", an answer such as "There are two tomatoes" is generated and notified to the user's device.

[0591] 6. Best before date notification method:

[0592] When the expiration date of an identified food item is approaching, the server sends a notification to the user's terminal. This notification allows the user to take action early on regarding the food item that is approaching its expiration date. For example, the user may receive a notification saying, "The expiration date of the cabbage is tomorrow."

[0593] This allows users to effectively manage the food in their refrigerator and reduce food waste. It also allows users to check the status of their refrigerator in real time while they are out or shopping, helping them avoid unnecessary purchases. Furthermore, users can instantly ask questions about the status of their refrigerator and get answers, enabling more efficient meal preparation.

[0594] Example 1:

[0595] When you open the smartphone app and tap the "Show latest refrigerator status" button, the app requests the latest food information from the server and displays it on the screen. You can instantly see information such as "There is one cabbage and two tomatoes in the refrigerator."

[0596] Example 2:

[0597] When the expiration date approaches, a notification will be sent to the smartphone, with the message "The cabbage's expiration date is tomorrow," allowing the user to use the cabbage sooner.

[0598] Example prompt for a generative AI model:

[0599] A user asks: "How many tomatoes are in the fridge right now?"

[0600] Prompt for the AI ​​model: "The user wants to know how many tomatoes are in the refrigerator. Please check your database and generate the correct answer."

[0601] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0602] Step 1:

[0603] Capture images of the inside of the refrigerator

[0604] The server activates a camera installed inside the refrigerator to capture an image of the inside of the refrigerator.

[0605] Input: Latest status of the refrigerator

[0606] Output: Image data of the inside of the refrigerator (e.g., JPEG image file)

[0607] Step 2:

[0608] Image preprocessing

[0609] The server performs preprocessing such as noise removal and color correction on the acquired image data.

[0610] Input: Raw captured image data

[0611] Output: Pre-processed image data (e.g., noise-removed and color-optimized images)

[0612] Step 3:

[0613] Food Identification

[0614] The server inputs the preprocessed image data into a food identification algorithm to identify the food items, for example, using a deep learning model (e.g., using Keras) to identify each food item in the refrigerator.

[0615] Input: Preprocessed image data

[0616] Output: A list of identified foods (e.g., {'cabbage': 1, 'tomato': 2})

[0617] Step 4:

[0618] Saving to a database

[0619] The server stores the identified food information (type, quantity) in a database such as SQLite, compares it with existing data, and updates it as necessary.

[0620] Input: List of identified foods

[0621] Output: Latest food information registered in the database

[0622] Step 5:

[0623] Notification to user device

[0624] The server sends the latest food information stored in the database to the user's smartphone or tablet, where the user can check this information in real time through a dedicated application.

[0625] Input: Latest food information from the database

[0626] Output: Food information displayed on the user's device (application)

[0627] Step 6:

[0628] Answering questions using natural language processing

[0629] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server. The server uses natural language processing technology to analyze the question, compares it with information in a database, generates an appropriate answer, and sends it to the device. For example, if a user asks, "How many tomatoes are there?", the server generates the answer, "There are two tomatoes."

[0630] Input: User question (voice or text)

[0631] Output: Answer based on the database (e.g., there are two tomatoes)

[0632] Step 7:

[0633] Best before date notification

[0634] The server uses the food information in the database to identify foods that are approaching their expiration date and sends a notification to the user's smartphone, such as "The expiration date for the cabbage is tomorrow."

[0635] Input: Food information and its expiration date in the database

[0636] Output: Expiration warning notification (e.g., the expiration date for cabbage is tomorrow)

[0637] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0638] This invention combines an emotion engine with a system for managing the contents of a refrigerator in real time. It captures images of the inside of the refrigerator, processes them to identify food items, stores the information in a database, and notifies the user as needed. It also has the ability to generate answers to user questions using natural language processing, and to analyze user emotions to customize notifications and responses.

[0639] Specific Embodiments of the System

[0640] 1. Installing and starting the camera

[0641] The server installs multiple cameras inside the refrigerator and periodically activates these cameras to capture images inside the refrigerator.

[0642] For example, a dedicated camera is installed in the vegetable compartment, and every time the refrigerator door is closed, the camera activates and captures an image of the entire vegetable compartment. The server receives this image in real time.

[0643] 2. Image Preprocessing and Analysis

[0644] The server pre-processes the received images, which includes noise reduction, color correction, etc.

[0645] The pre-processed images are then fed into an object recognition algorithm to identify the food.

[0646] For example, cabbage and tomatoes are identified from the captured image, and the server identifies the type and quantity of these foods.

[0647] 3. Data storage

[0648] The server stores the identified food information in a database, compares it with existing data, and updates it if there are any changes.

[0649] For example, if a new cabbage is added, it will be saved in the database as "1 cabbage." If a cabbage is already registered, the quantity will be updated.

[0650] 4. Notice to Users

[0651] The server sends the latest information about the contents of the refrigerator to the terminal, allowing the user to check the status of the refrigerator in real time via a smartphone or tablet.

[0652] For example, the terminal will display, "There is one cabbage and two tomatoes in the refrigerator." Furthermore, if the expiration date approaches, the server will send a notification to the terminal, warning the user, "The expiration date of the cabbage is tomorrow."

[0653] 5. Answering questions using natural language processing

[0654] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server.

[0655] The server analyzes the question, checks its database and generates an appropriate answer.

[0656] For example, if a user asks, "How many tomatoes are there?", the server generates an answer based on the information in the database: "There are two tomatoes," and notifies the user via the terminal.

[0657] 6. Implementing the Emotion Engine

[0658] The server is equipped with an emotion engine that recognizes emotions from the user's voice, text, facial expressions, etc. This allows it to analyze the user's emotions and customize corresponding notifications and responses.

[0659] For example, if the user is in a hurry, the system will calmly display a notification saying, "The cabbage's expiration date is approaching, so please use it up quickly."

[0660] 7. Customize notifications based on emotions

[0661] The server customizes the notification content and response based on the user's emotional information obtained by the emotion engine.

[0662] For example, if a user looks anxious, a notification offering assistance such as "I have two tomatoes, can I help you?" can be sent.

[0663] This system not only allows for efficient management of the contents of the refrigerator, but also takes into account the user's emotions, providing a more friendly user experience. By recognizing emotions and optimizing notifications and responses, we can expect to reduce user stress and improve the efficiency of food management.

[0664] The processing flow will be explained below.

[0665] Step 1:

[0666] The server periodically activates the camera installed inside the refrigerator to capture images of the inside of the refrigerator. For example, every time the refrigerator door is closed, the camera is automatically activated to capture a full image of the inside.

[0667] Step 2:

[0668] The server receives the captured image, which may be incomplete as it is, so it undergoes pre-processing such as noise reduction and color correction.

[0669] Step 3:

[0670] The server then inputs the preprocessed images into an object recognition algorithm, specifically using object recognition techniques such as YOLO (You Only Look Once) and Faster R-CNN to identify the food in the image.

[0671] Step 4:

[0672] The server identifies the type and quantity of food based on the results of object recognition. For example, if one cabbage and two tomatoes are identified from the image, the server extracts the information about each item as data.

[0673] Step 5:

[0674] The server stores the extracted food information in a database, compares it with existing data, and reflects any changes in stock or the addition of new food items. For example, if cabbage is already registered in the database, the quantity is updated.

[0675] Step 6:

[0676] The server periodically sends updated information about the contents of the refrigerator to the terminal, allowing the user to check the status of the refrigerator in real time via a smartphone or tablet.

[0677] Step 7:

[0678] The server monitors the expiration dates of food products and sends notifications to the device when the expiration date is approaching. For example, it sends a warning to the device saying, "The expiration date of the cabbage is tomorrow."

[0679] Step 8:

[0680] The user asks a question via voice or text about what's going on in the refrigerator, for example, "How many tomatoes are there in the refrigerator?"

[0681] Step 9:

[0682] The terminal receives the user's query and sends the request to the server.

[0683] Step 10:

[0684] The server analyzes the question using natural language processing and queries the database to generate an appropriate answer, such as "There are two tomatoes."

[0685] Step 11:

[0686] The terminal notifies the user of the generated answer, for example by displaying the message "There are two tomatoes" on the terminal.

[0687] Step 12:

[0688] The server analyzes the user's emotions using an emotion engine that recognizes emotions from the user's voice, text, facial expressions, etc. For example, if the user's tone of voice sounds impatient, the server recognizes that emotion.

[0689] Step 13:

[0690] The server customizes the notification content and response based on the user's emotional information obtained by the emotion engine. For example, if the user is feeling anxious, it generates a notification offering support, such as "I have two tomatoes. Can I help you?"

[0691] Step 14:

[0692] The device will display customized notifications to the user, for example, if the user is feeling busy, it will calmly display a notification saying, "The cabbage is nearing its expiration date, please use it up soon."

[0693] In this way, this system not only improves the efficiency of inventory management in the refrigerator, but also provides a more user-friendly interface by providing notifications and responses based on emotions.

[0694] Example 2

[0695] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0696] Conventional refrigerator management systems have the drawback of requiring a lot of time and effort to understand the status of food in the refrigerator, resulting in low user convenience. Furthermore, they lack the ability to provide appropriate notifications and responses that reflect the user's emotions and state of mind, making it difficult to improve the user experience. Furthermore, they lack the functionality to manage food expiration dates or provide specific advice, which can easily lead to food waste.

[0697] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0698] In this invention, the server includes means for capturing images of the inside of the refrigerator using a camera, means for preprocessing the images and identifying food items using an object recognition algorithm, means for saving information about the identified food items in a database, means for notifying a user terminal of the information in the database, means for analyzing questions from users using natural language processing and generating responses based on the information in the database, means for analyzing the user's voice, text, and facial expressions to recognize emotions, and means for customizing notification content and responses based on the recognized emotions. This makes food management in the refrigerator more efficient and enables appropriate responses based on the user's emotions, which is expected to improve the user experience and reduce food waste.

[0699] The "camera" is a photographing device installed to capture images inside the refrigerator.

[0700] "Preprocessing" refers to processes such as noise removal and color correction that are performed to make the acquired image easier to analyze.

[0701] An "object recognition algorithm" is a computational method for identifying food in an image and determining its type and quantity.

[0702] "Database" means a collection of digital data for storing and managing information about identified foods.

[0703] A "user terminal" is a device that a user uses to check information about the contents of the refrigerator and to communicate with the system.

[0704] "Natural language processing" is an artificial intelligence technology that analyzes questions from users and generates appropriate responses.

[0705] An "emotion engine" is a system that analyzes a user's voice, text, and facial expressions to recognize emotions.

[0706] "Customizing notification content and responses" refers to the process of adjusting the content of notifications and responses based on the user's emotions.

[0707] This invention relates to a system for efficiently managing food in a refrigerator, aiming to improve user convenience and reduce stress. The system consists of a camera installed in the refrigerator, a server, a user terminal, an object recognition algorithm, a database, natural language processing, and an emotion engine.

[0708] Hardware and Software Configuration

[0709] camera

[0710] There are multiple high-resolution cameras installed inside the refrigerator. These cameras (e.g., high-resolution cameras) have the ability to capture an image every time the refrigerator door is closed.

[0711] server

[0712] The server receives the image data sent from the camera and begins processing. First, the image is pre-processed. Specifically, camera noise is removed and color correction is performed. After this pre-processing is complete, the image is input into an object recognition algorithm (e.g., YOLO) to identify the type and quantity of food.

[0713] Database

[0714] The server stores the identified food information in a database (e.g., MySQL). When a new food is added, the information is updated in the database and existing food information is updated.

[0715] User terminal

[0716] The user terminal is a device such as a smartphone or tablet that allows the user to check the information inside the refrigerator. The latest food information is sent from the server to the terminal, allowing the user to check the status of the refrigerator in real time. For example, information such as "There is one cabbage and two tomatoes in the refrigerator" is displayed.

[0717] Natural Language Processing

[0718] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server. The server uses natural language processing (e.g., BERT) to analyze the question and compares it with information in the database to generate an appropriate answer. For example, the question "How many tomatoes are there?" will generate an answer such as "There are two tomatoes."

[0719] Emotion Engine

[0720] The server is equipped with an emotion engine (e.g., Affectiva SDK) that analyzes the user's voice, text, and facial expressions to recognize their emotions. This allows it to analyze the user's emotions and customize corresponding notifications and responses. For example, if the user is feeling anxious, the server may notify them that "The expiration date of the cabbage is approaching, so please use it up quickly."

[0721] Customize notifications based on emotions

[0722] The server customizes the notification content and response based on the user's emotional information obtained by the emotion engine. For example, if the user looks anxious, a notification offering support such as "There are two tomatoes. Can I help you?" will be sent.

[0723] Examples of concrete examples and prompts

[0724] For example, whenever a user opens or closes the refrigerator, the camera captures an image and the server updates the food information based on this. For example, when a user asks, "How many tomatoes are there?", the server responds via the device with, "There are two tomatoes."

[0725] Example prompt sentence:

[0726] "Capture an image of the inside of your refrigerator, identify the food in the crisper, and store it in a database."

[0727] This system allows users to manage the food in their refrigerator efficiently and without waste, and also provides a pleasant user experience by receiving appropriate notifications and responses based on their emotions.

[0728] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0729] Step 1:

[0730] Camera installation and image acquisition

[0731] The server installs multiple high-resolution cameras inside the refrigerator. Each time the refrigerator door is closed, these cameras are activated and capture images of the inside of the refrigerator. Specifically, the cameras are automatically activated when the refrigerator door sensor detects that the door is closing, and images of multiple locations inside the refrigerator are taken.

[0732] Input: Actual view inside the refrigerator, door sensor signal

[0733] Output: High-resolution image of the inside of the refrigerator

[0734] Step 2:

[0735] Image preprocessing

[0736] The server receives the captured images and performs pre-processing, such as noise reduction and color correction to improve image quality, resulting in image data ready to be input into object recognition algorithms.

[0737] Input: High-resolution image

[0738] Output: Preprocessed image

[0739] Step 3:

[0740] object recognition

[0741] The server inputs the preprocessed image into an object recognition algorithm (e.g., YOLO). The algorithm identifies the type and quantity of food and identifies the results. Specifically, the YOLO algorithm analyzes each pixel in the image and identifies specific objects (e.g., cabbage, tomato).

[0742] Input: Preprocessed image

[0743] Output: Identified food types and quantities

[0744] Step 4:

[0745] Data storage

[0746] The server stores the identified food information in a database (e.g., MySQL). When a new food is added, the information is saved in the database and existing food information is updated as necessary. Specifically, the new and existing data are compared and the differences are updated.

[0747] Input: Type and quantity of food identified

[0748] Output: Updated database contents

[0749] Step 5:

[0750] Notification to user device

[0751] The server sends the latest food information to the user's device (smartphone or tablet), allowing the user to check the status of the refrigerator in real time. For example, a notification might say, "There is one cabbage and two tomatoes in the refrigerator."

[0752] Input: Updated database contents

[0753] Output: Food information displayed on the user's device

[0754] Step 6:

[0755] Answering questions using natural language processing

[0756] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server. The server uses natural language processing (e.g., BERT) to analyze the question, compares it with information in the database, generates an appropriate answer, and notifies the user via the device. For example, the question "How many tomatoes are there?" is answered with "There are two tomatoes."

[0757] Input: User question (voice or text)

[0758] Output: Response to user terminal

[0759] Step 7:

[0760] Implementing the Emotion Engine

[0761] The server is equipped with an emotion engine (e.g., Affectiva SDK) that analyzes the user's voice, text, and facial expressions to recognize their emotions. This allows it to analyze the user's emotions and generate appropriate notifications and responses. For example, if the user is feeling impatient, the server will notify them by saying, "The cabbage's expiration date is approaching, so please use it up quickly."

[0762] Input: User voice, text, facial expressions

[0763] Output: Notification based on user sentiment

[0764] Step 8:

[0765] Customize notifications based on emotions

[0766] The server customizes the notification content and response based on the user's emotional information obtained from the emotion engine. For example, if the user looks anxious, the server sends a notification offering support such as, "There are two tomatoes. Can I help you?"

[0767] Input: User's emotional information

[0768] Output: Customized notification content

[0769] (Application example 2)

[0770] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0771] Conventional refrigerator management systems and store management systems only had the function of managing inventory information and were unable to respond or notify users based on their emotions. Furthermore, when it came to in-store product management, it was difficult to check inventory in real time, making it impossible to respond quickly to customer needs. This resulted in a poor user experience and a decline in the efficiency of inventory management and customer service.

[0772] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring images of the inside of the refrigerator and the inside of the store, means for processing the images and identifying food and products, and means for analyzing customer emotions and customizing notifications and responses. This enables appropriate responses according to user emotions and real-time management of inventory in the refrigerator and the store.

[0773] A "server" is a device that provides computing resources and storage, analyzes and stores various types of data, and manages communications.

[0774] The "means for acquiring images inside the refrigerator" is a technology for capturing images using a camera installed inside the refrigerator and acquiring them as data.

[0775] The "means for processing the image and identifying the food" refers to a technology for preprocessing the acquired image and identifying the food using an object recognition algorithm.

[0776] The "means for storing information on the identified food in a database" refers to a technique for recording data on the type and quantity of the identified food in a database.

[0777] "Means for notifying the user terminal of the information in the database" refers to technology that sends food information and update information stored in the database to the user's terminal such as a smartphone or tablet.

[0778] "Means for analyzing questions from users using natural language processing and generating responses based on information in the database" refers to technology that analyzes voice or text questions uttered by users and generates appropriate answers based on information in the database.

[0779] The "means for acquiring images within the store" refers to a technology for capturing images using cameras installed within the store and acquiring them as data.

[0780] The "means for processing the image and identifying the product" refers to a technology for preprocessing images acquired in a store and identifying the product using an object recognition algorithm.

[0781] "Means for analyzing customer emotions and customizing notifications and responses" refers to technology that reads emotions from a customer's facial expressions, voice, text, etc., and uses that information to customize the content of notifications and responses.

[0782] "Means for periodically reacquiring images of the inside of the refrigerator and the inside of the store and updating them to the latest state" refers to a technology that periodically operates the camera to acquire the latest images and keep the contents of the database up to date.

[0783] "Means for sending notifications to a user terminal based on the expiration dates of the food and products" refers to a technology that sends warnings and notifications to a user terminal about products that are approaching their expiration date based on expiration date information of stored food and products.

[0784] The embodiments for carrying out the present invention are as follows.

[0785] 1. System Overview

[0786] This system improves the efficiency of inventory management in refrigerators and physical stores, and provides responses based on the user's emotions. The system mainly consists of a server, a terminal, a camera, an emotion analysis engine, an object recognition algorithm, and a database.

[0787] 2. Hardware and Software

[0788] Cameras: Installed in refrigerators and in stores. Example: Logitech C920 webcam

[0789] Server: Responsible for data processing and storage. Example: AWS EC2 instance

[0790] Terminal: The device on which the user receives information. Examples: smartphone, tablet

[0791] software:

[0792] OpenCV: Image capture and preprocessing library

[0793] Object Recognition Algorithm: Identifying food and goods using YOLOv3 and ResNet

[0794] Database: Manage inventory data with MySQL or Firebase

[0795] Sentiment analysis engine: Uses Microsoft Azure's Emotion API

[0796] Natural Language Processing: Question Answering with GPT-3

[0797] 3. System Operation

[0798] 1. Image Acquisition

[0799] The server periodically activates the cameras installed in the refrigerator and in the store to capture the latest images. For example, the server can activate the camera and capture an image every time the refrigerator door is closed.

[0800] 2. Image Preprocessing and Analysis

[0801] The server preprocesses the acquired images using OpenCV. Specifically, it performs noise removal and color correction. The preprocessed images are input into an object recognition algorithm, and the server identifies food and products. For example, it identifies cabbages and tomatoes from an image of a vegetable drawer and determines their quantities.

[0802] 3. Data storage

[0803] The server stores the identified food and product information in a database. If there are any changes compared to existing data, the information is updated. For example, if a new item, "1 cabbage," is added to the database, the information is updated.

[0804] 4. Notice to Users

[0805] The server sends the latest information about what's in the refrigerator and in the store to the terminal. Users can check inventory status in real time via their smartphone or tablet. For example, the terminal will display information like, "There is one cabbage and two tomatoes in the refrigerator." In addition, it will send a warning if the expiration date is approaching.

[0806] 5. Answering questions using natural language processing

[0807] When a user asks a question about the inventory in the refrigerator or store by voice or text, the device sends the question to the server. The server uses GPT-3 to analyze the question, compares it with the database, and generates an appropriate answer. For example, if a user asks, "How many tomatoes are there?", the device will respond, "There are two tomatoes."

[0808] 6. Implementing the Emotion Engine

[0809] The server is equipped with an emotion analysis engine that recognizes emotions from the user's voice, text, facial expressions, etc. This allows it to analyze the user's emotions and customize corresponding notifications and responses.

[0810] 7. Customize notifications based on emotions

[0811] The server customizes notifications and responses based on the user's emotional information obtained from the emotion analysis engine. For example, if the user is in a hurry, the server will calmly display a notification such as, "The expiration date of the cabbage is approaching. Please use it up quickly." If the user is anxious, the server will send a notification offering support, such as, "You have two tomatoes. Is there anything I can help you with?"

[0812] 8. Examples of prompt sentences

[0813] "Tell me the stock situation"

[0814] "Do you have product A?"

[0815] Specific use cases

[0816] When a user asks "Do you have any cabbages?" on their smartphone, the server checks the database and responds, "We have one cabbage." If the user looks anxious, the server responds with a gentler response, such as, "We have two tomatoes. May I help you?"

[0817] Such a system enables flexible responses that adapt to the user's emotions, and allows for appropriate management of inventory status in refrigerators and stores.

[0818] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0819] Step 1:

[0820] The server activates the cameras installed in the refrigerator and in the store to capture images. Based on the request, the server obtains image data from the cameras and preprocesses it. Preprocessing includes noise reduction and color correction. The input is the raw image captured by the camera, and the output is the preprocessed image.

[0821] Step 2:

[0822] The server inputs the preprocessed images into an object recognition algorithm (e.g., YOLOv3 or ResNet) to identify food and store items. The input is the preprocessed image, and the output is the type and quantity of identified food or items. Specifically, the algorithm identifies objects in the image and returns their labels and locations.

[0823] Step 3:

[0824] The server stores the identified food and product information (type, quantity) in a database. It compares the data with existing data and updates it if there are any changes. The input is the identified food or product data, and the output is the updated database entry. For example, information such as "two tomatoes were added" is recorded in the database.

[0825] Step 4:

[0826] The server sends the latest information about what's in the refrigerator and the store to the user terminal. Here, it retrieves the latest information from the database and generates a message to send to the user terminal. The input is inventory information retrieved from the database, and the output is a notification message to be sent to the user terminal. For example, the message might say, "There is one cabbage and two tomatoes in the refrigerator."

[0827] Step 5:

[0828] The user asks a question about the inventory in the refrigerator or in the store by voice or text. The terminal sends the question to the server. The input to the terminal is the user's question, and the output is the question data to the server. For example, the terminal sends a user's question such as "How many tomatoes are there?" to the server.

[0829] Step 6:

[0830] The server uses GPT-3 to analyze the question, compare it with the database, and generate an appropriate answer. The input is the user's question data received from the device, and the output is the generated answer. Specifically, it uses natural language processing to understand the intent of the question and constructs an answer based on related information in the database.

[0831] Step 7:

[0832] The server uses an emotion analysis engine to analyze the user's emotions and recognizes emotions from voice, text, and facial expressions. The input is the user's voice data, text data, and image data, and the output is analyzed emotional information. Specifically, the emotion analysis engine analyzes this data and determines whether the user is anxious or impatient.

[0833] Step 8:

[0834] The server customizes notifications and responses based on the user's emotional information obtained by the emotion analysis engine. The input is the user's emotional information and the generated response information, and the output is a customized notification message. For example, if the user is in a hurry, the server sends a notification such as, "Remain calm. The expiration date of the cabbage is approaching. Please use it up quickly."

[0835] Step 9:

[0836] The server sends a customized notification to the user terminal. The input is a customized notification message, and the output is a notification displayed on the user terminal, allowing the user to receive accurate information.

[0837] Through the above steps, a system is realized in which the server, terminals, and users work together to manage inventory and handle customer service.

[0838] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0839] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0840] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0841] [Third embodiment]

[0842] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0843] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0844] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0845] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0846] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0847] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0848] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0849] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0850] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0851] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0852] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0853] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0854] This system manages the contents of a refrigerator in real time by capturing images of the inside of the refrigerator, processing the images to identify food items, storing the information in a database, and notifying the user as needed. It also has the ability to generate answers to user questions using natural language processing.

[0855] Specific Embodiments of the System

[0856] 1. Installing and starting the camera

[0857] The server installs multiple cameras inside the refrigerator and periodically activates these cameras to capture images inside the refrigerator.

[0858] For example, a dedicated camera is installed in the vegetable compartment, and every time the refrigerator door is closed, the camera activates and captures an image of the entire vegetable compartment. The server receives this image in real time.

[0859] 2. Image Preprocessing and Analysis

[0860] The server pre-processes the received images, which includes noise reduction, color correction, etc.

[0861] The pre-processed images are then fed into an object recognition algorithm to identify the food.

[0862] For example, cabbage and tomatoes are identified from the captured image, and the server identifies the type and quantity of these foods.

[0863] 3. Data storage

[0864] The server stores the identified food information in a database, compares it with existing data, and updates it if there are any changes.

[0865] For example, if a new cabbage is added, it will be saved in the database as "1 cabbage." If a cabbage is already registered, the quantity will be updated.

[0866] 4. Notice to Users

[0867] The server sends the latest information about the contents of the refrigerator to the terminal, allowing the user to check the status of the refrigerator in real time via a smartphone or tablet.

[0868] For example, the terminal will display, "There is one cabbage and two tomatoes in the refrigerator." Furthermore, if the expiration date approaches, the server will send a notification to the terminal, warning the user, "The expiration date of the cabbage is tomorrow."

[0869] 5. Answering questions using natural language processing

[0870] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server.

[0871] The server analyzes the question, checks its database and generates an appropriate answer.

[0872] For example, if a user asks, "How many tomatoes are there?", the server generates the answer "There are two tomatoes" based on the information in the database and notifies the user via the terminal.

[0873] This system allows for efficient management of refrigerator contents and reduces food waste. Furthermore, users can check the status of their refrigerator anytime, anywhere, making it easier to plan their shopping and cooking. It also contributes to reducing food waste by managing expiration dates.

[0874] The processing flow will be explained below.

[0875] Step 1:

[0876] The server periodically activates a camera installed inside the refrigerator to capture images of the inside of the refrigerator. For example, the camera is set to activate every time the refrigerator door is closed.

[0877] Step 2:

[0878] The server receives the captured image. Since the received image may be imperfect as it is, it performs preprocessing such as noise removal and color correction. For example, if the image contains noise, a filter is applied to remove the noise.

[0879] Step 3:

[0880] The server then inputs the pre-processed images into an object recognition algorithm, which uses advanced object recognition techniques such as YOLO (You Only Look Once) and Faster R-CNN to identify the food items in the refrigerator.

[0881] Step 4:

[0882] The server identifies the type and quantity of food based on the results of object recognition. For example, if one cabbage and two tomatoes are identified from the image, the server extracts the information about each item as data.

[0883] Step 5:

[0884] The server stores the extracted food information in a database, compares it with existing data, and reflects any changes in stock or the addition of new food items. For example, if cabbage is already registered in the database, the quantity is updated.

[0885] Step 6:

[0886] The server periodically sends updated information about the contents of the refrigerator to the device, which then displays this information so the user can check the status of the refrigerator. For example, a notification might appear on the device saying, "There is one cabbage and two tomatoes in the refrigerator."

[0887] Step 7:

[0888] The server monitors the expiration dates of food products and sends notifications to the device when the expiration date is approaching. For example, it sends a warning to the device saying, "The expiration date of the cabbage is tomorrow."

[0889] Step 8:

[0890] The user asks a question about the status of the refrigerator via voice or text, for example, "How many tomatoes are there in the refrigerator?"

[0891] Step 9:

[0892] The terminal receives the query and sends the request to the server.

[0893] Step 10:

[0894] The server analyzes the question using natural language processing and queries the database to generate an appropriate answer, such as "There are two tomatoes."

[0895] Step 11:

[0896] The terminal will notify the user of the generated answer, for example, "There are two tomatoes" will be displayed on the terminal.

[0897] Example 1

[0898] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0899] In conventional refrigerator management systems, managing food inventory and expiration dates is often done manually, requiring users to frequently check the contents of the refrigerator. There is also a high risk of incorrect information being entered, which can lead to discrepancies between actual inventory and records. Furthermore, there is an insufficient system for users to quickly respond to questions about the food in the refrigerator.

[0900] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0901] In this invention, the server includes means for acquiring images of the inside of the refrigerator, means for preprocessing the images, means for analyzing the preprocessed images to identify food items, means for saving information about the identified food items in a database, means for notifying a user terminal of information in the database, and means for analyzing questions from users using natural language processing and generating responses based on information in the database. This allows for accurate real-time management of food items in the refrigerator and enables users to easily check the inventory status and expiration dates in the refrigerator. Furthermore, a question-answering function using natural language processing improves user convenience.

[0902] "Means for acquiring images inside the refrigerator" refers to a function that periodically takes images of the food and containers inside the refrigerator using a photographic device such as a camera installed inside the refrigerator and acquires them as digital data.

[0903] "Preprocessing" refers to a series of processes that involve processing the acquired image data, such as noise removal and color correction, to improve the quality of the image.

[0904] "Analysis" or "image analysis" refers to the use of algorithms and artificial intelligence to analyze pre-processed image data and recognize and identify objects (in this case, food) within the image.

[0905] "Means for identifying food" refers to the function of using image analysis to identify individual food items in the refrigerator and clarify their type and quantity.

[0906] "Means for storing in a database" refers to the technology for recording and storing information on identified food (such as type, quantity, location, etc.) in a database as digital data.

[0907] "Means for notifying user devices" refers to the function of analyzing information stored in the database and sending necessary notifications in real time to the user's electronic device (such as a smartphone or tablet).

[0908] "Natural language processing" refers to technology that understands, analyzes, and responds appropriately to natural human language. In this context, it includes the ability for a system to generate appropriate answers to user inquiries.

[0909] "Means for generating a response" refers to technology that uses natural language processing to understand a user's question, constructs an appropriate response based on information in a database, and notifies the user's device of this response.

[0910] "Means for sending notifications based on expiration dates" refers to a function that monitors expiration date information for each food item stored in the database and automatically sends notifications to users when the expiration date approaches.

[0911] The present invention relates to a system for managing food in a refrigerator in real time. The program processing of this system is specifically explained below. Specific examples of the system's hardware and software include cameras, servers, user terminals, and artificial intelligence models.

[0912] 1. Installing and starting the camera

[0913] The server installs high-resolution cameras at key locations inside the refrigerator. Each time the refrigerator door is closed, the cameras are automatically activated and capture images of the interior. A typical example of such cameras is a general-purpose digital camera.

[0914] Example: When the refrigerator door is closed, the server sets the camera to immediately take pictures of the vegetable compartment and main storage area, allowing the overall status of the refrigerator to be grasped in real time.

[0915] 2. Image Preprocessing

[0916] The server preprocesses the acquired image data using libraries such as OpenCV. This preprocessing includes noise reduction and color correction, improving the quality of the images and the accuracy of subsequent analysis.

[0917] Example: The server applies noise filtering and adjusts the brightness and contrast of the image to clarify fuzzy areas.

[0918] 3. Image Analysis

[0919] The pre-processed image data is then fed into an object recognition algorithm such as YOLO (You Only Look Once) to identify the food items. The server then analyzes the data and identifies the type and quantity of food items in the refrigerator.

[0920] Example: For example, the server uses an object recognition algorithm to identify one cabbage and two tomatoes in an image.

[0921] 4. Data storage

[0922] The server stores the information of the identified food in a database using MySQL or PostgreSQL, comparing it with existing data and updating it immediately if there are any changes.

[0923] Example: When a new cabbage is added, the server stores it in the database as "1 cabbage." If a cabbage is already registered, the server updates the quantity.

[0924] 5. Notice to Users

[0925] The server sends the latest information about the contents of the refrigerator to the user's device. Users can check the status of the refrigerator in real time via their smartphone or tablet. The server also notifies users when the expiration date of each food item is approaching.

[0926] Example: The user device displays "There is one cabbage and two tomatoes in the refrigerator." If the expiration date is one day away, the device displays a notification saying "The cabbage's expiration date is tomorrow."

[0927] 6. Answering questions using natural language processing

[0928] When a user asks a question about the status of the refrigerator by voice or text, the user device sends the question to the server, which analyzes the question, checks the database, and generates an appropriate answer.

[0929] Example: When a user asks, "How many tomatoes are there?", the server generates the answer "There are two tomatoes" based on the information in the database and notifies the user via their device.

[0930] Prompt Sentence Examples

[0931] 1. "Please tell me how many tomatoes are in the refrigerator."

[0932] 2. "When is the expiration date for cabbage?"

[0933] This system allows users to accurately manage the contents of their refrigerator in real time, greatly improving user convenience. It allows users to efficiently grasp food inventory and expiration dates, reducing food waste and making shopping and cooking planning easier. Furthermore, a question-and-answer function using natural language processing allows users to quickly and accurately obtain information about what's inside their refrigerator.

[0934] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0935] Step 1:

[0936] Camera installation and startup

[0937] The server installs a high-resolution camera inside the refrigerator, which automatically activates every time the door is closed.

[0938] Input: Refrigerator door closing trigger signal

[0939] Data processing and calculation: Camera activation and image capture

[0940] Output: High-resolution image data of the inside of the refrigerator

[0941] Specific operation: When the refrigerator door is closed, the camera automatically starts up, takes a picture of the inside of the refrigerator, and sends it to the server.

[0942] Step 2:

[0943] Image preprocessing

[0944] The server performs pre-processing such as noise removal and color correction on the received image data.

[0945] Input: High-resolution image data of the inside of the refrigerator

[0946] Data processing and calculation: noise filtering, color correction

[0947] Output: Preprocessed image data

[0948] What it does: The server uses libraries such as OpenCV to remove noise from the image and balance the colors to produce a clear image.

[0949] Step 3:

[0950] Image analysis

[0951] The server inputs the preprocessed image data into an object recognition algorithm (e.g., YOLO) to identify the food item.

[0952] Input: Preprocessed image data

[0953] Data processing and calculation: Analysis using object recognition algorithms

[0954] Output: Identified food type and quantity data

[0955] How it works: The analysis algorithm scans the image and identifies foods such as cabbage and tomatoes, and then identifies the type and quantity of each food item and stores it as data.

[0956] Step 4:

[0957] Data storage

[0958] The server stores the identified food information in a database.

[0959] Input: Identified food type and quantity data

[0960] Data processing and calculation: writing information to the database, updating existing data

[0961] Output: A database containing the latest food information

[0962] Specific operation: The server uses MySQL or PostgreSQL to record food information in a database and compare it with existing data to update it.

[0963] Step 5:

[0964] User Notification

[0965] The server notifies the user terminal of the latest refrigerator information.

[0966] Input: A database containing the latest food information

[0967] Data processing and calculation: Generates notification messages and sends them to user terminals

[0968] Output: Latest refrigerator information displayed on the user's device

[0969] Specific operation: The server retrieves the latest information from the database and displays the status of the refrigerator on the user's device. For example, it displays "There is one cabbage and two tomatoes in the refrigerator."

[0970] Step 6:

[0971] Answering questions using natural language processing

[0972] The user asks questions about the status of the refrigerator by voice or text, and the device sends the questions to the server, which analyzes the questions and generates appropriate answers.

[0973] Input: User question (voice or text)

[0974] Data processing and calculation: Question analysis using natural language processing, database matching, and response generation

[0975] Output: Response message to be sent to the user

[0976] Specific operation: When a user asks "How many tomatoes are there?", the server analyzes the question, retrieves information from the database, and sends the answer "There are two tomatoes" to the user's terminal.

[0977] (Application example 1)

[0978] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0979] In modern households, managing food in the refrigerator is extremely time-consuming, and food waste due to expired food and unnecessary duplicate purchases is a major problem. It is also difficult to keep track of the refrigerator's contents while out or cooking, making it difficult to efficiently plan shopping and meals. Furthermore, there is a lack of systems that can quickly and accurately respond to users' questions about the refrigerator's contents, and there is a need to solve this problem.

[0980] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0981] In this invention, the server includes means for acquiring images of the inside of the refrigerator, means for processing the images and identifying foods, means for storing information about the identified foods in a database, means for notifying a user terminal of the information in the database, means for analyzing questions from users using natural language processing and generating responses based on the information in the database, means for users to check the information in the database in real time through an application installed on a smart device, and means for notifying a user terminal of foods that are approaching their expiration date. This allows for efficient management of foods in the refrigerator, reduces food waste, and enables users to keep track of the status of their refrigerator anytime, anywhere.

[0982] The "means for acquiring images inside the refrigerator" is a mechanism for capturing images of the inside of the refrigerator using a camera installed inside the refrigerator and acquiring the image data.

[0983] The "means for processing the image and identifying the food" refers to an algorithm or program that analyzes the acquired image data and identifies the various foods present in the refrigerator.

[0984] The "means for storing information on the identified food in a database" is a mechanism for recording and managing data such as the type and quantity of food identified through analysis in a database.

[0985] The "means for notifying the user terminal of the information in the database" is a mechanism for transmitting and displaying the food information stored in the database to a terminal such as a smartphone or tablet owned by the user.

[0986] "Means for analyzing questions from users using natural language processing and generating responses based on information in the database" refers to a mechanism that uses natural language processing technology to analyze voice or text questions from users, compares them with information in the database, and generates appropriate responses.

[0987] "A means by which users can check the information in the database in real time through an application installed on a smart device" refers to a mechanism that allows users to view the latest food information in the database in real time using a dedicated application installed on a smart device such as a smartphone.

[0988] The "means for notifying a user terminal of food products approaching their expiration date" is a mechanism that sends a notification to a user terminal about food products that are approaching their expiration date based on food information in the database.

[0989] The present invention provides a system that efficiently manages food in a refrigerator, reduces food waste, and allows a user to grasp the status of the refrigerator anytime, anywhere.

[0990] A system for implementing the present invention comprises the following components:

[0991] 1. Image acquisition method:

[0992] There are multiple cameras installed inside the refrigerator, and these cameras periodically capture images of the inside of the refrigerator. For example, every time the refrigerator door is closed, a camera is automatically activated and takes a picture of the inside of the refrigerator.

[0993] 2. Imaging and Food Identification Methods:

[0994] The captured images are first preprocessed with noise removal and color correction, then an object recognition algorithm (e.g., a deep learning model using Keras) is used to identify food items. For example, cabbage and tomatoes are identified, and the quantity of each is also determined.

[0995] 3. Database storage method:

[0996] Information on the identified food (type, quantity, etc.) is stored in a database such as SQLite. If an update is required, this information is compared with existing data and updated to the latest version.

[0997] 4. User terminal notification means:

[0998] The identified food information is sent in real time to the user's device, such as a smartphone or tablet. The user can check the status of the refrigerator through a dedicated application. For example, information such as "There is one cabbage and two tomatoes in the refrigerator" is displayed.

[0999] 5. Natural Language Processing Tools:

[1000] Voice and text questions from users are sent to the server via their device. The questions are analyzed using natural language processing technology and compared with information in a database to generate an appropriate answer. For example, in response to the question "How many tomatoes are there?", an answer such as "There are two tomatoes" is generated and notified to the user's device.

[1001] 6. Best before date notification method:

[1002] When the expiration date of an identified food item is approaching, the server sends a notification to the user's terminal. This notification allows the user to take action early on regarding the food item that is approaching its expiration date. For example, the user may receive a notification saying, "The expiration date of the cabbage is tomorrow."

[1003] This allows users to effectively manage the food in their refrigerator and reduce food waste. It also allows users to check the status of their refrigerator in real time while they are out or shopping, helping them avoid unnecessary purchases. Furthermore, users can instantly ask questions about the status of their refrigerator and get answers, enabling more efficient meal preparation.

[1004] Example 1:

[1005] When you open the smartphone app and tap the "Show latest refrigerator status" button, the app requests the latest food information from the server and displays it on the screen. You can instantly see information such as "There is one cabbage and two tomatoes in the refrigerator."

[1006] Example 2:

[1007] When the expiration date approaches, a notification will be sent to the smartphone, with the message "The cabbage's expiration date is tomorrow," allowing the user to use the cabbage sooner.

[1008] Example prompt for a generative AI model:

[1009] A user asks: "How many tomatoes are in the fridge right now?"

[1010] Prompt for the AI ​​model: "The user wants to know how many tomatoes are in the refrigerator. Please check your database and generate the correct answer."

[1011] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1012] Step 1:

[1013] Capture images of the inside of the refrigerator

[1014] The server activates a camera installed inside the refrigerator to capture an image of the inside of the refrigerator.

[1015] Input: Latest status of the refrigerator

[1016] Output: Image data of the inside of the refrigerator (e.g., JPEG image file)

[1017] Step 2:

[1018] Image preprocessing

[1019] The server performs preprocessing such as noise removal and color correction on the acquired image data.

[1020] Input: Raw captured image data

[1021] Output: Pre-processed image data (e.g., noise-removed and color-optimized images)

[1022] Step 3:

[1023] Food Identification

[1024] The server inputs the preprocessed image data into a food identification algorithm to identify the food items, for example, using a deep learning model (e.g., using Keras) to identify each food item in the refrigerator.

[1025] Input: Preprocessed image data

[1026] Output: A list of identified foods (e.g., {'cabbage': 1, 'tomato': 2})

[1027] Step 4:

[1028] Saving to a database

[1029] The server stores the identified food information (type, quantity) in a database such as SQLite, compares it with existing data, and updates it as necessary.

[1030] Input: List of identified foods

[1031] Output: Latest food information registered in the database

[1032] Step 5:

[1033] Notification to user device

[1034] The server sends the latest food information stored in the database to the user's smartphone or tablet, where the user can check this information in real time through a dedicated application.

[1035] Input: Latest food information from the database

[1036] Output: Food information displayed on the user's device (application)

[1037] Step 6:

[1038] Answering questions using natural language processing

[1039] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server. The server uses natural language processing technology to analyze the question, compares it with information in a database, generates an appropriate answer, and sends it to the device. For example, if a user asks, "How many tomatoes are there?", the server generates the answer, "There are two tomatoes."

[1040] Input: User question (voice or text)

[1041] Output: Answer based on the database (e.g., there are two tomatoes)

[1042] Step 7:

[1043] Best before date notification

[1044] The server uses the food information in the database to identify foods that are approaching their expiration date and sends a notification to the user's smartphone, such as "The expiration date for the cabbage is tomorrow."

[1045] Input: Food information and its expiration date in the database

[1046] Output: Expiration warning notification (e.g., the expiration date for cabbage is tomorrow)

[1047] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1048] This invention combines an emotion engine with a system for managing the contents of a refrigerator in real time. It captures images of the inside of the refrigerator, processes them to identify food items, stores the information in a database, and notifies the user as needed. It also has the ability to generate answers to user questions using natural language processing, and to analyze user emotions to customize notifications and responses.

[1049] Specific Embodiments of the System

[1050] 1. Installing and starting the camera

[1051] The server installs multiple cameras inside the refrigerator and periodically activates these cameras to capture images inside the refrigerator.

[1052] For example, a dedicated camera is installed in the vegetable compartment, and every time the refrigerator door is closed, the camera activates and captures an image of the entire vegetable compartment. The server receives this image in real time.

[1053] 2. Image Preprocessing and Analysis

[1054] The server pre-processes the received images, which includes noise reduction, color correction, etc.

[1055] The pre-processed images are then fed into an object recognition algorithm to identify the food.

[1056] For example, cabbage and tomatoes are identified from the captured image, and the server identifies the type and quantity of these foods.

[1057] 3. Data storage

[1058] The server stores the identified food information in a database, compares it with existing data, and updates it if there are any changes.

[1059] For example, if a new cabbage is added, it will be saved in the database as "1 cabbage." If a cabbage is already registered, the quantity will be updated.

[1060] 4. Notice to Users

[1061] The server sends the latest information about the contents of the refrigerator to the terminal, allowing the user to check the status of the refrigerator in real time via a smartphone or tablet.

[1062] For example, the terminal will display, "There is one cabbage and two tomatoes in the refrigerator." Furthermore, if the expiration date approaches, the server will send a notification to the terminal, warning the user, "The expiration date of the cabbage is tomorrow."

[1063] 5. Answering questions using natural language processing

[1064] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server.

[1065] The server analyzes the question, checks its database and generates an appropriate answer.

[1066] For example, if a user asks, "How many tomatoes are there?", the server generates an answer based on the information in the database: "There are two tomatoes," and notifies the user via the terminal.

[1067] 6. Implementing the Emotion Engine

[1068] The server is equipped with an emotion engine that recognizes emotions from the user's voice, text, facial expressions, etc. This allows it to analyze the user's emotions and customize corresponding notifications and responses.

[1069] For example, if the user is in a hurry, the system will calmly display a notification saying, "The cabbage's expiration date is approaching, so please use it up quickly."

[1070] 7. Customize notifications based on emotions

[1071] The server customizes the notification content and response based on the user's emotional information obtained by the emotion engine.

[1072] For example, if a user looks anxious, a notification offering assistance such as "I have two tomatoes, can I help you?" can be sent.

[1073] This system not only allows for efficient management of the contents of the refrigerator, but also takes into account the user's emotions, providing a more friendly user experience. By recognizing emotions and optimizing notifications and responses, we can expect to reduce user stress and improve the efficiency of food management.

[1074] The processing flow will be explained below.

[1075] Step 1:

[1076] The server periodically activates the camera installed inside the refrigerator to capture images of the inside of the refrigerator. For example, every time the refrigerator door is closed, the camera is automatically activated to capture a full image of the inside.

[1077] Step 2:

[1078] The server receives the captured image, which may be incomplete as it is, so it undergoes pre-processing such as noise reduction and color correction.

[1079] Step 3:

[1080] The server then inputs the preprocessed images into an object recognition algorithm, specifically using object recognition techniques such as YOLO (You Only Look Once) and Faster R-CNN to identify the food in the image.

[1081] Step 4:

[1082] The server identifies the type and quantity of food based on the results of object recognition. For example, if one cabbage and two tomatoes are identified from the image, the server extracts the information about each item as data.

[1083] Step 5:

[1084] The server stores the extracted food information in a database, compares it with existing data, and reflects any changes in stock or the addition of new food items. For example, if cabbage is already registered in the database, the quantity is updated.

[1085] Step 6:

[1086] The server periodically sends updated information about the contents of the refrigerator to the terminal, allowing the user to check the status of the refrigerator in real time via a smartphone or tablet.

[1087] Step 7:

[1088] The server monitors the expiration dates of food products and sends notifications to the device when the expiration date is approaching. For example, it sends a warning to the device saying, "The expiration date of the cabbage is tomorrow."

[1089] Step 8:

[1090] The user asks a question via voice or text about what's going on in the refrigerator, for example, "How many tomatoes are there in the refrigerator?"

[1091] Step 9:

[1092] The terminal receives the user's query and sends the request to the server.

[1093] Step 10:

[1094] The server analyzes the question using natural language processing and queries the database to generate an appropriate answer, such as "There are two tomatoes."

[1095] Step 11:

[1096] The terminal notifies the user of the generated answer, for example by displaying the message "There are two tomatoes" on the terminal.

[1097] Step 12:

[1098] The server analyzes the user's emotions using an emotion engine that recognizes emotions from the user's voice, text, facial expressions, etc. For example, if the user's tone of voice sounds impatient, the server recognizes that emotion.

[1099] Step 13:

[1100] The server customizes the notification content and response based on the user's emotional information obtained by the emotion engine. For example, if the user is feeling anxious, it generates a notification offering support, such as "I have two tomatoes. Can I help you?"

[1101] Step 14:

[1102] The device will display customized notifications to the user, for example, if the user is feeling busy, it will calmly display a notification saying, "The cabbage is nearing its expiration date, please use it up soon."

[1103] In this way, this system not only improves the efficiency of inventory management in the refrigerator, but also provides a more user-friendly interface by providing notifications and responses based on emotions.

[1104] Example 2

[1105] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1106] Conventional refrigerator management systems have the drawback of requiring a lot of time and effort to understand the status of food in the refrigerator, resulting in low user convenience. Furthermore, they lack the ability to provide appropriate notifications and responses that reflect the user's emotions and state of mind, making it difficult to improve the user experience. Furthermore, they lack the functionality to manage food expiration dates or provide specific advice, which can easily lead to food waste.

[1107] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1108] In this invention, the server includes means for capturing images of the inside of the refrigerator using a camera, means for preprocessing the images and identifying food items using an object recognition algorithm, means for saving information about the identified food items in a database, means for notifying a user terminal of the information in the database, means for analyzing questions from users using natural language processing and generating responses based on the information in the database, means for analyzing the user's voice, text, and facial expressions to recognize emotions, and means for customizing notification content and responses based on the recognized emotions. This makes food management in the refrigerator more efficient and enables appropriate responses based on the user's emotions, which is expected to improve the user experience and reduce food waste.

[1109] The "camera" is a photographing device installed to capture images inside the refrigerator.

[1110] "Preprocessing" refers to processes such as noise removal and color correction that are performed to make the acquired image easier to analyze.

[1111] An "object recognition algorithm" is a computational method for identifying food in an image and determining its type and quantity.

[1112] "Database" means a collection of digital data for storing and managing information about identified foods.

[1113] A "user terminal" is a device that a user uses to check information about the contents of the refrigerator and to communicate with the system.

[1114] "Natural language processing" is an artificial intelligence technology that analyzes questions from users and generates appropriate responses.

[1115] An "emotion engine" is a system that analyzes a user's voice, text, and facial expressions to recognize emotions.

[1116] "Customizing notification content and responses" refers to the process of adjusting the content of notifications and responses based on the user's emotions.

[1117] This invention relates to a system for efficiently managing food in a refrigerator, aiming to improve user convenience and reduce stress. The system consists of a camera installed in the refrigerator, a server, a user terminal, an object recognition algorithm, a database, natural language processing, and an emotion engine.

[1118] Hardware and Software Configuration

[1119] camera

[1120] There are multiple high-resolution cameras installed inside the refrigerator. These cameras (e.g., high-resolution cameras) have the ability to capture an image every time the refrigerator door is closed.

[1121] server

[1122] The server receives the image data sent from the camera and begins processing. First, the image is pre-processed. Specifically, camera noise is removed and color correction is performed. After this pre-processing is complete, the image is input into an object recognition algorithm (e.g., YOLO) to identify the type and quantity of food.

[1123] Database

[1124] The server stores the identified food information in a database (e.g., MySQL). When a new food is added, the information is updated in the database and existing food information is updated.

[1125] User terminal

[1126] The user terminal is a device such as a smartphone or tablet that allows the user to check the information inside the refrigerator. The latest food information is sent from the server to the terminal, allowing the user to check the status of the refrigerator in real time. For example, information such as "There is one cabbage and two tomatoes in the refrigerator" is displayed.

[1127] Natural Language Processing

[1128] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server. The server uses natural language processing (e.g., BERT) to analyze the question and compares it with information in the database to generate an appropriate answer. For example, the question "How many tomatoes are there?" will generate an answer such as "There are two tomatoes."

[1129] Emotion Engine

[1130] The server is equipped with an emotion engine (e.g., Affectiva SDK) that analyzes the user's voice, text, and facial expressions to recognize their emotions. This allows it to analyze the user's emotions and customize corresponding notifications and responses. For example, if the user is feeling anxious, the server may notify them that "The expiration date of the cabbage is approaching, so please use it up quickly."

[1131] Customize notifications based on emotions

[1132] The server customizes the notification content and response based on the user's emotional information obtained by the emotion engine. For example, if the user looks anxious, a notification offering support such as "There are two tomatoes. Can I help you?" will be sent.

[1133] Examples of concrete examples and prompts

[1134] For example, whenever a user opens or closes the refrigerator, the camera captures an image and the server updates the food information based on this. For example, when a user asks, "How many tomatoes are there?", the server responds via the device with, "There are two tomatoes."

[1135] Example prompt sentence:

[1136] "Capture an image of the inside of your refrigerator, identify the food in the crisper, and store it in a database."

[1137] This system allows users to manage the food in their refrigerator efficiently and without waste, and also provides a pleasant user experience by receiving appropriate notifications and responses based on their emotions.

[1138] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1139] Step 1:

[1140] Camera installation and image acquisition

[1141] The server installs multiple high-resolution cameras inside the refrigerator. Each time the refrigerator door is closed, these cameras are activated and capture images of the inside of the refrigerator. Specifically, the cameras are automatically activated when the refrigerator door sensor detects that the door is closing, and images of multiple locations inside the refrigerator are taken.

[1142] Input: Actual view inside the refrigerator, door sensor signal

[1143] Output: High-resolution image of the inside of the refrigerator

[1144] Step 2:

[1145] Image preprocessing

[1146] The server receives the captured images and performs pre-processing, such as noise reduction and color correction to improve image quality, resulting in image data ready to be input into object recognition algorithms.

[1147] Input: High-resolution image

[1148] Output: Preprocessed image

[1149] Step 3:

[1150] object recognition

[1151] The server inputs the preprocessed image into an object recognition algorithm (e.g., YOLO). The algorithm identifies the type and quantity of food and identifies the results. Specifically, the YOLO algorithm analyzes each pixel in the image and identifies specific objects (e.g., cabbage, tomato).

[1152] Input: Preprocessed image

[1153] Output: Identified food types and quantities

[1154] Step 4:

[1155] Data storage

[1156] The server stores the identified food information in a database (e.g., MySQL). When a new food is added, the information is saved in the database and existing food information is updated as necessary. Specifically, the new and existing data are compared and the differences are updated.

[1157] Input: Type and quantity of food identified

[1158] Output: Updated database contents

[1159] Step 5:

[1160] Notification to user device

[1161] The server sends the latest food information to the user's device (smartphone or tablet), allowing the user to check the status of the refrigerator in real time. For example, a notification might say, "There is one cabbage and two tomatoes in the refrigerator."

[1162] Input: Updated database contents

[1163] Output: Food information displayed on the user's device

[1164] Step 6:

[1165] Answering questions using natural language processing

[1166] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server. The server uses natural language processing (e.g., BERT) to analyze the question, compares it with information in the database, generates an appropriate answer, and notifies the user via the device. For example, the question "How many tomatoes are there?" is answered with "There are two tomatoes."

[1167] Input: User question (voice or text)

[1168] Output: Response to user terminal

[1169] Step 7:

[1170] Implementing the Emotion Engine

[1171] The server is equipped with an emotion engine (e.g., Affectiva SDK) that analyzes the user's voice, text, and facial expressions to recognize their emotions. This allows it to analyze the user's emotions and generate appropriate notifications and responses. For example, if the user is feeling impatient, the server will notify them by saying, "The cabbage's expiration date is approaching, so please use it up quickly."

[1172] Input: User voice, text, facial expressions

[1173] Output: Notification based on user sentiment

[1174] Step 8:

[1175] Customize notifications based on emotions

[1176] The server customizes the notification content and response based on the user's emotional information obtained from the emotion engine. For example, if the user looks anxious, the server sends a notification offering support such as, "There are two tomatoes. Can I help you?"

[1177] Input: User's emotional information

[1178] Output: Customized notification content

[1179] (Application example 2)

[1180] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1181] Conventional refrigerator management systems and store management systems only had the function of managing inventory information and were unable to respond or notify users based on their emotions. Furthermore, when it came to in-store product management, it was difficult to check inventory in real time, making it impossible to respond quickly to customer needs. This resulted in a poor user experience and a decline in the efficiency of inventory management and customer service.

[1182] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring images of the inside of the refrigerator and the inside of the store, means for processing the images and identifying food and products, and means for analyzing customer emotions and customizing notifications and responses. This enables appropriate responses according to user emotions and real-time management of inventory in the refrigerator and the store.

[1183] A "server" is a device that provides computing resources and storage, analyzes and stores various types of data, and manages communications.

[1184] The "means for acquiring images inside the refrigerator" is a technology for capturing images using a camera installed inside the refrigerator and acquiring them as data.

[1185] The "means for processing the image and identifying the food" refers to a technology for preprocessing the acquired image and identifying the food using an object recognition algorithm.

[1186] The "means for storing information on the identified food in a database" refers to a technique for recording data on the type and quantity of the identified food in a database.

[1187] "Means for notifying the user terminal of the information in the database" refers to technology that sends food information and update information stored in the database to the user's terminal such as a smartphone or tablet.

[1188] "Means for analyzing questions from users using natural language processing and generating responses based on information in the database" refers to technology that analyzes voice or text questions uttered by users and generates appropriate answers based on information in the database.

[1189] The "means for acquiring images within the store" refers to a technology for capturing images using cameras installed within the store and acquiring them as data.

[1190] The "means for processing the image and identifying the product" refers to a technology for preprocessing images acquired in a store and identifying the product using an object recognition algorithm.

[1191] "Means for analyzing customer emotions and customizing notifications and responses" refers to technology that reads emotions from a customer's facial expressions, voice, text, etc., and uses that information to customize the content of notifications and responses.

[1192] "Means for periodically reacquiring images of the inside of the refrigerator and the inside of the store and updating them to the latest state" refers to a technology that periodically operates the camera to acquire the latest images and keep the contents of the database up to date.

[1193] "Means for sending notifications to a user terminal based on the expiration dates of the food and products" refers to a technology that sends warnings and notifications to a user terminal about products that are approaching their expiration date based on expiration date information of stored food and products.

[1194] The embodiments for carrying out the present invention are as follows.

[1195] 1. System Overview

[1196] This system improves the efficiency of inventory management in refrigerators and physical stores, and provides responses based on the user's emotions. The system mainly consists of a server, a terminal, a camera, an emotion analysis engine, an object recognition algorithm, and a database.

[1197] 2. Hardware and Software

[1198] Cameras: Installed in refrigerators and in stores. Example: Logitech C920 webcam

[1199] Server: Responsible for data processing and storage. Example: AWS EC2 instance

[1200] Terminal: The device on which the user receives information. Examples: smartphone, tablet

[1201] software:

[1202] OpenCV: Image capture and preprocessing library

[1203] Object Recognition Algorithm: Identifying food and goods using YOLOv3 and ResNet

[1204] Database: Manage inventory data with MySQL or Firebase

[1205] Sentiment analysis engine: Uses Microsoft Azure's Emotion API

[1206] Natural Language Processing: Question Answering with GPT-3

[1207] 3. System Operation

[1208] 1. Image Acquisition

[1209] The server periodically activates the cameras installed in the refrigerator and in the store to capture the latest images. For example, the server can activate the camera and capture an image every time the refrigerator door is closed.

[1210] 2. Image Preprocessing and Analysis

[1211] The server preprocesses the acquired images using OpenCV. Specifically, it performs noise removal and color correction. The preprocessed images are input into an object recognition algorithm, and the server identifies food and products. For example, it identifies cabbages and tomatoes from an image of a vegetable drawer and determines their quantities.

[1212] 3. Data storage

[1213] The server stores the identified food and product information in a database. If there are any changes compared to existing data, the information is updated. For example, if a new item, "1 cabbage," is added to the database, the information is updated.

[1214] 4. Notice to Users

[1215] The server sends the latest information about what's in the refrigerator and in the store to the terminal. Users can check inventory status in real time via their smartphone or tablet. For example, the terminal will display information like, "There is one cabbage and two tomatoes in the refrigerator." In addition, it will send a warning if the expiration date is approaching.

[1216] 5. Answering questions using natural language processing

[1217] When a user asks a question about the inventory in the refrigerator or store by voice or text, the device sends the question to the server. The server uses GPT-3 to analyze the question, compares it with the database, and generates an appropriate answer. For example, if a user asks, "How many tomatoes are there?", the device will respond, "There are two tomatoes."

[1218] 6. Implementing the Emotion Engine

[1219] The server is equipped with an emotion analysis engine that recognizes emotions from the user's voice, text, facial expressions, etc. This allows it to analyze the user's emotions and customize corresponding notifications and responses.

[1220] 7. Customize notifications based on emotions

[1221] The server customizes notifications and responses based on the user's emotional information obtained from the emotion analysis engine. For example, if the user is in a hurry, the server will calmly display a notification such as, "The expiration date of the cabbage is approaching. Please use it up quickly." If the user is anxious, the server will send a notification offering support, such as, "You have two tomatoes. Is there anything I can help you with?"

[1222] 8. Examples of prompt sentences

[1223] "Tell me the stock situation"

[1224] "Do you have product A?"

[1225] Specific use cases

[1226] When a user asks "Do you have any cabbages?" on their smartphone, the server checks the database and responds, "We have one cabbage." If the user looks anxious, the server responds with a gentler response, such as, "We have two tomatoes. May I help you?"

[1227] Such a system enables flexible responses that adapt to the user's emotions, and allows for appropriate management of inventory status in refrigerators and stores.

[1228] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1229] Step 1:

[1230] The server activates the cameras installed in the refrigerator and in the store to capture images. Based on the request, the server obtains image data from the cameras and preprocesses it. Preprocessing includes noise reduction and color correction. The input is the raw image captured by the camera, and the output is the preprocessed image.

[1231] Step 2:

[1232] The server inputs the preprocessed images into an object recognition algorithm (e.g., YOLOv3 or ResNet) to identify food and store items. The input is the preprocessed image, and the output is the type and quantity of identified food or items. Specifically, the algorithm identifies objects in the image and returns their labels and locations.

[1233] Step 3:

[1234] The server stores the identified food and product information (type, quantity) in a database. It compares the data with existing data and updates it if there are any changes. The input is the identified food or product data, and the output is the updated database entry. For example, information such as "two tomatoes were added" is recorded in the database.

[1235] Step 4:

[1236] The server sends the latest information about what's in the refrigerator and the store to the user terminal. Here, it retrieves the latest information from the database and generates a message to send to the user terminal. The input is inventory information retrieved from the database, and the output is a notification message to be sent to the user terminal. For example, the message might say, "There is one cabbage and two tomatoes in the refrigerator."

[1237] Step 5:

[1238] The user asks a question about the inventory in the refrigerator or in the store by voice or text. The terminal sends the question to the server. The input to the terminal is the user's question, and the output is the question data to the server. For example, the terminal sends a user's question such as "How many tomatoes are there?" to the server.

[1239] Step 6:

[1240] The server uses GPT-3 to analyze the question, compare it with the database, and generate an appropriate answer. The input is the user's question data received from the device, and the output is the generated answer. Specifically, it uses natural language processing to understand the intent of the question and constructs an answer based on related information in the database.

[1241] Step 7:

[1242] The server uses an emotion analysis engine to analyze the user's emotions and recognizes emotions from voice, text, and facial expressions. The input is the user's voice data, text data, and image data, and the output is analyzed emotional information. Specifically, the emotion analysis engine analyzes this data and determines whether the user is anxious or impatient.

[1243] Step 8:

[1244] The server customizes notifications and responses based on the user's emotional information obtained by the emotion analysis engine. The input is the user's emotional information and the generated response information, and the output is a customized notification message. For example, if the user is in a hurry, the server sends a notification such as, "Remain calm. The expiration date of the cabbage is approaching. Please use it up quickly."

[1245] Step 9:

[1246] The server sends a customized notification to the user terminal. The input is a customized notification message, and the output is a notification displayed on the user terminal, allowing the user to receive accurate information.

[1247] Through the above steps, a system is realized in which the server, terminals, and users work together to manage inventory and handle customer service.

[1248] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1249] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1250] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1251] [Fourth embodiment]

[1252] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1253] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1254] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1255] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1256] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1257] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1258] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1259] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1260] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1261] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1262] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1263] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1264] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1265] This system manages the contents of a refrigerator in real time by capturing images of the inside of the refrigerator, processing the images to identify food items, storing the information in a database, and notifying the user as needed. It also has the ability to generate answers to user questions using natural language processing.

[1266] Specific Embodiments of the System

[1267] 1. Installing and starting the camera

[1268] The server installs multiple cameras inside the refrigerator and periodically activates these cameras to capture images inside the refrigerator.

[1269] For example, a dedicated camera is installed in the vegetable compartment, and every time the refrigerator door is closed, the camera activates and captures an image of the entire vegetable compartment. The server receives this image in real time.

[1270] 2. Image Preprocessing and Analysis

[1271] The server pre-processes the received images, which includes noise reduction, color correction, etc.

[1272] The pre-processed images are then fed into an object recognition algorithm to identify the food.

[1273] For example, cabbage and tomatoes are identified from the captured image, and the server identifies the type and quantity of these foods.

[1274] 3. Data storage

[1275] The server stores the identified food information in a database, compares it with existing data, and updates it if there are any changes.

[1276] For example, if a new cabbage is added, it will be saved in the database as "1 cabbage." If a cabbage is already registered, the quantity will be updated.

[1277] 4. Notice to Users

[1278] The server sends the latest information about the contents of the refrigerator to the terminal, allowing the user to check the status of the refrigerator in real time via a smartphone or tablet.

[1279] For example, the terminal will display, "There is one cabbage and two tomatoes in the refrigerator." Furthermore, if the expiration date approaches, the server will send a notification to the terminal, warning the user, "The expiration date of the cabbage is tomorrow."

[1280] 5. Answering questions using natural language processing

[1281] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server.

[1282] The server analyzes the question, checks its database and generates an appropriate answer.

[1283] For example, if a user asks, "How many tomatoes are there?", the server generates the answer "There are two tomatoes" based on the information in the database and notifies the user via the terminal.

[1284] This system allows for efficient management of refrigerator contents and reduces food waste. Furthermore, users can check the status of their refrigerator anytime, anywhere, making it easier to plan their shopping and cooking. It also contributes to reducing food waste by managing expiration dates.

[1285] The processing flow will be explained below.

[1286] Step 1:

[1287] The server periodically activates a camera installed inside the refrigerator to capture images of the inside of the refrigerator. For example, the camera is set to activate every time the refrigerator door is closed.

[1288] Step 2:

[1289] The server receives the captured image. Since the received image may be imperfect as it is, it performs preprocessing such as noise removal and color correction. For example, if the image contains noise, a filter is applied to remove the noise.

[1290] Step 3:

[1291] The server then inputs the pre-processed images into an object recognition algorithm, which uses advanced object recognition techniques such as YOLO (You Only Look Once) and Faster R-CNN to identify the food items in the refrigerator.

[1292] Step 4:

[1293] The server identifies the type and quantity of food based on the results of object recognition. For example, if one cabbage and two tomatoes are identified from the image, the server extracts the information about each item as data.

[1294] Step 5:

[1295] The server stores the extracted food information in a database, compares it with existing data, and reflects any changes in stock or the addition of new food items. For example, if cabbage is already registered in the database, the quantity is updated.

[1296] Step 6:

[1297] The server periodically sends updated information about the contents of the refrigerator to the device, which then displays this information so the user can check the status of the refrigerator. For example, a notification might appear on the device saying, "There is one cabbage and two tomatoes in the refrigerator."

[1298] Step 7:

[1299] The server monitors the expiration dates of food products and sends notifications to the device when the expiration date is approaching. For example, it sends a warning to the device saying, "The expiration date of the cabbage is tomorrow."

[1300] Step 8:

[1301] The user asks a question about the status of the refrigerator via voice or text, for example, "How many tomatoes are there in the refrigerator?"

[1302] Step 9:

[1303] The terminal receives the query and sends the request to the server.

[1304] Step 10:

[1305] The server analyzes the question using natural language processing and queries the database to generate an appropriate answer, such as "There are two tomatoes."

[1306] Step 11:

[1307] The terminal will notify the user of the generated answer, for example, "There are two tomatoes" will be displayed on the terminal.

[1308] Example 1

[1309] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1310] In conventional refrigerator management systems, managing food inventory and expiration dates is often done manually, requiring users to frequently check the contents of the refrigerator. There is also a high risk of incorrect information being entered, which can lead to discrepancies between actual inventory and records. Furthermore, there is an insufficient system for users to quickly respond to questions about the food in the refrigerator.

[1311] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1312] In this invention, the server includes means for acquiring images of the inside of the refrigerator, means for preprocessing the images, means for analyzing the preprocessed images to identify food items, means for saving information about the identified food items in a database, means for notifying a user terminal of information in the database, and means for analyzing questions from users using natural language processing and generating responses based on information in the database. This allows for accurate real-time management of food items in the refrigerator and enables users to easily check the inventory status and expiration dates in the refrigerator. Furthermore, a question-answering function using natural language processing improves user convenience.

[1313] "Means for acquiring images inside the refrigerator" refers to a function that periodically takes images of the food and containers inside the refrigerator using a photographic device such as a camera installed inside the refrigerator and acquires them as digital data.

[1314] "Preprocessing" refers to a series of processes that involve processing the acquired image data, such as noise removal and color correction, to improve the quality of the image.

[1315] "Analysis" or "image analysis" refers to the use of algorithms and artificial intelligence to analyze pre-processed image data and recognize and identify objects (in this case, food) within the image.

[1316] "Means for identifying food" refers to the function of using image analysis to identify individual food items in the refrigerator and clarify their type and quantity.

[1317] "Means for storing in a database" refers to the technology for recording and storing information on identified food (such as type, quantity, location, etc.) in a database as digital data.

[1318] "Means for notifying user devices" refers to the function of analyzing information stored in the database and sending necessary notifications in real time to the user's electronic device (such as a smartphone or tablet).

[1319] "Natural language processing" refers to technology that understands, analyzes, and responds appropriately to natural human language. In this context, it includes the ability for a system to generate appropriate answers to user inquiries.

[1320] "Means for generating a response" refers to technology that uses natural language processing to understand a user's question, constructs an appropriate response based on information in a database, and notifies the user's device of this response.

[1321] "Means for sending notifications based on expiration dates" refers to a function that monitors expiration date information for each food item stored in the database and automatically sends notifications to users when the expiration date approaches.

[1322] The present invention relates to a system for managing food in a refrigerator in real time. The program processing of this system is specifically explained below. Specific examples of the system's hardware and software include cameras, servers, user terminals, and artificial intelligence models.

[1323] 1. Installing and starting the camera

[1324] The server installs high-resolution cameras at key locations inside the refrigerator. Each time the refrigerator door is closed, the cameras are automatically activated and capture images of the interior. A typical example of such cameras is a general-purpose digital camera.

[1325] Example: When the refrigerator door is closed, the server sets the camera to immediately take pictures of the vegetable compartment and main storage area, allowing the overall status of the refrigerator to be grasped in real time.

[1326] 2. Image Preprocessing

[1327] The server preprocesses the acquired image data using libraries such as OpenCV. This preprocessing includes noise reduction and color correction, improving the quality of the images and the accuracy of subsequent analysis.

[1328] Example: The server applies noise filtering and adjusts the brightness and contrast of the image to clarify fuzzy areas.

[1329] 3. Image Analysis

[1330] The pre-processed image data is then fed into an object recognition algorithm such as YOLO (You Only Look Once) to identify the food items. The server then analyzes the data and identifies the type and quantity of food items in the refrigerator.

[1331] Example: For example, the server uses an object recognition algorithm to identify one cabbage and two tomatoes in an image.

[1332] 4. Data storage

[1333] The server stores the information of the identified food in a database using MySQL or PostgreSQL, comparing it with existing data and updating it immediately if there are any changes.

[1334] Example: When a new cabbage is added, the server stores it in the database as "1 cabbage." If a cabbage is already registered, the server updates the quantity.

[1335] 5. Notice to Users

[1336] The server sends the latest information about the contents of the refrigerator to the user's device. Users can check the status of the refrigerator in real time via their smartphone or tablet. The server also notifies users when the expiration date of each food item is approaching.

[1337] Example: The user device displays "There is one cabbage and two tomatoes in the refrigerator." If the expiration date is one day away, the device displays a notification saying "The cabbage's expiration date is tomorrow."

[1338] 6. Answering questions using natural language processing

[1339] When a user asks a question about the status of the refrigerator by voice or text, the user device sends the question to the server, which analyzes the question, checks the database, and generates an appropriate answer.

[1340] Example: When a user asks, "How many tomatoes are there?", the server generates the answer "There are two tomatoes" based on the information in the database and notifies the user via their device.

[1341] Prompt Sentence Examples

[1342] 1. "Please tell me how many tomatoes are in the refrigerator."

[1343] 2. "When is the expiration date for cabbage?"

[1344] This system allows users to accurately manage the contents of their refrigerator in real time, greatly improving user convenience. It allows users to efficiently grasp food inventory and expiration dates, reducing food waste and making shopping and cooking planning easier. Furthermore, a question-and-answer function using natural language processing allows users to quickly and accurately obtain information about what's inside their refrigerator.

[1345] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1346] Step 1:

[1347] Camera installation and startup

[1348] The server installs a high-resolution camera inside the refrigerator, which automatically activates every time the door is closed.

[1349] Input: Refrigerator door closing trigger signal

[1350] Data processing and calculation: Camera activation and image capture

[1351] Output: High-resolution image data of the inside of the refrigerator

[1352] Specific operation: When the refrigerator door is closed, the camera automatically starts up, takes a picture of the inside of the refrigerator, and sends it to the server.

[1353] Step 2:

[1354] Image preprocessing

[1355] The server performs pre-processing such as noise removal and color correction on the received image data.

[1356] Input: High-resolution image data of the inside of the refrigerator

[1357] Data processing and calculation: noise filtering, color correction

[1358] Output: Preprocessed image data

[1359] What it does: The server uses libraries such as OpenCV to remove noise from the image and balance the colors to produce a clear image.

[1360] Step 3:

[1361] Image analysis

[1362] The server inputs the preprocessed image data into an object recognition algorithm (e.g., YOLO) to identify the food item.

[1363] Input: Preprocessed image data

[1364] Data processing and calculation: Analysis using object recognition algorithms

[1365] Output: Identified food type and quantity data

[1366] How it works: The analysis algorithm scans the image and identifies foods such as cabbage and tomatoes, and then identifies the type and quantity of each food item and stores it as data.

[1367] Step 4:

[1368] Data storage

[1369] The server stores the identified food information in a database.

[1370] Input: Identified food type and quantity data

[1371] Data processing and calculation: writing information to the database, updating existing data

[1372] Output: A database containing the latest food information

[1373] Specific operation: The server uses MySQL or PostgreSQL to record food information in a database and compare it with existing data to update it.

[1374] Step 5:

[1375] User Notification

[1376] The server notifies the user terminal of the latest refrigerator information.

[1377] Input: A database containing the latest food information

[1378] Data processing and calculation: Generates notification messages and sends them to user terminals

[1379] Output: Latest refrigerator information displayed on the user's device

[1380] Specific operation: The server retrieves the latest information from the database and displays the status of the refrigerator on the user's device. For example, it displays "There is one cabbage and two tomatoes in the refrigerator."

[1381] Step 6:

[1382] Answering questions using natural language processing

[1383] The user asks questions about the status of the refrigerator by voice or text, and the device sends the questions to the server, which analyzes the questions and generates appropriate answers.

[1384] Input: User question (voice or text)

[1385] Data processing and calculation: Question analysis using natural language processing, database matching, and response generation

[1386] Output: Response message to be sent to the user

[1387] Specific operation: When a user asks "How many tomatoes are there?", the server analyzes the question, retrieves information from the database, and sends the answer "There are two tomatoes" to the user's terminal.

[1388] (Application example 1)

[1389] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1390] In modern households, managing food in the refrigerator is extremely time-consuming, and food waste due to expired food and unnecessary duplicate purchases is a major problem. It is also difficult to keep track of the refrigerator's contents while out or cooking, making it difficult to efficiently plan shopping and meals. Furthermore, there is a lack of systems that can quickly and accurately respond to users' questions about the refrigerator's contents, and there is a need to solve this problem.

[1391] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1392] In this invention, the server includes means for acquiring images of the inside of the refrigerator, means for processing the images and identifying foods, means for storing information about the identified foods in a database, means for notifying a user terminal of the information in the database, means for analyzing questions from users using natural language processing and generating responses based on the information in the database, means for users to check the information in the database in real time through an application installed on a smart device, and means for notifying a user terminal of foods that are approaching their expiration date. This allows for efficient management of foods in the refrigerator, reduces food waste, and enables users to keep track of the status of their refrigerator anytime, anywhere.

[1393] The "means for acquiring images inside the refrigerator" is a mechanism for capturing images of the inside of the refrigerator using a camera installed inside the refrigerator and acquiring the image data.

[1394] The "means for processing the image and identifying the food" refers to an algorithm or program that analyzes the acquired image data and identifies the various foods present in the refrigerator.

[1395] The "means for storing information on the identified food in a database" is a mechanism for recording and managing data such as the type and quantity of food identified through analysis in a database.

[1396] The "means for notifying the user terminal of the information in the database" is a mechanism for transmitting and displaying the food information stored in the database to a terminal such as a smartphone or tablet owned by the user.

[1397] "Means for analyzing questions from users using natural language processing and generating responses based on information in the database" refers to a mechanism that uses natural language processing technology to analyze voice or text questions from users, compares them with information in the database, and generates appropriate responses.

[1398] "A means by which users can check the information in the database in real time through an application installed on a smart device" refers to a mechanism that allows users to view the latest food information in the database in real time using a dedicated application installed on a smart device such as a smartphone.

[1399] The "means for notifying a user terminal of food products approaching their expiration date" is a mechanism that sends a notification to a user terminal about food products that are approaching their expiration date based on food information in the database.

[1400] The present invention provides a system that efficiently manages food in a refrigerator, reduces food waste, and allows a user to grasp the status of the refrigerator anytime, anywhere.

[1401] A system for implementing the present invention comprises the following components:

[1402] 1. Image acquisition method:

[1403] There are multiple cameras installed inside the refrigerator, and these cameras periodically capture images of the inside of the refrigerator. For example, every time the refrigerator door is closed, a camera is automatically activated and takes a picture of the inside of the refrigerator.

[1404] 2. Imaging and Food Identification Methods:

[1405] The captured images are first preprocessed with noise removal and color correction, then an object recognition algorithm (e.g., a deep learning model using Keras) is used to identify food items. For example, cabbage and tomatoes are identified, and the quantity of each is also determined.

[1406] 3. Database storage method:

[1407] Information on the identified food (type, quantity, etc.) is stored in a database such as SQLite. If an update is required, this information is compared with existing data and updated to the latest version.

[1408] 4. User terminal notification means:

[1409] The identified food information is sent in real time to the user's device, such as a smartphone or tablet. The user can check the status of the refrigerator through a dedicated application. For example, information such as "There is one cabbage and two tomatoes in the refrigerator" is displayed.

[1410] 5. Natural Language Processing Tools:

[1411] Voice and text questions from users are sent to the server via their device. The questions are analyzed using natural language processing technology and compared with information in a database to generate an appropriate answer. For example, in response to the question "How many tomatoes are there?", an answer such as "There are two tomatoes" is generated and notified to the user's device.

[1412] 6. Best before date notification method:

[1413] When the expiration date of an identified food item is approaching, the server sends a notification to the user's terminal. This notification allows the user to take action early on regarding the food item that is approaching its expiration date. For example, the user may receive a notification saying, "The expiration date of the cabbage is tomorrow."

[1414] This allows users to effectively manage the food in their refrigerator and reduce food waste. It also allows users to check the status of their refrigerator in real time while they are out or shopping, helping them avoid unnecessary purchases. Furthermore, users can instantly ask questions about the status of their refrigerator and get answers, enabling more efficient meal preparation.

[1415] Example 1:

[1416] When you open the smartphone app and tap the "Show latest refrigerator status" button, the app requests the latest food information from the server and displays it on the screen. You can instantly see information such as "There is one cabbage and two tomatoes in the refrigerator."

[1417] Example 2:

[1418] When the expiration date approaches, a notification will be sent to the smartphone, with the message "The cabbage's expiration date is tomorrow," allowing the user to use the cabbage sooner.

[1419] Example prompt for a generative AI model:

[1420] A user asks: "How many tomatoes are in the fridge right now?"

[1421] Prompt for the AI ​​model: "The user wants to know how many tomatoes are in the refrigerator. Please check your database and generate the correct answer."

[1422] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1423] Step 1:

[1424] Capture images of the inside of the refrigerator

[1425] The server activates a camera installed inside the refrigerator to capture an image of the inside of the refrigerator.

[1426] Input: Latest status of the refrigerator

[1427] Output: Image data of the inside of the refrigerator (e.g., JPEG image file)

[1428] Step 2:

[1429] Image preprocessing

[1430] The server performs preprocessing such as noise removal and color correction on the acquired image data.

[1431] Input: Raw captured image data

[1432] Output: Pre-processed image data (e.g., noise-removed and color-optimized images)

[1433] Step 3:

[1434] Food Identification

[1435] The server inputs the preprocessed image data into a food identification algorithm to identify the food items, for example, using a deep learning model (e.g., using Keras) to identify each food item in the refrigerator.

[1436] Input: Preprocessed image data

[1437] Output: A list of identified foods (e.g., {'cabbage': 1, 'tomato': 2})

[1438] Step 4:

[1439] Saving to a database

[1440] The server stores the identified food information (type, quantity) in a database such as SQLite, compares it with existing data, and updates it as necessary.

[1441] Input: List of identified foods

[1442] Output: Latest food information registered in the database

[1443] Step 5:

[1444] Notification to user device

[1445] The server sends the latest food information stored in the database to the user's smartphone or tablet, where the user can check this information in real time through a dedicated application.

[1446] Input: Latest food information from the database

[1447] Output: Food information displayed on the user's device (application)

[1448] Step 6:

[1449] Answering questions using natural language processing

[1450] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server. The server uses natural language processing technology to analyze the question, compares it with information in a database, generates an appropriate answer, and sends it to the device. For example, if a user asks, "How many tomatoes are there?", the server generates the answer, "There are two tomatoes."

[1451] Input: User question (voice or text)

[1452] Output: Answer based on the database (e.g., there are two tomatoes)

[1453] Step 7:

[1454] Best before date notification

[1455] The server uses the food information in the database to identify foods that are approaching their expiration date and sends a notification to the user's smartphone, such as "The expiration date for the cabbage is tomorrow."

[1456] Input: Food information and its expiration date in the database

[1457] Output: Expiration warning notification (e.g., the expiration date for cabbage is tomorrow)

[1458] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1459] This invention combines an emotion engine with a system for managing the contents of a refrigerator in real time. It captures images of the inside of the refrigerator, processes them to identify food items, stores the information in a database, and notifies the user as needed. It also has the ability to generate answers to user questions using natural language processing, and to analyze user emotions to customize notifications and responses.

[1460] Specific Embodiments of the System

[1461] 1. Installing and starting the camera

[1462] The server installs multiple cameras inside the refrigerator and periodically activates these cameras to capture images inside the refrigerator.

[1463] For example, a dedicated camera is installed in the vegetable compartment, and every time the refrigerator door is closed, the camera activates and captures an image of the entire vegetable compartment. The server receives this image in real time.

[1464] 2. Image Preprocessing and Analysis

[1465] The server pre-processes the received images, which includes noise reduction, color correction, etc.

[1466] The pre-processed images are then fed into an object recognition algorithm to identify the food.

[1467] For example, cabbage and tomatoes are identified from the captured image, and the server identifies the type and quantity of these foods.

[1468] 3. Data storage

[1469] The server stores the identified food information in a database, compares it with existing data, and updates it if there are any changes.

[1470] For example, if a new cabbage is added, it will be saved in the database as "1 cabbage." If a cabbage is already registered, the quantity will be updated.

[1471] 4. Notice to Users

[1472] The server sends the latest information about the contents of the refrigerator to the terminal, allowing the user to check the status of the refrigerator in real time via a smartphone or tablet.

[1473] For example, the terminal will display, "There is one cabbage and two tomatoes in the refrigerator." Furthermore, if the expiration date approaches, the server will send a notification to the terminal, warning the user, "The expiration date of the cabbage is tomorrow."

[1474] 5. Answering questions using natural language processing

[1475] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server.

[1476] The server analyzes the question, checks its database and generates an appropriate answer.

[1477] For example, if a user asks, "How many tomatoes are there?", the server generates an answer based on the information in the database: "There are two tomatoes," and notifies the user via the terminal.

[1478] 6. Implementing the Emotion Engine

[1479] The server is equipped with an emotion engine that recognizes emotions from the user's voice, text, facial expressions, etc. This allows it to analyze the user's emotions and customize corresponding notifications and responses.

[1480] For example, if the user is in a hurry, the system will calmly display a notification saying, "The cabbage's expiration date is approaching, so please use it up quickly."

[1481] 7. Customize notifications based on emotions

[1482] The server customizes the notification content and response based on the user's emotional information obtained by the emotion engine.

[1483] For example, if a user looks anxious, a notification offering assistance such as "I have two tomatoes, can I help you?" can be sent.

[1484] This system not only allows for efficient management of the contents of the refrigerator, but also takes into account the user's emotions, providing a more friendly user experience. By recognizing emotions and optimizing notifications and responses, we can expect to reduce user stress and improve the efficiency of food management.

[1485] The processing flow will be explained below.

[1486] Step 1:

[1487] The server periodically activates the camera installed inside the refrigerator to capture images of the inside of the refrigerator. For example, every time the refrigerator door is closed, the camera is automatically activated to capture a full image of the inside.

[1488] Step 2:

[1489] The server receives the captured image, which may be incomplete as it is, so it undergoes pre-processing such as noise reduction and color correction.

[1490] Step 3:

[1491] The server then inputs the preprocessed images into an object recognition algorithm, specifically using object recognition techniques such as YOLO (You Only Look Once) and Faster R-CNN to identify the food in the image.

[1492] Step 4:

[1493] The server identifies the type and quantity of food based on the results of object recognition. For example, if one cabbage and two tomatoes are identified from the image, the server extracts the information about each item as data.

[1494] Step 5:

[1495] The server stores the extracted food information in a database, compares it with existing data, and reflects any changes in stock or the addition of new food items. For example, if cabbage is already registered in the database, the quantity is updated.

[1496] Step 6:

[1497] The server periodically sends updated information about the contents of the refrigerator to the terminal, allowing the user to check the status of the refrigerator in real time via a smartphone or tablet.

[1498] Step 7:

[1499] The server monitors the expiration dates of food products and sends notifications to the device when the expiration date is approaching. For example, it sends a warning to the device saying, "The expiration date of the cabbage is tomorrow."

[1500] Step 8:

[1501] The user asks a question via voice or text about what's going on in the refrigerator, for example, "How many tomatoes are there in the refrigerator?"

[1502] Step 9:

[1503] The terminal receives the user's query and sends the request to the server.

[1504] Step 10:

[1505] The server analyzes the question using natural language processing and queries the database to generate an appropriate answer, such as "There are two tomatoes."

[1506] Step 11:

[1507] The terminal notifies the user of the generated answer, for example by displaying the message "There are two tomatoes" on the terminal.

[1508] Step 12:

[1509] The server analyzes the user's emotions using an emotion engine that recognizes emotions from the user's voice, text, facial expressions, etc. For example, if the user's tone of voice sounds impatient, the server recognizes that emotion.

[1510] Step 13:

[1511] The server customizes the notification content and response based on the user's emotional information obtained by the emotion engine. For example, if the user is feeling anxious, it generates a notification offering support, such as "I have two tomatoes. Can I help you?"

[1512] Step 14:

[1513] The device will display customized notifications to the user, for example, if the user is feeling busy, it will calmly display a notification saying, "The cabbage is nearing its expiration date, please use it up soon."

[1514] In this way, this system not only improves the efficiency of inventory management in the refrigerator, but also provides a more user-friendly interface by providing notifications and responses based on emotions.

[1515] Example 2

[1516] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1517] Conventional refrigerator management systems have the drawback of requiring a lot of time and effort to understand the status of food in the refrigerator, resulting in low user convenience. Furthermore, they lack the ability to provide appropriate notifications and responses that reflect the user's emotions and state of mind, making it difficult to improve the user experience. Furthermore, they lack the functionality to manage food expiration dates or provide specific advice, which can easily lead to food waste.

[1518] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1519] In this invention, the server includes means for capturing images of the inside of the refrigerator using a camera, means for preprocessing the images and identifying food items using an object recognition algorithm, means for saving information about the identified food items in a database, means for notifying a user terminal of the information in the database, means for analyzing questions from users using natural language processing and generating responses based on the information in the database, means for analyzing the user's voice, text, and facial expressions to recognize emotions, and means for customizing notification content and responses based on the recognized emotions. This makes food management in the refrigerator more efficient and enables appropriate responses based on the user's emotions, which is expected to improve the user experience and reduce food waste.

[1520] The "camera" is a photographing device installed to capture images inside the refrigerator.

[1521] "Preprocessing" refers to processes such as noise removal and color correction that are performed to make the acquired image easier to analyze.

[1522] An "object recognition algorithm" is a computational method for identifying food in an image and determining its type and quantity.

[1523] "Database" means a collection of digital data for storing and managing information about identified foods.

[1524] A "user terminal" is a device that a user uses to check information about the contents of the refrigerator and to communicate with the system.

[1525] "Natural language processing" is an artificial intelligence technology that analyzes questions from users and generates appropriate responses.

[1526] An "emotion engine" is a system that analyzes a user's voice, text, and facial expressions to recognize emotions.

[1527] "Customizing notification content and responses" refers to the process of adjusting the content of notifications and responses based on the user's emotions.

[1528] This invention relates to a system for efficiently managing food in a refrigerator, aiming to improve user convenience and reduce stress. The system consists of a camera installed in the refrigerator, a server, a user terminal, an object recognition algorithm, a database, natural language processing, and an emotion engine.

[1529] Hardware and Software Configuration

[1530] camera

[1531] There are multiple high-resolution cameras installed inside the refrigerator. These cameras (e.g., high-resolution cameras) have the ability to capture an image every time the refrigerator door is closed.

[1532] server

[1533] The server receives the image data sent from the camera and begins processing. First, the image is pre-processed. Specifically, camera noise is removed and color correction is performed. After this pre-processing is complete, the image is input into an object recognition algorithm (e.g., YOLO) to identify the type and quantity of food.

[1534] Database

[1535] The server stores the identified food information in a database (e.g., MySQL). When a new food is added, the information is updated in the database and existing food information is updated.

[1536] User terminal

[1537] The user terminal is a device such as a smartphone or tablet that allows the user to check the information inside the refrigerator. The latest food information is sent from the server to the terminal, allowing the user to check the status of the refrigerator in real time. For example, information such as "There is one cabbage and two tomatoes in the refrigerator" is displayed.

[1538] Natural Language Processing

[1539] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server. The server uses natural language processing (e.g., BERT) to analyze the question and compares it with information in the database to generate an appropriate answer. For example, the question "How many tomatoes are there?" will generate an answer such as "There are two tomatoes."

[1540] Emotion Engine

[1541] The server is equipped with an emotion engine (e.g., Affectiva SDK) that analyzes the user's voice, text, and facial expressions to recognize their emotions. This allows it to analyze the user's emotions and customize corresponding notifications and responses. For example, if the user is feeling anxious, the server may notify them that "The expiration date of the cabbage is approaching, so please use it up quickly."

[1542] Customize notifications based on emotions

[1543] The server customizes the notification content and response based on the user's emotional information obtained by the emotion engine. For example, if the user looks anxious, a notification offering support such as "There are two tomatoes. Can I help you?" will be sent.

[1544] Examples of concrete examples and prompts

[1545] For example, whenever a user opens or closes the refrigerator, the camera captures an image and the server updates the food information based on this. For example, when a user asks, "How many tomatoes are there?", the server responds via the device with, "There are two tomatoes."

[1546] Example prompt sentence:

[1547] "Capture an image of the inside of your refrigerator, identify the food in the crisper, and store it in a database."

[1548] This system allows users to manage the food in their refrigerator efficiently and without waste, and also provides a pleasant user experience by receiving appropriate notifications and responses based on their emotions.

[1549] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1550] Step 1:

[1551] Camera installation and image acquisition

[1552] The server installs multiple high-resolution cameras inside the refrigerator. Each time the refrigerator door is closed, these cameras are activated and capture images of the inside of the refrigerator. Specifically, the cameras are automatically activated when the refrigerator door sensor detects that the door is closing, and images of multiple locations inside the refrigerator are taken.

[1553] Input: Actual view inside the refrigerator, door sensor signal

[1554] Output: High-resolution image of the inside of the refrigerator

[1555] Step 2:

[1556] Image preprocessing

[1557] The server receives the captured images and performs pre-processing, such as noise reduction and color correction to improve image quality, resulting in image data ready to be input into object recognition algorithms.

[1558] Input: High-resolution image

[1559] Output: Preprocessed image

[1560] Step 3:

[1561] object recognition

[1562] The server inputs the preprocessed image into an object recognition algorithm (e.g., YOLO). The algorithm identifies the type and quantity of food and identifies the results. Specifically, the YOLO algorithm analyzes each pixel in the image and identifies specific objects (e.g., cabbage, tomato).

[1563] Input: Preprocessed image

[1564] Output: Identified food types and quantities

[1565] Step 4:

[1566] Data storage

[1567] The server stores the identified food information in a database (e.g., MySQL). When a new food is added, the information is saved in the database and existing food information is updated as necessary. Specifically, the new and existing data are compared and the differences are updated.

[1568] Input: Type and quantity of food identified

[1569] Output: Updated database contents

[1570] Step 5:

[1571] Notification to user device

[1572] The server sends the latest food information to the user's device (smartphone or tablet), allowing the user to check the status of the refrigerator in real time. For example, a notification might say, "There is one cabbage and two tomatoes in the refrigerator."

[1573] Input: Updated database contents

[1574] Output: Food information displayed on the user's device

[1575] Step 6:

[1576] Answering questions using natural language processing

[1577] When a user asks a question about the status of the refrigerator by voice or text, the device sends the question to the server. The server uses natural language processing (e.g., BERT) to analyze the question, compares it with information in the database, generates an appropriate answer, and notifies the user via the device. For example, the question "How many tomatoes are there?" is answered with "There are two tomatoes."

[1578] Input: User question (voice or text)

[1579] Output: Response to user terminal

[1580] Step 7:

[1581] Implementing the Emotion Engine

[1582] The server is equipped with an emotion engine (e.g., Affectiva SDK) that analyzes the user's voice, text, and facial expressions to recognize their emotions. This allows it to analyze the user's emotions and generate appropriate notifications and responses. For example, if the user is feeling impatient, the server will notify them by saying, "The cabbage's expiration date is approaching, so please use it up quickly."

[1583] Input: User voice, text, facial expressions

[1584] Output: Notification based on user sentiment

[1585] Step 8:

[1586] Customize notifications based on emotions

[1587] The server customizes the notification content and response based on the user's emotional information obtained from the emotion engine. For example, if the user looks anxious, the server sends a notification offering support such as, "There are two tomatoes. Can I help you?"

[1588] Input: User's emotional information

[1589] Output: Customized notification content

[1590] (Application example 2)

[1591] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1592] Conventional refrigerator management systems and store management systems only had the function of managing inventory information and were unable to respond or notify users based on their emotions. Furthermore, when it came to in-store product management, it was difficult to check inventory in real time, making it impossible to respond quickly to customer needs. This resulted in a poor user experience and a decline in the efficiency of inventory management and customer service.

[1593] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring images of the inside of the refrigerator and the inside of the store, means for processing the images and identifying food and products, and means for analyzing customer emotions and customizing notifications and responses. This enables appropriate responses according to user emotions and real-time management of inventory in the refrigerator and the store.

[1594] A "server" is a device that provides computing resources and storage, analyzes and stores various types of data, and manages communications.

[1595] The "means for acquiring images inside the refrigerator" is a technology for capturing images using a camera installed inside the refrigerator and acquiring them as data.

[1596] The "means for processing the image and identifying the food" refers to a technology for preprocessing the acquired image and identifying the food using an object recognition algorithm.

[1597] The "means for storing information on the identified food in a database" refers to a technique for recording data on the type and quantity of the identified food in a database.

[1598] "Means for notifying the user terminal of the information in the database" refers to technology that sends food information and update information stored in the database to the user's terminal such as a smartphone or tablet.

[1599] "Means for analyzing questions from users using natural language processing and generating responses based on information in the database" refers to technology that analyzes voice or text questions uttered by users and generates appropriate answers based on information in the database.

[1600] The "means for acquiring images within the store" refers to a technology for capturing images using cameras installed within the store and acquiring them as data.

[1601] The "means for processing the image and identifying the product" refers to a technology for preprocessing images acquired in a store and identifying the product using an object recognition algorithm.

[1602] "Means for analyzing customer emotions and customizing notifications and responses" refers to technology that reads emotions from a customer's facial expressions, voice, text, etc., and uses that information to customize the content of notifications and responses.

[1603] "Means for periodically reacquiring images of the inside of the refrigerator and the inside of the store and updating them to the latest state" refers to a technology that periodically operates the camera to acquire the latest images and keep the contents of the database up to date.

[1604] "Means for sending notifications to a user terminal based on the expiration dates of the food and products" refers to a technology that sends warnings and notifications to a user terminal about products that are approaching their expiration date based on expiration date information of stored food and products.

[1605] The embodiments for carrying out the present invention are as follows.

[1606] 1. System Overview

[1607] This system improves the efficiency of inventory management in refrigerators and physical stores, and provides responses based on the user's emotions. The system mainly consists of a server, a terminal, a camera, an emotion analysis engine, an object recognition algorithm, and a database.

[1608] 2. Hardware and Software

[1609] Cameras: Installed in refrigerators and in stores. Example: Logitech C920 webcam

[1610] Server: Responsible for data processing and storage. Example: AWS EC2 instance

[1611] Terminal: The device on which the user receives information. Examples: smartphone, tablet

[1612] software:

[1613] OpenCV: Image capture and preprocessing library

[1614] Object Recognition Algorithm: Identifying food and goods using YOLOv3 and ResNet

[1615] Database: Manage inventory data with MySQL or Firebase

[1616] Sentiment analysis engine: Uses Microsoft Azure's Emotion API

[1617] Natural Language Processing: Question Answering with GPT-3

[1618] 3. System Operation

[1619] 1. Image Acquisition

[1620] The server periodically activates the cameras installed in the refrigerator and in the store to capture the latest images. For example, the server can activate the camera and capture an image every time the refrigerator door is closed.

[1621] 2. Image Preprocessing and Analysis

[1622] The server preprocesses the acquired images using OpenCV. Specifically, it performs noise removal and color correction. The preprocessed images are input into an object recognition algorithm, and the server identifies food and products. For example, it identifies cabbages and tomatoes from an image of a vegetable drawer and determines their quantities.

[1623] 3. Data storage

[1624] The server stores the identified food and product information in a database. If there are any changes compared to existing data, the information is updated. For example, if a new item, "1 cabbage," is added to the database, the information is updated.

[1625] 4. Notice to Users

[1626] The server sends the latest information about what's in the refrigerator and in the store to the terminal. Users can check inventory status in real time via their smartphone or tablet. For example, the terminal will display information like, "There is one cabbage and two tomatoes in the refrigerator." In addition, it will send a warning if the expiration date is approaching.

[1627] 5. Answering questions using natural language processing

[1628] When a user asks a question about the inventory in the refrigerator or store by voice or text, the device sends the question to the server. The server uses GPT-3 to analyze the question, compares it with the database, and generates an appropriate answer. For example, if a user asks, "How many tomatoes are there?", the device will respond, "There are two tomatoes."

[1629] 6. Implementing the Emotion Engine

[1630] The server is equipped with an emotion analysis engine that recognizes emotions from the user's voice, text, facial expressions, etc. This allows it to analyze the user's emotions and customize corresponding notifications and responses.

[1631] 7. Customize notifications based on emotions

[1632] The server customizes notifications and responses based on the user's emotional information obtained from the emotion analysis engine. For example, if the user is in a hurry, the server will calmly display a notification such as, "The expiration date of the cabbage is approaching. Please use it up quickly." If the user is anxious, the server will send a notification offering support, such as, "You have two tomatoes. Is there anything I can help you with?"

[1633] 8. Examples of prompt sentences

[1634] "Tell me the stock situation"

[1635] "Do you have product A?"

[1636] Specific use cases

[1637] When a user asks "Do you have any cabbages?" on their smartphone, the server checks the database and responds, "We have one cabbage." If the user looks anxious, the server responds with a gentler response, such as, "We have two tomatoes. May I help you?"

[1638] Such a system enables flexible responses that adapt to the user's emotions, and allows for appropriate management of inventory status in refrigerators and stores.

[1639] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1640] Step 1:

[1641] The server activates the cameras installed in the refrigerator and in the store to capture images. Based on the request, the server obtains image data from the cameras and preprocesses it. Preprocessing includes noise reduction and color correction. The input is the raw image captured by the camera, and the output is the preprocessed image.

[1642] Step 2:

[1643] The server inputs the preprocessed images into an object recognition algorithm (e.g., YOLOv3 or ResNet) to identify food and store items. The input is the preprocessed image, and the output is the type and quantity of identified food or items. Specifically, the algorithm identifies objects in the image and returns their labels and locations.

[1644] Step 3:

[1645] The server stores the identified food and product information (type, quantity) in a database. It compares the data with existing data and updates it if there are any changes. The input is the identified food or product data, and the output is the updated database entry. For example, information such as "two tomatoes were added" is recorded in the database.

[1646] Step 4:

[1647] The server sends the latest information about what's in the refrigerator and the store to the user terminal. Here, it retrieves the latest information from the database and generates a message to send to the user terminal. The input is inventory information retrieved from the database, and the output is a notification message to be sent to the user terminal. For example, the message might say, "There is one cabbage and two tomatoes in the refrigerator."

[1648] Step 5:

[1649] The user asks a question about the inventory in the refrigerator or in the store by voice or text. The terminal sends the question to the server. The input to the terminal is the user's question, and the output is the question data to the server. For example, the terminal sends a user's question such as "How many tomatoes are there?" to the server.

[1650] Step 6:

[1651] The server uses GPT-3 to analyze the question, compare it with the database, and generate an appropriate answer. The input is the user's question data received from the device, and the output is the generated answer. Specifically, it uses natural language processing to understand the intent of the question and constructs an answer based on related information in the database.

[1652] Step 7:

[1653] The server uses an emotion analysis engine to analyze the user's emotions and recognizes emotions from voice, text, and facial expressions. The input is the user's voice data, text data, and image data, and the output is analyzed emotional information. Specifically, the emotion analysis engine analyzes this data and determines whether the user is anxious or impatient.

[1654] Step 8:

[1655] The server customizes notifications and responses based on the user's emotional information obtained by the emotion analysis engine. The input is the user's emotional information and the generated response information, and the output is a customized notification message. For example, if the user is in a hurry, the server sends a notification such as, "Remain calm. The expiration date of the cabbage is approaching. Please use it up quickly."

[1656] Step 9:

[1657] The server sends a customized notification to the user terminal. The input is a customized notification message, and the output is a notification displayed on the user terminal, allowing the user to receive accurate information.

[1658] Through the above steps, a system is realized in which the server, terminals, and users work together to manage inventory and handle customer service.

[1659] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1660] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1661] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1662] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1663] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1664] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1665] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1666] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1667] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1668] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1669] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1670] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1671] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1672] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1673] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1674] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1675] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1676] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1677] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1678] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1679] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1680] The following is further disclosed regarding the above embodiment.

[1681] (Claim 1)

[1682] A means for acquiring an image of the inside of a refrigerator;

[1683] means for processing the image and identifying the food product;

[1684] means for storing information of the identified food products in a database;

[1685] means for notifying a user terminal of information in the database;

[1686] means for analyzing a question from a user using natural language processing and generating a response based on information in the database;

[1687] A system including:

[1688] (Claim 2)

[1689] 10. The system of claim 1, further comprising means for periodically reacquiring images of the interior of the refrigerator and updating the images to the latest state.

[1690] (Claim 3)

[1691] 10. The system of claim 1, further comprising means for sending a notification to a user terminal based on the expiration date of the food product.

[1692] "Example 1"

[1693] (Claim 1)

[1694] A means for acquiring an image of the inside of a refrigerator;

[1695] means for preprocessing the image;

[1696] means for analyzing the preprocessed image to identify the food product;

[1697] means for storing information of the identified food products in a database;

[1698] means for notifying a user terminal of information in the database;

[1699] means for analyzing a question from a user using natural language processing and generating a response based on information in the database;

[1700] A system including:

[1701] (Claim 2)

[1702] 10. The system of claim 1, further comprising means for periodically reacquiring images of the interior of the refrigerator and updating the images to the latest state.

[1703] (Claim 3)

[1704] 10. The system of claim 1, further comprising means for sending a notification to a user terminal based on the expiration date of the food product.

[1705] "Application Example 1"

[1706] (Claim 1)

[1707] A means for acquiring an image of the inside of a refrigerator;

[1708] means for processing the image and identifying the food product;

[1709] means for storing information of the identified food products in a database;

[1710] means for notifying a user terminal of information in the database;

[1711] means for analyzing a question from a user using natural language processing and generating a response based on information in the database;

[1712] A means for allowing a user to check the information in the database in real time through an application installed on a smart device;

[1713] means for notifying a user terminal of food products approaching their expiration date;

[1714] A system including:

[1715] (Claim 2)

[1716] 10. The system of claim 1, further comprising means for periodically reacquiring images of the interior of the refrigerator and updating the images to the latest state.

[1717] (Claim 3)

[1718] 10. The system of claim 1, further comprising means for sending a notification to a user terminal based on the expiration date of the food product.

[1719] "Example 2: Combining Emotion Engines"

[1720] (Claim 1)

[1721] A means for acquiring an image of the inside of the refrigerator by a camera;

[1722] means for pre-processing the images and identifying food items using an object recognition algorithm;

[1723] means for storing information of the identified food products in a database;

[1724] means for notifying a user terminal of information in the database;

[1725] means for analyzing a question from a user using natural language processing and generating a response based on information in the database;

[1726] A means for analyzing a user's voice, text, and facial expressions to recognize emotions;

[1727] means for customizing notification content and responses based on the recognized emotion;

[1728] A system including:

[1729] (Claim 2)

[1730] 10. The system of claim 1, further comprising means for periodically reacquiring images of the interior of the refrigerator and updating the images to the latest state.

[1731] (Claim 3)

[1732] 10. The system of claim 1, further comprising means for sending a notification to a user terminal based on the expiration date of the food product.

[1733] "Application example 2 when combining emotion engines"

[1734] (Claim 1)

[1735] A means for acquiring an image of the inside of a refrigerator;

[1736] means for processing the image and identifying the food product;

[1737] means for storing information of the identified food products in a database;

[1738] means for notifying a user terminal of information in the database;

[1739] means for analyzing a question from a user using natural language processing and generating a response based on information in the database;

[1740] A means for acquiring images of the inside of the store;

[1741] means for processing the image and identifying the product;

[1742] A way to analyze customer sentiment and customize notifications and responses;

[1743] A system including:

[1744] (Claim 2)

[1745] The system of claim 1, further comprising means for periodically reacquiring images of the interior of the refrigerator and the interior of the store and updating them to the latest state.

[1746] (Claim 3)

[1747] The system of claim 1 , further comprising means for sending notifications to a user terminal based on the expiration dates of the food products and merchandise. [Explanation of symbols]

[1748] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for acquiring an image of the inside of a refrigerator; means for processing the image and identifying the food product; means for storing information of the identified food products in a database; means for notifying a user terminal of information in the database; means for analyzing a question from a user using natural language processing and generating a response based on information in the database; A system including:

2. 2. The system of claim 1, further comprising means for periodically reacquiring images of the interior of the refrigerator and updating the images to the latest state.

3. The system of claim 1 , further comprising means for sending a notification to a user terminal based on the expiration date of the food product.

Images