system

Abstract

Provide a system. [Solution] a camera means for taking pictures of the inside of the refrigerator; image analysis means for analyzing image data captured by the camera means and classifying the food items by category; an inventory management means for managing inventory information for each food category recognized by the image analysis means; an EC automatic ordering means for automatically ordering shortage items based on the inventory management means; a recipe suggestion means for suggesting recipes that can be used by the user based on the inventory management means; The system includes an advertisement product notification means for collecting advertisement product information from designated nearby stores and notifying the user of the information.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In the current situation where it is difficult to accurately grasp the inventory status in the refrigerator, duplicate purchases and shortages of food are likely to occur, resulting in food waste, wasteful shopping, and difficulty in making daily cooking decisions. Furthermore, users often have difficulty deciding which store to shop at, making it difficult to make the most economical choice. This invention aims to solve these problems and provide efficient and waste-free food management and cooking support. [Means for solving the problem]

[0005] This invention provides a system that includes a camera means for taking pictures of the inside of a refrigerator and an image analysis means for analyzing the image data taken by the camera means and identifying the food by category. It also includes an inventory management means for managing inventory information for each food category recognized by the image analysis means, and an EC automatic ordering means for automatically ordering missing items based on the inventory management means. It also includes a recipe suggestion means for suggesting recipes available to the user based on the inventory management means, and an advertised item notification means for collecting advertised item information from selected nearby stores and notifying the user. Furthermore, the advertised item notification means includes an algorithm that calculates the user's location information and the distance to the store and presents the most economical options, thereby providing a system that comprehensively supports the user's daily life.

[0006] "Camera means" means a photographic device positioned to take images of food items within the refrigerator.

[0007] The "image analysis means" is software or an algorithm for analyzing image data acquired by the camera means and identifying food items by category.

[0008] The "inventory management means" is a system or method for managing inventory information for each food category recognized by the image analysis means in a database or on the cloud.

[0009] "EC automatic ordering means" refers to a system or function for automatically placing online orders for items that are in short supply based on inventory information.

[0010] A "recipe suggestion means" is a system or function that suggests recipes that can be used by the user based on inventory information.

[0011] The "advertised item notification means" is a system or method that collects advertised item information from designated nearby stores and notifies the user of that information.

[0012] An "algorithm" is a procedure or computational method for solving a specific problem, and in this case refers to a computational procedure for presenting economic options. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0014] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0015] First, the terms used in the following description will be explained.

[0016] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0017] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0018] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0019] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0020] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0021] [First embodiment]

[0022] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0023] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0024] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0025] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0026] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0027] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0028] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0029] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0030] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0031] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0032] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0033] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0034] The present invention relates to a system for efficiently managing inventory in a refrigerator, allowing users to achieve optimal shopping and cooking. Detailed embodiments of the system will be described below.

[0035] System Overview

[0036] This system includes a camera installed inside the refrigerator, image analysis, inventory management, automatic e-commerce ordering, recipe suggestion, and advertisement notification. By linking these means, users can grasp the food inventory in their refrigerator in real time and receive suggestions for efficient shopping and cooking.

[0037] Photo taken inside the refrigerator

[0038] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, it takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. These pictures are used to record the types of food in the refrigerator and their arrangement.

[0039] Image data transmission and analysis

[0040] The captured image data is sent from the device (refrigerator camera) to a server via the Internet. The server analyzes the received images and uses an image recognition algorithm to identify the food by category. The analysis results are stored in a database and managed as inventory information.

[0041] Managing and checking inventory information

[0042] Users can check the inventory information in their refrigerator in real time through a dedicated app. The app retrieves the latest inventory information from the server and displays it to provide users with the current inventory status. For example, when a user opens the app, they can immediately see how many items are in the refrigerator.

[0043] Automatic EC ordering of missing items

[0044] Based on inventory information, food items that fall below a set threshold are automatically ordered online. The server periodically checks inventory information, and when the required food items become scarce, it automatically places an order with the registered e-commerce site. For example, if there is less than one bottle of milk, an order is automatically placed.

[0045] Recipe suggestions

[0046] Based on the inventory information, the server suggests recipes that can be made by the user. The server analyzes the inventory data, selects appropriate recipes, and sends them to the app. The user can check the suggested recipes on the app and start cooking right away. For example, if there is chicken and vegetables in the refrigerator, a recipe such as "stir-fried chicken and vegetables" will be suggested.

[0047] Advertisement Notice

[0048] The system also includes a function to collect advertised product information from selected nearby stores and notify users. The server obtains advertising information from local supermarkets and compares it with inventory information to identify products that users should be notified about. It then calculates the user's location and the distance to the store, and uses an algorithm to present the most economical option. For example, a LINE notification might be sent to the user saying, "Oranges are on sale at a nearby supermarket. It's a good deal if you go and buy some."

[0049] Example: User A's experience

[0050] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User A opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[0051] In the evening, Mr. A notices that there is little yogurt in the refrigerator, but the system has already automatically ordered it and notifies him that it will be delivered. Furthermore, he receives a LINE notification that a nearby supermarket is having a special sale on oranges, so he heads to the supermarket and does his shopping efficiently.

[0052] In this way, the present invention provides comprehensive support for the user's daily food management, shopping, and cooking, reducing waste and improving efficiency.

[0053] The processing flow will be explained below.

[0054] Step 1:

[0055] The device (refrigerator camera) takes pictures of the entire interior of the refrigerator according to a set schedule, for example, at 8:00 AM and 8:00 PM every day. These pictures are taken with a wide angle to capture the entire interior of the refrigerator.

[0056] Step 2:

[0057] The captured image data is compressed from the device (refrigerator camera) and sent to a server via the Internet, along with an acknowledgement message to confirm successful data transmission.

[0058] Step 3:

[0059] The server saves the received image data in a directory for analysis. If the save is successful, it records that fact in the log.

[0060] Step 4:

[0061] The server then analyzes the stored image data using image recognition algorithms, which identify the food items in the image by category and determine their type and quantity (for example, three apples, two tomatoes, etc.).

[0062] Step 5:

[0063] Based on the analysis results, the server updates the database with inventory information for each food category. The inventory information is updated in real time, ensuring that it is always up to date.

[0064] Step 6:

[0065] The user launches a dedicated app, which sends a request to the server to check stock availability.

[0066] Step 7:

[0067] The server retrieves the latest inventory information from the database and sends it back to the app, which displays the information to the user in real time through the app's UI.

[0068] Step 8:

[0069] The server periodically checks inventory to see if any food items are running low, and if they fall below a threshold, it generates an automatic order request for the item.

[0070] Step 9:

[0071] The server executes the automated order through the API of the e-commerce site, using shipping and payment information based on the user's pre-defined settings.

[0072] Step 10:

[0073] The server suggests recipes that can be made by the user based on inventory information, selects appropriate recipes, and sends the suggestions to the app.

[0074] Step 11:

[0075] Users can check the suggested recipes on the app, refer to the cooking instructions if necessary, and start cooking according to the recipe they select.

[0076] Step 12:

[0077] The server collects advertising information from local supermarkets and compares it with current inventory information to identify items that need to be purchased.

[0078] Step 13:

[0079] The server takes into account the user's location information, calculates travel distance and economic efficiency based on the displayed advertising information, and determines the most effective shopping method.

[0080] Step 14:

[0081] The server will provide users with advertisements for specific products and recommended store information for purchasing those products via LINE notifications, allowing users to shop in the most economical way.

[0082] Example 1

[0083] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0084] Conventional refrigerator inventory management requires manual effort and time from the user. There is also a high possibility of food shortages or waste due to duplicate purchases, making efficient food management difficult. Furthermore, there is no way to check inventory information in real time, making it inconvenient to be unable to check the inventory status of the refrigerator while shopping on the go. A new system is needed to solve these problems.

[0085] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0086] In this invention, the server includes an imaging means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the imaging means and classifying food items by category, a data management means for managing inventory information for each food category recognized by the image analysis means, an automatic e-commerce ordering means for automatically ordering missing items based on the data management means, a suggestion means for suggesting recipes that the user can use based on the data management means, an advertisement information notification means for collecting advertisement information from designated nearby stores and notifying the user, and a cloud data synchronization means for saving the inventory information saved in the data management means to the cloud and enabling the information to be checked in real time. This allows the user to efficiently manage their refrigerator inventory, prevent stockouts, and purchase and cook optimal ingredients.

[0087] "Imaging means" refers to a device for capturing images of the inside of the refrigerator, and includes cameras and other imaging devices for recording the types and arrangement of food items inside the refrigerator.

[0088] The "image analysis means" is a computer algorithm or program for analyzing image data acquired by the imaging means and identifying food items by category.

[0089] The "data management means" is a system for storing and managing inventory information for each food category recognized by the image analysis means, and includes a database and cloud storage.

[0090] "Automatic e-commerce ordering means" refers to a system that has the function of automatically ordering shortage items online based on data management means, and includes linkage with an e-commerce platform.

[0091] The "suggestion means" is a system that has the function of suggesting recipes that can be used by the user based on the data management means, and includes an algorithm that works in conjunction with the recipe database.

[0092] The "advertising information notification means" is a system that has the function of collecting advertising information from designated nearby stores and notifying the user, and includes an algorithm that works in conjunction with the user's location information.

[0093] The "cloud data synchronization means" is a system that stores inventory information stored in the data management means in the cloud, enabling information to be checked in real time.

[0094] The present invention relates to a system that efficiently manages food inventory in a refrigerator, enabling users to optimize their shopping and cooking. The system includes an imaging unit, an image analysis unit, a data management unit, an automatic e-commerce ordering unit, a suggestion unit, a promotional information notification unit, and a cloud data synchronization unit, all installed in the refrigerator.

[0095] Hardware or software configuration

[0096] Imaging means

[0097] The terminal (refrigerator camera) is a device for taking high-resolution images, and periodically takes pictures of the inside of the refrigerator, for example, every 24 hours. This camera has wireless communication capabilities and sends image data to a server via the Internet.

[0098] Image analysis methods

[0099] The server uses a high-performance image recognition algorithm (e.g., YOLO, ResNet) to analyze the received image data. This algorithm classifies food by category and identifies the food based on the extracted features.

[0100] Data Management Measures

[0101] The server stores the identified food information in a database, which includes information such as the food ingredient classification, quantity, and location, and manages it so that users can easily check the inventory status.

[0102] E-commerce automatic ordering tool

[0103] The server has the ability to automatically order food items online based on data management methods. For example, if there is less than one bottle of milk, a new order is placed via a pre-registered e-commerce site API.

[0104] Proposal means

[0105] The server proposes recipes that the user can cook based on the data management method. The server analyzes inventory data, selects appropriate recipes, and sends them to the app. The user can check the proposed recipes on the app and start cooking right away.

[0106] Promotional information notification means

[0107] The server collects promotional information from nearby stores and notifies the user. This function presents the most economical option based on the user's location and the distance to the store. For example, a LINE notification can be sent to the user to let them know about special sales at a nearby supermarket.

[0108] Cloud data synchronization method

[0109] The server stores inventory information on the cloud and updates it in real time, allowing users to check the inventory status of their refrigerators even when they are out and about.

[0110] Specific examples

[0111] User A's experience

[0112] A camera installed in the refrigerator takes an image of the inside of the refrigerator at 8:00 AM and sends it to a server. The server analyzes the image and determines that there are three apples and two tomatoes. User A opens a dedicated app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "apple and tomato salad."

[0113] In the evening, User A notices that there is little yogurt in the refrigerator, but is notified that new yogurt will arrive, as the system has already automatically placed an order. Furthermore, User A receives a LINE notification that a nearby supermarket is having a special sale on oranges, allowing User A to go to the supermarket and complete their shopping efficiently.

[0114] Example prompts to input to the generative AI model

[0115] "A camera installed inside the refrigerator takes pictures of the inside at 8:00 AM and sends them to a server. The server then analyzes the images and displays inventory information on an app. With this app, users can check the inventory in their refrigerator in real time and receive recipe suggestions, automatic ordering, and notifications of advertised products. Please explain this system."

[0116] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0117] Step 1: Photographing the inside of the refrigerator using imaging means

[0118] The device (refrigerator camera) takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. This camera is positioned to cover the entire inside of the refrigerator and captures high-resolution images. The input is the image data of the inside of the refrigerator taken by the camera, and the output is the image file. In this step, the types and arrangement of food in the refrigerator are recorded in detail.

[0119] Step 2: Sending image data

[0120] The device (refrigerator camera) sends the captured image data to a server via the Internet. Here, the input is the captured image data, and the output is the image data sent to the server. This step requires highly reliable data transmission, and data is transferred using protocols such as HTTP.

[0121] Step 3: Image analysis

[0122] The server analyzes the received image data using an image analysis algorithm (e.g., YOLO, ResNet). The input is the image data sent to the server, and the output is food data identified by category. In this step, food items in the image are identified by category and features such as size, shape, and color are extracted. For example, the server accurately recognizes foods such as apples, tomatoes, and milk from the image and registers them in a database.

[0123] Step 4: Update data management

[0124] The server stores the identified food information in a database and updates the inventory information. The input is the food identified by category, and the output is the updated database. In this step, information such as food classification, quantity, and location is recorded in the database and a timestamp is added to allow tracking of inventory fluctuations over time.

[0125] Step 5: Check inventory information

[0126] The user checks the inventory information in the refrigerator in real time through a dedicated app. The app periodically connects to the server to retrieve and display the latest inventory information. The input is the inventory information retrieved from the server, and the output is the inventory information displayed to the user. When the user opens the app, specific inventory information, such as "3 apples, 2 tomatoes," is immediately displayed.

[0127] Step 6: Automatically order missing items online

[0128] The server periodically checks the inventory information in the database and automatically places an order online for any food items that are running low. The input is the updated inventory data, and the output is the order data sent to the e-commerce platform. For example, if there is only one bottle of milk left, the server sends the necessary information to the e-commerce site API to place an automatic order.

[0129] Step 7: Recipe suggestions

[0130] The server suggests recipes that can be made by the user based on inventory information. The input is inventory information in the database, and the output is suggested recipe information. The server analyzes the inventory data, selects appropriate recipes from the database, and sends them to the app. The user can then view specific recipe suggestions, such as "apple and tomato salad," on the app.

[0131] Step 8: Promotional Information Notification

[0132] The server collects advertising information from nearby stores and notifies the user. The input is the collected advertising information, and the output is the advertising information notified to the user. The server obtains advertising data from local supermarkets and compares it with inventory information to identify products to notify the user about. Furthermore, the server calculates the travel distance to the optimal store based on the user's location information, and sends the user, for example, information about special sales at nearby supermarkets via LINE notifications.

[0133] (Application example 1)

[0134] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0135] Conventional refrigerator inventory management systems are limited to efficient inventory management and shopping optimization within the home, and have not been applied to inventory management or product recommendation efficiency in physical stores. In addition, there is a lack of a system for store clerks to check inventory information in real time and make optimal product recommendations to customers, so improving inventory management efficiency and customer satisfaction are issues. Furthermore, there is a need for a means to increase customer purchasing motivation by appropriately notifying customers of advertising information and promotions.

[0136] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0137] In this invention, the server includes an imaging means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the imaging means and classifying the food by category, an inventory management means for managing inventory information for each food category recognized by the image analysis means, an automatic e-commerce ordering means for automatically ordering missing items based on the inventory management means, a recipe suggestion means for suggesting recipes available to the user based on the inventory management means, an advertised item notification means for collecting advertised item information from selected nearby stores and notifying the user, a visualization means for displaying in-store inventory information in real time on a display device worn by the user and checking inventory and suggesting products, an automatic replenishment means for automatically replenishing in-store inventory when inventory falls below a certain threshold, and a notification means for notifying the display device of advertising information and promotional information. This improves the efficiency of inventory management in physical stores, enables store staff to suggest optimal products to customers in real time, and improves customer satisfaction by effectively notifying them of advertising information and promotional information.

[0138] The "imaging means" is a device that takes images of the inside of a refrigerator or a store.

[0139] The "image analysis means" is a device or software that has the function of analyzing captured image data and identifying food or product categories.

[0140] The "inventory management means" is a device or software that has the function of managing inventory information recognized by the image analysis means and updating it in real time.

[0141] An "automated electronic commerce ordering means" is a device or software that has the function of automatically ordering products that are in short supply based on inventory information.

[0142] The "recipe suggestion means" is a device or software that has the function of suggesting recipes that can be used by the user based on inventory information.

[0143] The "advertised item notification means" is a device or software that has the function of notifying the user of advertised item information collected from designated nearby stores.

[0144] A "display device" is a display device worn by a user to visually confirm information.

[0145] The "visualization means" is a device or software that displays inventory information in real time through a display device and has the function of checking inventory and making product suggestions.

[0146] An "automatic replenishment means" is a device or software that has the function of automatically replenishing products when inventory falls below a certain threshold.

[0147] The "notification means" is a device or software that has the function of notifying the display device of advertising information or promotional information.

[0148] The present invention relates to a system for improving the efficiency of inventory management and product recommendations in a physical store. Detailed embodiments of the present invention will be described below.

[0149] System Overview

[0150] This system is comprised of an imaging means, an image analysis means, an inventory management means, an automatic e-commerce ordering means, a recipe suggestion means, an advertised product notification means, a visualization means, an automatic replenishment means, and a notification means. By linking these means, efficient inventory management, product suggestions, and advertising information notifications in physical stores become possible.

[0151] Photography and image analysis

[0152] The server first periodically captures images of the product shelves in the store using an imaging device. This capture is performed automatically at regular intervals. The captured image data is sent to the server and analyzed by the image analysis device. The image analysis device then classifies the food and product by category and stores the data in a database as inventory information.

[0153] Inventory management and e-commerce automatic ordering

[0154] The server updates the inventory information recognized by the image analysis means in real time using the inventory management means. The user can wear a display device and instantly check the current inventory information using the visualization means. Based on this inventory information, if the inventory falls below a certain threshold, the e-commerce automatic ordering means automatically places a replenishment order.

[0155] Recipe suggestions and promotional product notifications

[0156] The server is equipped with a recipe suggestion unit that suggests recipes available to the user based on inventory information. Furthermore, an advertised product notification unit notifies the user of advertised product information collected from designated nearby stores. The server calculates the user's location information and the travel distance to the store, and uses an algorithm to present the most economical options.

[0157] In-store use

[0158] Store staff wear display devices and use visualization tools to check inventory information in real time. For example, if an item on a shelf runs low, an automatic replenishment tool is activated to immediately replenish it. It is also possible to check whether the product desired by the customer is in stock and present alternative options.

[0159] Notification means

[0160] The advertising information and promotion information are displayed on the display device using the notification means, allowing the store staff to provide the customer with appropriate product information.

[0161] Examples of concrete examples and prompts

[0162] For example, when a store clerk wears a display device and activates the "Check Stock" command, real-time inventory information is displayed on the display. For example, it may show "Milk: 10, Eggs: 5, Bread: 2." Also, when the "Auto Order" command is executed, inventory below a threshold (for example, bread) is automatically ordered.

[0163] Below are some example prompts for a generative AI model:

[0164] Can you give us an example of a system that streamlines refrigerator inventory management? A system that meets the following requirements: 1. Periodically takes pictures of the inside of the refrigerator using a camera. 2. Analyzes the images and updates inventory information. 3. Checks inventory information in real time using a smartphone or smart glasses. 4. Automatically places e-commerce orders when inventory is low. 5. Suggests recipes that can be made. 6. Notifies users of advertised product information.

[0165] The above is the details of the embodiment of the invention. This system is expected to improve the efficiency of inventory management in stores and increase customer satisfaction.

[0166] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0167] Step 1:

[0168] The server periodically captures images of the product shelves in the store using an imaging device. It receives the time interval and camera position as input and obtains the captured image data as output. The images captured by this imaging device are saved in a storage directory.

[0169] Step 2:

[0170] The server sends the captured image data to the image analysis means, which receives the image data as input and outputs the analysis results. The image analysis means uses machine learning algorithms to identify each item in the image by category and stores the results in a database.

[0171] Step 3:

[0172] The server retrieves inventory information from the database and manages the latest inventory information using inventory management means. It receives analysis result data as input and saves updated inventory information in the database as output. This is done in real time by the inventory management means.

[0173] Step 4:

[0174] The server uses visualization means to display the latest inventory information via a display device worn by the user. The latest inventory information is acquired as input, and the inventory information displayed on the display is obtained as output. This allows the user to check the current inventory status.

[0175] Step 5:

[0176] When inventory information falls below a certain threshold, the server automatically orders the missing items using an automatic e-commerce ordering system. The server receives inventory information and a set threshold as input, and obtains data related to the automatic order as output. This allows for a rapid response when inventory shortages occur.

[0177] Step 6:

[0178] The server activates a recipe suggestion means that suggests available recipes to the user based on the inventory information. The server receives the inventory information as input and generates suggested recipe information as output. The user can check these suggested recipes on the display device.

[0179] Step 7:

[0180] Using the advertising product notification means, the server notifies the user of advertising information collected from the set nearby stores. It receives advertising information and the user's location information as input and generates an optimal advertising notification as output. The user can learn about economical options through the display device.

[0181] Step 8:

[0182] The advertising information and promotion information are displayed on the display device by the notification means. The advertising information is used as input and the notification on the display is used as output. This allows the user and store clerk to instantly check the latest promotion information.

[0183] Step 9:

[0184] The user checks inventory and makes product suggestions via a display device, and replenishes products or makes suggestions to customers based on the information displayed on the display.The system receives instructions from the display device as input and generates specific product suggestions and replenishment actions as output.

[0185] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0186] The present invention is a system that provides a more personalized dining experience by suggesting menu items that take into account not only refrigerator inventory management but also the user's emotional state. Detailed embodiments of this system are described below.

[0187] System Overview

[0188] This system includes a camera installed inside the refrigerator, image analysis, inventory management, automatic e-commerce ordering, recipe suggestion, advertisement notification, and an emotion engine that recognizes the user's emotions. By linking these mechanisms, users can grasp the inventory in their refrigerator in real time and receive efficient shopping and cooking suggestions that match their emotional state.

[0189] Photo taken inside the refrigerator

[0190] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, it takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. These pictures are used to record the types of food in the refrigerator and their arrangement.

[0191] Image data transmission and analysis

[0192] The captured image data is sent from the device (refrigerator camera) to a server via the Internet. The server analyzes the received images and uses an image recognition algorithm to identify the food by category. The analysis results are stored in a database and managed as inventory information.

[0193] Managing and checking inventory information

[0194] Users can check the inventory information in their refrigerator in real time through a dedicated app. The app retrieves the latest inventory information from the server and displays it to provide users with the current inventory status. For example, when a user opens the app, they can immediately see how many items are in the refrigerator.

[0195] Automatic EC ordering of missing items

[0196] Based on inventory information, food items that fall below a set threshold are automatically ordered online. The server periodically checks inventory information, and when the required food items become scarce, it automatically places an order with the registered e-commerce site. For example, if there is less than one bottle of milk, an order is automatically placed.

[0197] Linking recipe suggestions with an emotion engine

[0198] Based on inventory information and the user's emotional state, the server suggests recipes that the user can make. The emotion engine analyzes the user's facial expressions, voice, and bio-signals to identify their emotional state. For example, if it detects that the user is feeling stressed, it will suggest simple recipes that have a relaxing effect. The user can check the suggested recipes on the app and proceed with selecting and preparing the food. For example, if there is chicken and vegetables in the refrigerator and the user is tired, it will suggest a recipe such as "easy stir-fried chicken and vegetables."

[0199] Advertisement Notice

[0200] This system also includes a function to collect information about products advertised in nearby stores and notify users. The server obtains advertising information from nearby stores and compares it with inventory information to identify products that should be notified to users. It then calculates the user's location and the distance to the store, and uses an algorithm to present the most economical option. For example, a LINE notification may be sent to the user saying, "Oranges are on sale at a nearby supermarket. It's a good deal if you go and buy some."

[0201] Example: User B's experience

[0202] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User B opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[0203] Furthermore, the emotion engine detects that B is tired at the time and suggests other easy-to-make recipes. By suggesting easy-to-make menus such as "Easy Tomato and Cheese Pizza," B's burden is reduced. In addition, information about items on sale at nearby supermarkets is sent via LINE notifications, allowing for economical shopping.

[0204] In this way, the present invention, including the emotion engine, provides comprehensive support for the user's daily food management, shopping, and cooking, reducing waste and improving efficiency.

[0205] The processing flow will be explained below.

[0206] Step 1:

[0207] The device (refrigerator camera) takes pictures of the inside of the refrigerator periodically according to a set schedule, for example, at 8:00 AM and 8:00 PM every day.

[0208] Step 2:

[0209] The captured image data is sent from the device (refrigerator camera) to the server via the Internet. Once the transmission is complete, an acknowledgement message is sent.

[0210] Step 3:

[0211] The server saves the received image data in a directory for analysis. If the save is successful, it records that fact in the log.

[0212] Step 4:

[0213] The server then analyzes the stored image data using an image recognition algorithm, which identifies the food items in the image by category and identifies their type and quantity (for example, three apples, two tomatoes, etc.).

[0214] Step 5:

[0215] Based on the analysis results, the server updates the database with inventory information for each food category. The inventory information is updated in real time, ensuring that it is always up to date.

[0216] Step 6:

[0217] The user launches a dedicated app, which sends a request to the server to check stock availability.

[0218] Step 7:

[0219] The server retrieves the latest inventory information from the database and sends it back to the app, which displays the information to the user in real time through the app's UI.

[0220] Step 8:

[0221] The server periodically checks inventory to see if any food items are running low, and if they fall below a threshold, it generates an automatic order request for the item.

[0222] Step 9:

[0223] The server executes the automated order through the API of the e-commerce site, using shipping and payment information based on the user's pre-defined settings.

[0224] Step 10:

[0225] Based on the user's refrigerator inventory and special requests, the app executes an emotion recognition program and sends emotional data to the server.

[0226] Step 11:

[0227] The server selects appropriate recipes based on the user's emotional state. For example, if it detects that the user is feeling stressed, it will select simple, relaxing recipes.

[0228] Step 12:

[0229] The server sends the selected recipe to the app and displays it to the user, who can then review the suggested recipes and refer to cooking instructions as needed.

[0230] Step 13:

[0231] The server collects advertising information from local supermarkets, compares it with current inventory information, and identifies items that need to be purchased.

[0232] Step 14:

[0233] The server takes into account the user's location information, calculates travel distance and economic efficiency based on the displayed advertising information, and determines the most effective shopping method.

[0234] Step 15:

[0235] The server sends LINE notifications to users, advertising specific products and providing them with recommended store information for purchasing them. For example, it can notify users that oranges are on sale at a nearby supermarket, helping them shop in the most economical way.

[0236] Example: User B's experience

[0237] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User B opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[0238] Furthermore, the emotion engine detects that B is tired at the time and suggests other easy-to-make recipes. By suggesting easy-to-make menus such as "Easy Tomato and Cheese Pizza," B's burden is reduced. In addition, information about items on sale at nearby supermarkets is sent via LINE notifications, allowing for efficient shopping.

[0239] Example 2

[0240] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0241] In modern life, efficient refrigerator inventory management and consumption are important challenges for many households. In particular, reducing food waste, replenishing needed ingredients in a timely manner, and providing users with recipes tailored to their needs are difficult. Furthermore, there is a need for systems that can provide meal suggestions based on the user's emotional state and economical shopping options, but no systems exist that can comprehensively address these challenges.

[0242] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a camera means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the camera means and classifying the food by category, an inventory management means for managing inventory information for each food category recognized by the image analysis means, an EC automatic ordering means for automatically ordering missing items, an emotion analysis means for recognizing the emotional state of the user, a recipe suggestion means for suggesting recipes that the user can use based on the inventory management means, and an advertised item notification means for collecting advertised item information from nearby stores and notifying the user. This makes it possible to efficiently manage inventory in the refrigerator and provide personalized meal suggestions and economical shopping options according to the user's emotional state.

[0243] The "camera means" is a device that takes pictures of the inside of the refrigerator.

[0244] The "image analysis means" is a module that analyzes the captured image data and identifies the food by category.

[0245] The "inventory management means" is a system that manages inventory information for each food category recognized by the image analysis means.

[0246] The "EC automatic ordering means" is a module that automatically places online orders for items that are in short supply based on the inventory management means.

[0247] The "recipe suggestion means" is a system that suggests recipes that can be used by the user based on the inventory management means.

[0248] The "advertised item notification means" is a module that collects advertised item information from designated nearby stores and notifies the user of the information.

[0249] An "emotion analysis means" is a system for recognizing a user's emotional state.

[0250] The present invention is a system that not only manages inventory in a refrigerator but also proposes menus that take into account the emotional state of the user, thereby providing a more personalized dining experience. Detailed embodiments of this system are described below.

[0251] System Overview

[0252] This system includes a camera installed inside the refrigerator, image analysis, inventory management, automatic e-commerce ordering, recipe suggestion, advertisement notification, and emotion analysis that recognizes the user's emotions. By linking these means, the user can grasp the inventory in the refrigerator in real time and receive efficient shopping and cooking suggestions that match their emotional state.

[0253] Photo taken inside the refrigerator

[0254] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, it takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. These pictures are used to record the types of food in the refrigerator and their arrangement.

[0255] Image data transmission and analysis

[0256] The captured image data is sent from the device (refrigerator camera) to a server via the Internet. The server analyzes the received images and uses an image recognition algorithm to identify the food by category. This analysis is performed using image recognition software such as TENSORFLOW (registered trademark) or OpenCV. The results of this analysis are stored in a database and managed as inventory information.

[0257] Managing and checking inventory information

[0258] Users can check the inventory information in their refrigerator in real time through a dedicated app. The app retrieves the latest inventory information from the server and displays it to provide users with the current inventory status. For example, when a user opens the app, they can immediately see how many items are in the refrigerator.

[0259] Automatic EC ordering of missing items

[0260] Based on inventory information, food items that fall below a set threshold are automatically ordered online. The server periodically checks inventory information, and when the required food items become scarce, it automatically places an order with the registered e-commerce site (e.g., Amazon or Rakuten Ichiba). For example, an order is automatically placed when there is less than one bottle of milk left.

[0261] Linking recipe suggestions with an emotion engine

[0262] Based on inventory information and the user's emotional state, the server suggests recipes that the user can make. The emotion engine analyzes the user's facial expressions, voice, and bio-signals to identify their emotional state. For example, if it detects that the user is feeling stressed, it will suggest simple recipes that have a relaxing effect. The user can check the suggested recipes on the app and proceed with selecting and preparing the food. For example, if there is chicken and vegetables in the refrigerator and the user is tired, it will suggest a recipe such as "easy stir-fried chicken and vegetables."

[0263] Advertisement Notice

[0264] The server retrieves advertising information from nearby stores and compares it with inventory information to identify products that should be notified to the user. It then calculates the user's location and the distance to the store, and uses an algorithm to present the most economical option. For example, a LINE notification might be sent to the user saying, "Oranges are on sale at a nearby supermarket. It's a good deal if you stop by to buy some."

[0265] Example: User B's experience

[0266] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User B opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[0267] Furthermore, the emotion engine detects that B is tired at the time and suggests other easy-to-make recipes. By suggesting easy-to-make menus such as "Easy Tomato and Cheese Pizza," B's burden is reduced. In addition, information about items on sale at nearby supermarkets is sent via LINE notifications, allowing for economical shopping.

[0268] Example prompts to input to a generative AI model:

[0269] Please provide a detailed explanation of the specific processing steps and operations of a system that manages refrigerator inventory and suggests menu items while also taking into account the user's emotions. Please also mention what hardware and software are used in the process, and what data processing and calculations are performed.

[0270] In this way, the present invention, including the emotion engine, provides comprehensive support for the user's daily food management, shopping, and cooking, reducing waste and improving efficiency.

[0271] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0272] Step 1: Photographing the inside of the refrigerator

[0273] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, take pictures of the inside every day at 8:00 AM and 8:00 PM. The input to this step is the schedule setting information, and the output is the image data of the inside of the refrigerator. The camera operates and automatically records the food in the refrigerator at the specified time.

[0274] Step 2: Sending image data

[0275] The device (refrigerator camera) sends the captured image data to a server via the Internet. The input of this step is the captured image data, and the output is the image data sent to the server. The camera uses a network connection to upload the captured image data to the server.

[0276] Step 3: Receiving image data

[0277] The server receives image data sent via the Internet. The input of this step is the image data sent from the device, and the output is the storage of the received image data. The server receives the image data using a specific API endpoint.

[0278] Step 4: Image analysis

[0279] The server applies image analysis algorithms to the received images to recognize food in the image. The input of this step is the received image data, and the output is the recognized food data. Specifically, the server uses software such as TensorFlow and OpenCV to identify foods such as apples and tomatoes.

[0280] Step 5: Update inventory data

[0281] Based on the analysis results, the server updates the database with the food category and quantity. The input to this step is the recognized food data, and the output is the updated inventory information. For example, if the analysis finds that there are 3 apples and 2 tomatoes, this information is registered in the database.

[0282] Step 6: Get inventory information

[0283] The user opens the dedicated app and sends a request to the server to check the inventory information in the refrigerator. The input of this step is the user's request data, and the output is a request to obtain inventory information. The user operates the app interface to send the request to the server.

[0284] Step 7: Submit inventory information

[0285] The server retrieves the latest inventory information from the database and sends it to the user's app. The input to this step is a request to retrieve inventory information, and the output is the latest inventory information. The server queries the inventory data and sends the retrieved information to the user's app.

[0286] Step 8: View inventory information

[0287] The user checks the inventory information in the app. The input of this step is the inventory information sent from the server, and the output is the displayed inventory information. The user views the inventory information on the app screen.

[0288] Step 9: Check inventory

[0289] The server periodically checks inventory information and identifies foods that have fallen below a set threshold. The input to this step is the inventory information in the database, and the output is a list of items that are in short supply. The server scans the database and extracts the items that are in short supply.

[0290] Step 10: Execute automatic orders

[0291] When the server identifies missing items, it automatically places an order with the registered e-commerce site. The input for this step is a list of missing items, and the output is order confirmation information. The server calls the e-commerce site's API based on the list of missing items and performs the order procedure.

[0292] Step 11: Order confirmation

[0293] The user can check the automatically ordered food and its status on the app. The input of this step is the order confirmation information, and the output is the displayed order confirmation information. The user checks the order status on the app screen.

[0294] Step 12: Obtaining Emotion Data

[0295] The user's emotional state is acquired by a facial expression recognition camera or a biometric signal sensor and transmitted to the server via the device. The input of this step is emotional data, and the output is the emotional data transmitted to the server. The device detects the user's emotional data and transmits it to the server.

[0296] Step 13: Sentiment Analysis

[0297] The server analyzes the received emotion data to identify the user's current emotional state. The input of this step is the received emotion data, and the output is the identified emotional state. The server performs an operation to evaluate the user's emotional state using an emotion analysis algorithm.

[0298] Step 14: Generate recipe suggestions

[0299] The server uses a generative AI model to create an appropriate recipe based on inventory information and the user's emotional state. The inputs for this step are inventory information and the identified emotional state, and the output is a suggested recipe. The server uses the generative AI model to generate a personalized recipe.

[0300] Step 15: Serve the recipe

[0301] The user can view the suggested recipes in the app, select a dish, and prepare it. The input for this step is the suggested recipe, and the output is the displayed recipe information. The user browses the recipe on the app screen and proceeds with cooking.

[0302] Step 16: Collect advertising information

[0303] The server collects advertising information for nearby stores that is publicly available online. The input for this step is advertising information for nearby stores, and the output is the collected advertising data. The server uses scraping technology and APIs to collect advertising information.

[0304] Step 17: Check against inventory information

[0305] The server compares the collected advertising information with inventory information and identifies information that is useful to the user. The input for this step is the collected advertising information and inventory information, and the output is the advertising information to be notified. The server compares the inventory data with the advertising information and identifies the notification target.

[0306] Step 18: Execute Notification

[0307] The server identifies the most suitable store and product based on the user's location information and sends a notification. The input for this step is the advertising information to be notified and the user's location information, and the output is the sent notification information. The server provides useful advertising information to the user via LINE notifications, etc.

[0308] (Application example 2)

[0309] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0310] Today's busy consumers often have little time to manage the food in their refrigerators or purchase food, making it difficult to avoid food waste and irregular eating habits. Furthermore, the lack of a system that can provide appropriate meal suggestions based on users' emotional state is a problem, leading to a decline in the quality of users' diets. Furthermore, there is a need for efficient food management and economical shopping, such as notifications of low inventory in the refrigerator and advertised products at nearby stores.

[0311] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0312] In this invention, the server includes a camera means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the camera means and identifying food categories, an inventory management means for managing inventory information for each food category recognized by the image analysis means, an EC automatic ordering means for automatically ordering missing items based on the inventory management means, a recipe suggestion means for suggesting recipes available to the user based on the inventory management means, an emotion analysis means for analyzing the user's emotional state and suggesting recipes or meal plans based on that state, a food delivery ordering means for automatically placing an order with a food delivery service if ingredients needed for a suggested recipe or meal plan are missing, and an advertised item notification means for collecting advertised item information from designated nearby stores and notifying the user. This allows the user to efficiently manage inventory in the refrigerator, receive meal suggestions based on the user's emotional state, and automatically order necessary ingredients from a delivery service, thereby reducing waste and achieving efficient food management.

[0313] The "camera means" is a camera device for taking pictures of food in the refrigerator.

[0314] The "image analysis means" is a device or software for analyzing captured image data and identifying food items by category.

[0315] The "inventory management means" is a system for managing inventory information for each food category recognized by the image analysis means.

[0316] "EC automatic ordering means" means a device or software for automatically ordering shortage items via the Internet based on the inventory management means.

[0317] The "recipe suggestion means" is a device or software that suggests recipes that can be used by the user based on the inventory management means.

[0318] An "emotion analysis means" is a device or software that analyzes a user's emotional state and suggests recipes or meal plans based on that state.

[0319] "Food delivery ordering device" means a device or software that automatically places an order with a food delivery service when ingredients required for a suggested recipe or meal plan are missing.

[0320] The "advertised item notification means" is a device or software for collecting advertised item information from designated nearby stores and notifying the user of the information.

[0321] The present invention provides a food delivery system that takes into account the emotional state of the user in addition to the inventory management function in the refrigerator. Detailed embodiments of this system will be described below.

[0322] Hardware and software used

[0323] 1. Refrigerator camera means: A camera device for photographing food in the refrigerator.

[0324] 2. Smartphone: A device that allows users to receive emotion recognition and notifications.

[0325] 3. Server:

[0326] Image analysis means: A device that analyzes captured image data and identifies food categories.

[0327] Inventory management method: A system that manages analyzed inventory information.

[0328] Sentiment analysis means: A system that analyzes the user's emotional state and suggests recipes and meal plans based on that state.

[0329] Generative AI model: A model that generates recipes based on the user's emotional state and inventory information.

[0330] Food delivery ordering method: A system that automatically orders the necessary ingredients from a delivery service.

[0331] Advertised item notification means: A device that collects advertised item information from designated nearby stores and notifies the user.

[0332] System processing flow

[0333] The overall flow of the system is as follows: First, the camera means inside the refrigerator periodically takes pictures of the inside of the refrigerator. The captured image data is sent to a server via the Internet, where the image analysis means analyzes it and identifies the type and quantity of food stored inside. This data is then stored in the inventory management means.

[0334] When a user launches the application on their smartphone, the emotion analyzer captures and analyzes the user's emotional state using the camera and microphone. For example, if the user is feeling stressed, the emotion analyzer sends a prompt to the generative AI model to generate a "relaxing meal suitable for a stressed user."

[0335] The generative AI model generates an optimal recipe based on the prompts and the inventory information in the refrigerator. The recipe suggestion means suggests this recipe to the user, and if necessary ingredients are missing, the food delivery ordering means automatically places an order with a delivery service.

[0336] Users can review these suggestions and view inventory information and suggested recipes at any time. The advertising notification tool also collects advertising information from nearby stores, providing users with the latest economical options.

[0337] Specific examples

[0338] For example, a camera means installed inside a refrigerator takes an image of the inside of the refrigerator at 8:00 AM and sends it to a server. The server analyzes the image and recognizes that there are two apples and one piece of chicken inside. After that, when a user launches the app on their smartphone, the emotion analysis means analyzes the user's emotional state. If the user is feeling stressed, the generative AI model will receive the following prompt:

[0339] "Please suggest a meal that will help the user relax when they are feeling stressed. Below is the inventory in the refrigerator: 2 apples, 1 piece of chicken."

[0340] Based on this prompt, the generative AI model suggests recipes such as "Relaxed Apple and Chicken Salad." If other ingredients required for this recipe are missing, the food delivery ordering tool automatically places an order with the delivery service.

[0341] In this way, users can manage their refrigerator inventory, receive personalized meal suggestions based on their emotional state, and even have missing ingredients automatically ordered from a delivery service.

[0342] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0343] Step 1:

[0344] Photo taken inside the refrigerator

[0345] Input: Food status in the refrigerator, shooting schedule

[0346] Processing: The camera means installed in the terminal periodically takes pictures of the inside of the refrigerator. For example, take pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM.

[0347] Output: Image data of the inside of the refrigerator

[0348] Step 2:

[0349] Sending image data

[0350] Input: Image data of the inside of the refrigerator

[0351] Processing: The device sends the captured image data to a server via the Internet.

[0352] Output: Image data sent to the server

[0353] Step 3:

[0354] Image data analysis

[0355] Input: Submitted image data

[0356] Processing: The image analysis means installed on the server analyzes the transmitted image data and identifies each food item by category. For example, the food items are classified as apples, chicken, etc.

[0357] Output: Inventory data identified for each food category

[0358] Step 4:

[0359] Inventory management

[0360] Input: Identified inventory data

[0361] Processing: The inventory management means of the server stores the identified inventory data in a database and manages it as inventory information.

[0362] Output: Updated inventory information

[0363] Step 5:

[0364] User emotional state analysis

[0365] Input: User's facial expressions, voice, and biometric signals

[0366] Processing: When a user launches the app on their smartphone, the device's camera and microphone capture the user's emotional state and send it to the server, where the server's emotion analysis means analyzes it and generates the user's emotional state data.

[0367] Output: Emotional state data

[0368] Step 6:

[0369] Recipe generation and suggestions

[0370] Input: inventory information, emotional state data

[0371] Processing: The server's AI model generates a recipe suitable for the user based on inventory information and the user's emotional state data. For example, a recipe may be generated based on a prompt such as "a meal that will have a relaxing effect when the user is feeling stressed." The generated recipe is then suggested to the user through the recipe suggestion mechanism.

[0372] Output: Suggested recipe

[0373] Step 7:

[0374] Automatic ordering of missing ingredients

[0375] Input: Suggested recipes, inventory information

[0376] Processing: If there is a shortage of ingredients in stock based on the recipe suggested by the recipe suggestion means, the food delivery ordering means of the server automatically orders the necessary ingredients via a delivery service.

[0377] Output: Order confirmation notification

[0378] Step 8:

[0379] Advertisement notification

[0380] Input: Updated inventory information, user location information

[0381] Processing: The server's advertisement notification means collects advertisement information from the set nearby stores and transmits advertisement information to the user showing the most economical options based on the user's location information.

[0382] Output: Advertisement notification

[0383] Step 9:

[0384] User confirmation

[0385] Input: Suggested recipes, order confirmation notifications for missing ingredients, promotional product notifications

[0386] Processing: The user checks the suggested recipes, order confirmation notifications, and promotional product notifications on their smartphone, allowing them to stay informed of suggested dishes, purchased ingredients, and special promotional products.

[0387] Output: User verification data

[0388] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0389] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0390] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0391] [Second embodiment]

[0392] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0393] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0394] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0395] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0396] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0397] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0398] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0399] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0400] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0401] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0402] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0403] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0404] The present invention relates to a system for efficiently managing inventory in a refrigerator, allowing users to achieve optimal shopping and cooking. Detailed embodiments of the system will be described below.

[0405] System Overview

[0406] This system includes a camera installed inside the refrigerator, image analysis, inventory management, automatic e-commerce ordering, recipe suggestion, and advertisement notification. By linking these means, users can grasp the food inventory in their refrigerator in real time and receive suggestions for efficient shopping and cooking.

[0407] Photo taken inside the refrigerator

[0408] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, it takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. These pictures are used to record the types of food in the refrigerator and their arrangement.

[0409] Image data transmission and analysis

[0410] The captured image data is sent from the device (refrigerator camera) to a server via the Internet. The server analyzes the received images and uses an image recognition algorithm to identify the food by category. The analysis results are stored in a database and managed as inventory information.

[0411] Managing and checking inventory information

[0412] Users can check the inventory information in their refrigerator in real time through a dedicated app. The app retrieves the latest inventory information from the server and displays it to provide users with the current inventory status. For example, when a user opens the app, they can immediately see how many items are in the refrigerator.

[0413] Automatic EC ordering of missing items

[0414] Based on inventory information, food items that fall below a set threshold are automatically ordered online. The server periodically checks inventory information, and when the required food items become scarce, it automatically places an order with the registered e-commerce site. For example, if there is less than one bottle of milk, an order is automatically placed.

[0415] Recipe suggestions

[0416] Based on the inventory information, the server suggests recipes that can be made by the user. The server analyzes the inventory data, selects appropriate recipes, and sends them to the app. The user can check the suggested recipes on the app and start cooking right away. For example, if there is chicken and vegetables in the refrigerator, a recipe such as "stir-fried chicken and vegetables" will be suggested.

[0417] Advertisement Notice

[0418] The system also includes a function to collect advertised product information from selected nearby stores and notify users. The server obtains advertising information from local supermarkets and compares it with inventory information to identify products that users should be notified about. It then calculates the user's location and the distance to the store, and uses an algorithm to present the most economical option. For example, a LINE notification might be sent to the user saying, "Oranges are on sale at a nearby supermarket. It's a good deal if you go and buy some."

[0419] Example: User A's experience

[0420] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User A opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[0421] In the evening, Mr. A notices that there is little yogurt in the refrigerator, but the system has already automatically ordered it and notifies him that it will be delivered. Furthermore, he receives a LINE notification that a nearby supermarket is having a special sale on oranges, so he heads to the supermarket and does his shopping efficiently.

[0422] In this way, the present invention provides comprehensive support for the user's daily food management, shopping, and cooking, reducing waste and improving efficiency.

[0423] The processing flow will be explained below.

[0424] Step 1:

[0425] The device (refrigerator camera) takes pictures of the entire interior of the refrigerator according to a set schedule, for example, at 8:00 AM and 8:00 PM every day. These pictures are taken with a wide angle to capture the entire interior of the refrigerator.

[0426] Step 2:

[0427] The captured image data is compressed from the device (refrigerator camera) and sent to a server via the Internet, along with an acknowledgement message to confirm successful data transmission.

[0428] Step 3:

[0429] The server saves the received image data in a directory for analysis. If the save is successful, it records that fact in the log.

[0430] Step 4:

[0431] The server then analyzes the stored image data using image recognition algorithms, which identify the food items in the image by category and determine their type and quantity (for example, three apples, two tomatoes, etc.).

[0432] Step 5:

[0433] Based on the analysis results, the server updates the database with inventory information for each food category. The inventory information is updated in real time, ensuring that it is always up to date.

[0434] Step 6:

[0435] The user launches a dedicated app, which sends a request to the server to check stock availability.

[0436] Step 7:

[0437] The server retrieves the latest inventory information from the database and sends it back to the app, which displays the information to the user in real time through the app's UI.

[0438] Step 8:

[0439] The server periodically checks inventory to see if any food items are running low, and if they fall below a threshold, it generates an automatic order request for the item.

[0440] Step 9:

[0441] The server executes the automated order through the API of the e-commerce site, using shipping and payment information based on the user's pre-defined settings.

[0442] Step 10:

[0443] The server suggests recipes that can be made by the user based on inventory information, selects appropriate recipes, and sends the suggestions to the app.

[0444] Step 11:

[0445] Users can check the suggested recipes on the app, refer to the cooking instructions if necessary, and start cooking according to the recipe they select.

[0446] Step 12:

[0447] The server collects advertising information from local supermarkets and compares it with current inventory information to identify items that need to be purchased.

[0448] Step 13:

[0449] The server takes into account the user's location information, calculates travel distance and economic efficiency based on the displayed advertising information, and determines the most effective shopping method.

[0450] Step 14:

[0451] The server will provide users with advertisements for specific products and recommended store information for purchasing those products via LINE notifications, allowing users to shop in the most economical way.

[0452] Example 1

[0453] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0454] Conventional refrigerator inventory management requires manual effort and time from the user. There is also a high possibility of food shortages or waste due to duplicate purchases, making efficient food management difficult. Furthermore, there is no way to check inventory information in real time, making it inconvenient to be unable to check the inventory status of the refrigerator while shopping on the go. A new system is needed to solve these problems.

[0455] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0456] In this invention, the server includes an imaging means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the imaging means and classifying food items by category, a data management means for managing inventory information for each food category recognized by the image analysis means, an automatic e-commerce ordering means for automatically ordering missing items based on the data management means, a suggestion means for suggesting recipes that the user can use based on the data management means, an advertisement information notification means for collecting advertisement information from designated nearby stores and notifying the user, and a cloud data synchronization means for saving the inventory information saved in the data management means to the cloud and enabling the information to be checked in real time. This allows the user to efficiently manage their refrigerator inventory, prevent stockouts, and purchase and cook optimal ingredients.

[0457] "Imaging means" refers to a device for capturing images of the inside of the refrigerator, and includes cameras and other imaging devices for recording the types and arrangement of food items inside the refrigerator.

[0458] The "image analysis means" is a computer algorithm or program for analyzing image data acquired by the imaging means and identifying food items by category.

[0459] The "data management means" is a system for storing and managing inventory information for each food category recognized by the image analysis means, and includes a database and cloud storage.

[0460] "Automatic e-commerce ordering means" refers to a system that has the function of automatically ordering shortage items online based on data management means, and includes linkage with an e-commerce platform.

[0461] The "suggestion means" is a system that has the function of suggesting recipes that can be used by the user based on the data management means, and includes an algorithm that works in conjunction with the recipe database.

[0462] The "advertising information notification means" is a system that has the function of collecting advertising information from designated nearby stores and notifying the user, and includes an algorithm that works in conjunction with the user's location information.

[0463] The "cloud data synchronization means" is a system that stores inventory information stored in the data management means in the cloud, enabling information to be checked in real time.

[0464] The present invention relates to a system that efficiently manages food inventory in a refrigerator, enabling users to optimize their shopping and cooking. The system includes an imaging unit, an image analysis unit, a data management unit, an automatic e-commerce ordering unit, a suggestion unit, a promotional information notification unit, and a cloud data synchronization unit, all installed in the refrigerator.

[0465] Hardware or software configuration

[0466] Imaging means

[0467] The terminal (refrigerator camera) is a device for taking high-resolution images, and periodically takes pictures of the inside of the refrigerator, for example, every 24 hours. This camera has wireless communication capabilities and sends image data to a server via the Internet.

[0468] Image analysis methods

[0469] The server uses a high-performance image recognition algorithm (e.g., YOLO, ResNet) to analyze the received image data. This algorithm classifies food by category and identifies the food based on the extracted features.

[0470] Data Management Measures

[0471] The server stores the identified food information in a database, which includes information such as the food ingredient classification, quantity, and location, and manages it so that users can easily check the inventory status.

[0472] E-commerce automatic ordering tool

[0473] The server has the ability to automatically order food items online based on data management methods. For example, if there is less than one bottle of milk, a new order is placed via a pre-registered e-commerce site API.

[0474] Proposal means

[0475] The server proposes recipes that the user can cook based on the data management method. The server analyzes inventory data, selects appropriate recipes, and sends them to the app. The user can check the proposed recipes on the app and start cooking right away.

[0476] Promotional information notification means

[0477] The server collects promotional information from nearby stores and notifies the user. This function presents the most economical option based on the user's location and the distance to the store. For example, a LINE notification can be sent to the user to let them know about special sales at a nearby supermarket.

[0478] Cloud data synchronization method

[0479] The server stores inventory information on the cloud and updates it in real time, allowing users to check the inventory status of their refrigerators even when they are out and about.

[0480] Specific examples

[0481] User A's experience

[0482] A camera installed in the refrigerator takes an image of the inside of the refrigerator at 8:00 AM and sends it to a server. The server analyzes the image and determines that there are three apples and two tomatoes. User A opens a dedicated app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "apple and tomato salad."

[0483] In the evening, User A notices that there is little yogurt in the refrigerator, but is notified that new yogurt will arrive, as the system has already automatically placed an order. Furthermore, User A receives a LINE notification that a nearby supermarket is having a special sale on oranges, allowing User A to go to the supermarket and complete their shopping efficiently.

[0484] Example prompts to input to the generative AI model

[0485] "A camera installed inside the refrigerator takes pictures of the inside at 8:00 AM and sends them to a server. The server then analyzes the images and displays inventory information on an app. With this app, users can check the inventory in their refrigerator in real time and receive recipe suggestions, automatic ordering, and notifications of advertised products. Please explain this system."

[0486] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0487] Step 1: Photographing the inside of the refrigerator using imaging means

[0488] The device (refrigerator camera) takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. This camera is positioned to cover the entire inside of the refrigerator and captures high-resolution images. The input is the image data of the inside of the refrigerator taken by the camera, and the output is the image file. In this step, the types and arrangement of food in the refrigerator are recorded in detail.

[0489] Step 2: Sending image data

[0490] The device (refrigerator camera) sends the captured image data to a server via the Internet. Here, the input is the captured image data, and the output is the image data sent to the server. This step requires highly reliable data transmission, and data is transferred using protocols such as HTTP.

[0491] Step 3: Image analysis

[0492] The server analyzes the received image data using an image analysis algorithm (e.g., YOLO, ResNet). The input is the image data sent to the server, and the output is food data identified by category. In this step, food items in the image are identified by category and features such as size, shape, and color are extracted. For example, the server accurately recognizes foods such as apples, tomatoes, and milk from the image and registers them in a database.

[0493] Step 4: Update data management

[0494] The server stores the identified food information in a database and updates the inventory information. The input is the food identified by category, and the output is the updated database. In this step, information such as food classification, quantity, and location is recorded in the database and a timestamp is added to allow tracking of inventory fluctuations over time.

[0495] Step 5: Check inventory information

[0496] The user checks the inventory information in the refrigerator in real time through a dedicated app. The app periodically connects to the server to retrieve and display the latest inventory information. The input is the inventory information retrieved from the server, and the output is the inventory information displayed to the user. When the user opens the app, specific inventory information, such as "3 apples, 2 tomatoes," is immediately displayed.

[0497] Step 6: Automatically order missing items online

[0498] The server periodically checks the inventory information in the database and automatically places an order online for any food items that are running low. The input is the updated inventory data, and the output is the order data sent to the e-commerce platform. For example, if there is only one bottle of milk left, the server sends the necessary information to the e-commerce site API to place an automatic order.

[0499] Step 7: Recipe suggestions

[0500] The server suggests recipes that can be made by the user based on inventory information. The input is inventory information in the database, and the output is suggested recipe information. The server analyzes the inventory data, selects appropriate recipes from the database, and sends them to the app. The user can then view specific recipe suggestions, such as "apple and tomato salad," on the app.

[0501] Step 8: Promotional Information Notification

[0502] The server collects advertising information from nearby stores and notifies the user. The input is the collected advertising information, and the output is the advertising information notified to the user. The server obtains advertising data from local supermarkets and compares it with inventory information to identify products to notify the user about. Furthermore, the server calculates the travel distance to the optimal store based on the user's location information, and sends the user, for example, information about special sales at nearby supermarkets via LINE notifications.

[0503] (Application example 1)

[0504] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0505] Conventional refrigerator inventory management systems are limited to efficient inventory management and shopping optimization within the home, and have not been applied to inventory management or product recommendation efficiency in physical stores. In addition, there is a lack of a system for store clerks to check inventory information in real time and make optimal product recommendations to customers, so improving inventory management efficiency and customer satisfaction are issues. Furthermore, there is a need for a means to increase customer purchasing motivation by appropriately notifying customers of advertising information and promotions.

[0506] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0507] In this invention, the server includes an imaging means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the imaging means and classifying the food by category, an inventory management means for managing inventory information for each food category recognized by the image analysis means, an automatic e-commerce ordering means for automatically ordering missing items based on the inventory management means, a recipe suggestion means for suggesting recipes available to the user based on the inventory management means, an advertised item notification means for collecting advertised item information from selected nearby stores and notifying the user, a visualization means for displaying in-store inventory information in real time on a display device worn by the user and checking inventory and suggesting products, an automatic replenishment means for automatically replenishing in-store inventory when inventory falls below a certain threshold, and a notification means for notifying the display device of advertising information and promotional information. This improves the efficiency of inventory management in physical stores, enables store staff to suggest optimal products to customers in real time, and improves customer satisfaction by effectively notifying them of advertising information and promotional information.

[0508] The "imaging means" is a device that takes images of the inside of a refrigerator or a store.

[0509] The "image analysis means" is a device or software that has the function of analyzing captured image data and identifying food or product categories.

[0510] The "inventory management means" is a device or software that has the function of managing inventory information recognized by the image analysis means and updating it in real time.

[0511] An "automated electronic commerce ordering means" is a device or software that has the function of automatically ordering products that are in short supply based on inventory information.

[0512] The "recipe suggestion means" is a device or software that has the function of suggesting recipes that can be used by the user based on inventory information.

[0513] The "advertised item notification means" is a device or software that has the function of notifying the user of advertised item information collected from designated nearby stores.

[0514] A "display device" is a display device worn by a user to visually confirm information.

[0515] The "visualization means" is a device or software that displays inventory information in real time through a display device and has the function of checking inventory and making product suggestions.

[0516] An "automatic replenishment means" is a device or software that has the function of automatically replenishing products when inventory falls below a certain threshold.

[0517] The "notification means" is a device or software that has the function of notifying the display device of advertising information or promotional information.

[0518] The present invention relates to a system for improving the efficiency of inventory management and product recommendations in a physical store. Detailed embodiments of the present invention will be described below.

[0519] System Overview

[0520] This system is comprised of an imaging means, an image analysis means, an inventory management means, an automatic e-commerce ordering means, a recipe suggestion means, an advertised product notification means, a visualization means, an automatic replenishment means, and a notification means. By linking these means, efficient inventory management, product suggestions, and advertising information notifications in physical stores become possible.

[0521] Photography and image analysis

[0522] The server first periodically captures images of the product shelves in the store using an imaging device. This capture is performed automatically at regular intervals. The captured image data is sent to the server and analyzed by the image analysis device. The image analysis device then classifies the food and product by category and stores the data in a database as inventory information.

[0523] Inventory management and e-commerce automatic ordering

[0524] The server updates the inventory information recognized by the image analysis means in real time using the inventory management means. The user can wear a display device and instantly check the current inventory information using the visualization means. Based on this inventory information, if the inventory falls below a certain threshold, the e-commerce automatic ordering means automatically places a replenishment order.

[0525] Recipe suggestions and promotional product notifications

[0526] The server is equipped with a recipe suggestion unit that suggests recipes available to the user based on inventory information. Furthermore, an advertised product notification unit notifies the user of advertised product information collected from designated nearby stores. The server calculates the user's location information and the travel distance to the store, and uses an algorithm to present the most economical options.

[0527] In-store use

[0528] Store staff wear display devices and use visualization tools to check inventory information in real time. For example, if an item on a shelf runs low, an automatic replenishment tool is activated to immediately replenish it. It is also possible to check whether the product desired by the customer is in stock and present alternative options.

[0529] Notification means

[0530] The advertising information and promotion information are displayed on the display device using the notification means, allowing the store staff to provide the customer with appropriate product information.

[0531] Examples of concrete examples and prompts

[0532] For example, when a store clerk wears a display device and activates the "Check Stock" command, real-time inventory information is displayed on the display. For example, it may show "Milk: 10, Eggs: 5, Bread: 2." Also, when the "Auto Order" command is executed, inventory below a threshold (for example, bread) is automatically ordered.

[0533] Below are some example prompts for a generative AI model:

[0534] Can you give us an example of a system that streamlines refrigerator inventory management? A system that meets the following requirements: 1. Periodically takes pictures of the inside of the refrigerator using a camera. 2. Analyzes the images and updates inventory information. 3. Checks inventory information in real time using a smartphone or smart glasses. 4. Automatically places e-commerce orders when inventory is low. 5. Suggests recipes that can be made. 6. Notifies users of advertised product information.

[0535] The above is the details of the embodiment of the invention. This system is expected to improve the efficiency of inventory management in stores and increase customer satisfaction.

[0536] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0537] Step 1:

[0538] The server periodically captures images of the product shelves in the store using an imaging device. It receives the time interval and camera position as input and obtains the captured image data as output. The images captured by this imaging device are saved in a storage directory.

[0539] Step 2:

[0540] The server sends the captured image data to the image analysis means, which receives the image data as input and outputs the analysis results. The image analysis means uses machine learning algorithms to identify each item in the image by category and stores the results in a database.

[0541] Step 3:

[0542] The server retrieves inventory information from the database and manages the latest inventory information using inventory management means. It receives analysis result data as input and saves updated inventory information in the database as output. This is done in real time by the inventory management means.

[0543] Step 4:

[0544] The server uses visualization means to display the latest inventory information via a display device worn by the user. The latest inventory information is acquired as input, and the inventory information displayed on the display is obtained as output. This allows the user to check the current inventory status.

[0545] Step 5:

[0546] When inventory information falls below a certain threshold, the server automatically orders the missing items using an automatic e-commerce ordering system. The server receives inventory information and a set threshold as input, and obtains data related to the automatic order as output. This allows for a rapid response when inventory shortages occur.

[0547] Step 6:

[0548] The server activates a recipe suggestion means that suggests available recipes to the user based on the inventory information. The server receives the inventory information as input and generates suggested recipe information as output. The user can check these suggested recipes on the display device.

[0549] Step 7:

[0550] Using the advertising product notification means, the server notifies the user of advertising information collected from the set nearby stores. It receives advertising information and the user's location information as input and generates an optimal advertising notification as output. The user can learn about economical options through the display device.

[0551] Step 8:

[0552] The advertising information and promotion information are displayed on the display device by the notification means. The advertising information is used as input and the notification on the display is used as output. This allows the user and store clerk to instantly check the latest promotion information.

[0553] Step 9:

[0554] The user checks inventory and makes product suggestions via a display device, and replenishes products or makes suggestions to customers based on the information displayed on the display.The system receives instructions from the display device as input and generates specific product suggestions and replenishment actions as output.

[0555] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0556] The present invention is a system that provides a more personalized dining experience by suggesting menu items that take into account not only refrigerator inventory management but also the user's emotional state. Detailed embodiments of this system are described below.

[0557] System Overview

[0558] This system includes a camera installed inside the refrigerator, image analysis, inventory management, automatic e-commerce ordering, recipe suggestion, advertisement notification, and an emotion engine that recognizes the user's emotions. By linking these mechanisms, users can grasp the inventory in their refrigerator in real time and receive efficient shopping and cooking suggestions that match their emotional state.

[0559] Photo taken inside the refrigerator

[0560] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, it takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. These pictures are used to record the types of food in the refrigerator and their arrangement.

[0561] Image data transmission and analysis

[0562] The captured image data is sent from the device (refrigerator camera) to a server via the Internet. The server analyzes the received images and uses an image recognition algorithm to identify the food by category. The analysis results are stored in a database and managed as inventory information.

[0563] Managing and checking inventory information

[0564] Users can check the inventory information in their refrigerator in real time through a dedicated app. The app retrieves the latest inventory information from the server and displays it to provide users with the current inventory status. For example, when a user opens the app, they can immediately see how many items are in the refrigerator.

[0565] Automatic EC ordering of missing items

[0566] Based on inventory information, food items that fall below a set threshold are automatically ordered online. The server periodically checks inventory information, and when the required food items become scarce, it automatically places an order with the registered e-commerce site. For example, if there is less than one bottle of milk, an order is automatically placed.

[0567] Linking recipe suggestions with an emotion engine

[0568] Based on inventory information and the user's emotional state, the server suggests recipes that the user can make. The emotion engine analyzes the user's facial expressions, voice, and bio-signals to identify their emotional state. For example, if it detects that the user is feeling stressed, it will suggest simple recipes that have a relaxing effect. The user can check the suggested recipes on the app and proceed with selecting and preparing the food. For example, if there is chicken and vegetables in the refrigerator and the user is tired, it will suggest a recipe such as "easy stir-fried chicken and vegetables."

[0569] Advertisement Notice

[0570] This system also includes a function to collect information about products advertised in nearby stores and notify users. The server obtains advertising information from nearby stores and compares it with inventory information to identify products that should be notified to users. It then calculates the user's location and the distance to the store, and uses an algorithm to present the most economical option. For example, a LINE notification may be sent to the user saying, "Oranges are on sale at a nearby supermarket. It's a good deal if you go and buy some."

[0571] Example: User B's experience

[0572] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User B opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[0573] Furthermore, the emotion engine detects that B is tired at the time and suggests other easy-to-make recipes. By suggesting easy-to-make menus such as "Easy Tomato and Cheese Pizza," B's burden is reduced. In addition, information about items on sale at nearby supermarkets is sent via LINE notifications, allowing for economical shopping.

[0574] In this way, the present invention, including the emotion engine, provides comprehensive support for the user's daily food management, shopping, and cooking, reducing waste and improving efficiency.

[0575] The processing flow will be explained below.

[0576] Step 1:

[0577] The device (refrigerator camera) takes pictures of the inside of the refrigerator periodically according to a set schedule, for example, at 8:00 AM and 8:00 PM every day.

[0578] Step 2:

[0579] The captured image data is sent from the device (refrigerator camera) to the server via the Internet. Once the transmission is complete, an acknowledgement message is sent.

[0580] Step 3:

[0581] The server saves the received image data in a directory for analysis. If the save is successful, it records that fact in the log.

[0582] Step 4:

[0583] The server then analyzes the stored image data using an image recognition algorithm, which identifies the food items in the image by category and identifies their type and quantity (for example, three apples, two tomatoes, etc.).

[0584] Step 5:

[0585] Based on the analysis results, the server updates the database with inventory information for each food category. The inventory information is updated in real time, ensuring that it is always up to date.

[0586] Step 6:

[0587] The user launches a dedicated app, which sends a request to the server to check stock availability.

[0588] Step 7:

[0589] The server retrieves the latest inventory information from the database and sends it back to the app, which displays the information to the user in real time through the app's UI.

[0590] Step 8:

[0591] The server periodically checks inventory to see if any food items are running low, and if they fall below a threshold, it generates an automatic order request for the item.

[0592] Step 9:

[0593] The server executes the automated order through the API of the e-commerce site, using shipping and payment information based on the user's pre-defined settings.

[0594] Step 10:

[0595] Based on the user's refrigerator inventory and special requests, the app executes an emotion recognition program and sends emotional data to the server.

[0596] Step 11:

[0597] The server selects appropriate recipes based on the user's emotional state. For example, if it detects that the user is feeling stressed, it will select simple, relaxing recipes.

[0598] Step 12:

[0599] The server sends the selected recipe to the app and displays it to the user, who can then review the suggested recipes and refer to cooking instructions as needed.

[0600] Step 13:

[0601] The server collects advertising information from local supermarkets, compares it with current inventory information, and identifies items that need to be purchased.

[0602] Step 14:

[0603] The server takes into account the user's location information, calculates travel distance and economic efficiency based on the displayed advertising information, and determines the most effective shopping method.

[0604] Step 15:

[0605] The server sends LINE notifications to users, advertising specific products and providing them with recommended store information for purchasing them. For example, it can notify users that oranges are on sale at a nearby supermarket, helping them shop in the most economical way.

[0606] Example: User B's experience

[0607] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User B opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[0608] Furthermore, the emotion engine detects that B is tired at the time and suggests other easy-to-make recipes. By suggesting easy-to-make menus such as "Easy Tomato and Cheese Pizza," B's burden is reduced. In addition, information about items on sale at nearby supermarkets is sent via LINE notifications, allowing for efficient shopping.

[0609] Example 2

[0610] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0611] In modern life, efficient refrigerator inventory management and consumption are important challenges for many households. In particular, reducing food waste, replenishing needed ingredients in a timely manner, and providing users with recipes tailored to their needs are difficult. Furthermore, there is a need for systems that can provide meal suggestions based on the user's emotional state and economical shopping options, but no systems exist that can comprehensively address these challenges.

[0612] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a camera means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the camera means and classifying the food by category, an inventory management means for managing inventory information for each food category recognized by the image analysis means, an EC automatic ordering means for automatically ordering missing items, an emotion analysis means for recognizing the emotional state of the user, a recipe suggestion means for suggesting recipes that the user can use based on the inventory management means, and an advertised item notification means for collecting advertised item information from nearby stores and notifying the user. This makes it possible to efficiently manage inventory in the refrigerator and provide personalized meal suggestions and economical shopping options according to the user's emotional state.

[0613] The "camera means" is a device that takes pictures of the inside of the refrigerator.

[0614] The "image analysis means" is a module that analyzes the captured image data and identifies the food by category.

[0615] The "inventory management means" is a system that manages inventory information for each food category recognized by the image analysis means.

[0616] The "EC automatic ordering means" is a module that automatically places online orders for items that are in short supply based on the inventory management means.

[0617] The "recipe suggestion means" is a system that suggests recipes that can be used by the user based on the inventory management means.

[0618] The "advertised item notification means" is a module that collects advertised item information from designated nearby stores and notifies the user of the information.

[0619] An "emotion analysis means" is a system for recognizing a user's emotional state.

[0620] The present invention is a system that not only manages inventory in a refrigerator but also proposes menus that take into account the emotional state of the user, thereby providing a more personalized dining experience. Detailed embodiments of this system are described below.

[0621] System Overview

[0622] This system includes a camera installed inside the refrigerator, image analysis, inventory management, automatic e-commerce ordering, recipe suggestion, advertisement notification, and emotion analysis that recognizes the user's emotions. By linking these means, the user can grasp the inventory in the refrigerator in real time and receive efficient shopping and cooking suggestions that match their emotional state.

[0623] Photo taken inside the refrigerator

[0624] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, it takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. These pictures are used to record the types of food in the refrigerator and their arrangement.

[0625] Image data transmission and analysis

[0626] The captured image data is sent from the device (refrigerator camera) to a server via the Internet. The server analyzes the received images and uses an image recognition algorithm to identify the food by category. This analysis is performed using image recognition software such as TensorFlow and OpenCV. The results of this analysis are stored in a database and managed as inventory information.

[0627] Managing and checking inventory information

[0628] Users can check the inventory information in their refrigerator in real time through a dedicated app. The app retrieves the latest inventory information from the server and displays it to provide users with the current inventory status. For example, when a user opens the app, they can immediately see how many items are in the refrigerator.

[0629] Automatic EC ordering of missing items

[0630] Based on inventory information, food items that fall below a set threshold are automatically ordered online. The server periodically checks inventory information, and when the required food items become scarce, it automatically places an order with the registered e-commerce site (e.g., Amazon or Rakuten Ichiba). For example, an order is automatically placed when there is less than one bottle of milk left.

[0631] Linking recipe suggestions with an emotion engine

[0632] Based on inventory information and the user's emotional state, the server suggests recipes that the user can make. The emotion engine analyzes the user's facial expressions, voice, and bio-signals to identify their emotional state. For example, if it detects that the user is feeling stressed, it will suggest simple recipes that have a relaxing effect. The user can check the suggested recipes on the app and proceed with selecting and preparing the food. For example, if there is chicken and vegetables in the refrigerator and the user is tired, it will suggest a recipe such as "easy stir-fried chicken and vegetables."

[0633] Advertisement Notice

[0634] The server retrieves advertising information from nearby stores and compares it with inventory information to identify products that should be notified to the user. It then calculates the user's location and the distance to the store, and uses an algorithm to present the most economical option. For example, a LINE notification might be sent to the user saying, "Oranges are on sale at a nearby supermarket. It's a good deal if you stop by to buy some."

[0635] Example: User B's experience

[0636] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User B opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[0637] Furthermore, the emotion engine detects that B is tired at the time and suggests other easy-to-make recipes. By suggesting easy-to-make menus such as "Easy Tomato and Cheese Pizza," B's burden is reduced. In addition, information about items on sale at nearby supermarkets is sent via LINE notifications, allowing for economical shopping.

[0638] Example prompts to input to a generative AI model:

[0639] Please provide a detailed explanation of the specific processing steps and operations of a system that manages refrigerator inventory and suggests menu items while also taking into account the user's emotions. Please also mention what hardware and software are used in the process, and what data processing and calculations are performed.

[0640] In this way, the present invention, including the emotion engine, provides comprehensive support for the user's daily food management, shopping, and cooking, reducing waste and improving efficiency.

[0641] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0642] Step 1: Photographing the inside of the refrigerator

[0643] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, take pictures of the inside every day at 8:00 AM and 8:00 PM. The input to this step is the schedule setting information, and the output is the image data of the inside of the refrigerator. The camera operates and automatically records the food in the refrigerator at the specified time.

[0644] Step 2: Sending image data

[0645] The device (refrigerator camera) sends the captured image data to a server via the Internet. The input of this step is the captured image data, and the output is the image data sent to the server. The camera uses a network connection to upload the captured image data to the server.

[0646] Step 3: Receiving image data

[0647] The server receives image data sent via the Internet. The input of this step is the image data sent from the device, and the output is the storage of the received image data. The server receives the image data using a specific API endpoint.

[0648] Step 4: Image analysis

[0649] The server applies image analysis algorithms to the received images to recognize food in the image. The input of this step is the received image data, and the output is the recognized food data. Specifically, the server uses software such as TensorFlow and OpenCV to identify foods such as apples and tomatoes.

[0650] Step 5: Update inventory data

[0651] Based on the analysis results, the server updates the database with the food category and quantity. The input to this step is the recognized food data, and the output is the updated inventory information. For example, if the analysis finds that there are 3 apples and 2 tomatoes, this information is registered in the database.

[0652] Step 6: Get inventory information

[0653] The user opens the dedicated app and sends a request to the server to check the inventory information in the refrigerator. The input of this step is the user's request data, and the output is a request to obtain inventory information. The user operates the app interface to send the request to the server.

[0654] Step 7: Submit inventory information

[0655] The server retrieves the latest inventory information from the database and sends it to the user's app. The input to this step is a request to retrieve inventory information, and the output is the latest inventory information. The server queries the inventory data and sends the retrieved information to the user's app.

[0656] Step 8: View inventory information

[0657] The user checks the inventory information in the app. The input of this step is the inventory information sent from the server, and the output is the displayed inventory information. The user views the inventory information on the app screen.

[0658] Step 9: Check inventory

[0659] The server periodically checks inventory information and identifies foods that have fallen below a set threshold. The input to this step is the inventory information in the database, and the output is a list of items that are in short supply. The server scans the database and extracts the items that are in short supply.

[0660] Step 10: Execute automatic orders

[0661] When the server identifies missing items, it automatically places an order with the registered e-commerce site. The input for this step is a list of missing items, and the output is order confirmation information. The server calls the e-commerce site's API based on the list of missing items and performs the order procedure.

[0662] Step 11: Order confirmation

[0663] The user can check the automatically ordered food and its status on the app. The input of this step is the order confirmation information, and the output is the displayed order confirmation information. The user checks the order status on the app screen.

[0664] Step 12: Obtaining Emotion Data

[0665] The user's emotional state is acquired by a facial expression recognition camera or a biometric signal sensor and transmitted to the server via the device. The input of this step is emotional data, and the output is the emotional data transmitted to the server. The device detects the user's emotional data and transmits it to the server.

[0666] Step 13: Sentiment Analysis

[0667] The server analyzes the received emotion data to identify the user's current emotional state. The input of this step is the received emotion data, and the output is the identified emotional state. The server performs an operation to evaluate the user's emotional state using an emotion analysis algorithm.

[0668] Step 14: Generate recipe suggestions

[0669] The server uses a generative AI model to create an appropriate recipe based on inventory information and the user's emotional state. The inputs for this step are inventory information and the identified emotional state, and the output is a suggested recipe. The server uses the generative AI model to generate a personalized recipe.

[0670] Step 15: Serve the recipe

[0671] The user can view the suggested recipes in the app, select a dish, and prepare it. The input for this step is the suggested recipe, and the output is the displayed recipe information. The user browses the recipe on the app screen and proceeds with cooking.

[0672] Step 16: Collect advertising information

[0673] The server collects advertising information for nearby stores that is publicly available online. The input for this step is advertising information for nearby stores, and the output is the collected advertising data. The server uses scraping technology and APIs to collect advertising information.

[0674] Step 17: Check against inventory information

[0675] The server compares the collected advertising information with inventory information and identifies information that is useful to the user. The input for this step is the collected advertising information and inventory information, and the output is the advertising information to be notified. The server compares the inventory data with the advertising information and identifies the notification target.

[0676] Step 18: Execute Notification

[0677] The server identifies the most suitable store and product based on the user's location information and sends a notification. The input for this step is the advertising information to be notified and the user's location information, and the output is the sent notification information. The server provides useful advertising information to the user via LINE notifications, etc.

[0678] (Application example 2)

[0679] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0680] Today's busy consumers often have little time to manage the food in their refrigerators or purchase food, making it difficult to avoid food waste and irregular eating habits. Furthermore, the lack of a system that can provide appropriate meal suggestions based on users' emotional state is a problem, leading to a decline in the quality of users' diets. Furthermore, there is a need for efficient food management and economical shopping, such as notifications of low inventory in the refrigerator and advertised products at nearby stores.

[0681] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0682] In this invention, the server includes a camera means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the camera means and identifying food categories, an inventory management means for managing inventory information for each food category recognized by the image analysis means, an EC automatic ordering means for automatically ordering missing items based on the inventory management means, a recipe suggestion means for suggesting recipes available to the user based on the inventory management means, an emotion analysis means for analyzing the user's emotional state and suggesting recipes or meal plans based on that state, a food delivery ordering means for automatically placing an order with a food delivery service if ingredients needed for a suggested recipe or meal plan are missing, and an advertised item notification means for collecting advertised item information from designated nearby stores and notifying the user. This allows the user to efficiently manage inventory in the refrigerator, receive meal suggestions based on the user's emotional state, and automatically order necessary ingredients from a delivery service, thereby reducing waste and achieving efficient food management.

[0683] The "camera means" is a camera device for taking pictures of food in the refrigerator.

[0684] The "image analysis means" is a device or software for analyzing captured image data and identifying food items by category.

[0685] The "inventory management means" is a system for managing inventory information for each food category recognized by the image analysis means.

[0686] "EC automatic ordering means" means a device or software for automatically ordering shortage items via the Internet based on the inventory management means.

[0687] The "recipe suggestion means" is a device or software that suggests recipes that can be used by the user based on the inventory management means.

[0688] An "emotion analysis means" is a device or software that analyzes a user's emotional state and suggests recipes or meal plans based on that state.

[0689] "Food delivery ordering device" means a device or software that automatically places an order with a food delivery service when ingredients required for a suggested recipe or meal plan are missing.

[0690] The "advertised item notification means" is a device or software for collecting advertised item information from designated nearby stores and notifying the user of the information.

[0691] The present invention provides a food delivery system that takes into account the emotional state of the user in addition to the inventory management function in the refrigerator. Detailed embodiments of this system will be described below.

[0692] Hardware and software used

[0693] 1. Refrigerator camera means: A camera device for photographing food in the refrigerator.

[0694] 2. Smartphone: A device that allows users to receive emotion recognition and notifications.

[0695] 3. Server:

[0696] Image analysis means: A device that analyzes captured image data and identifies food categories.

[0697] Inventory management method: A system that manages analyzed inventory information.

[0698] Sentiment analysis means: A system that analyzes the user's emotional state and suggests recipes and meal plans based on that state.

[0699] Generative AI model: A model that generates recipes based on the user's emotional state and inventory information.

[0700] Food delivery ordering method: A system that automatically orders the necessary ingredients from a delivery service.

[0701] Advertised item notification means: A device that collects advertised item information from designated nearby stores and notifies the user.

[0702] System processing flow

[0703] The overall flow of the system is as follows: First, the camera means inside the refrigerator periodically takes pictures of the inside of the refrigerator. The captured image data is sent to a server via the Internet, where the image analysis means analyzes it and identifies the type and quantity of food stored inside. This data is then stored in the inventory management means.

[0704] When a user launches the application on their smartphone, the emotion analyzer captures and analyzes the user's emotional state using the camera and microphone. For example, if the user is feeling stressed, the emotion analyzer sends a prompt to the generative AI model to generate a "relaxing meal suitable for a stressed user."

[0705] The generative AI model generates an optimal recipe based on the prompts and the inventory information in the refrigerator. The recipe suggestion means suggests this recipe to the user, and if necessary ingredients are missing, the food delivery ordering means automatically places an order with a delivery service.

[0706] Users can review these suggestions and view inventory information and suggested recipes at any time. The advertising notification tool also collects advertising information from nearby stores, providing users with the latest economical options.

[0707] Specific examples

[0708] For example, a camera means installed inside a refrigerator takes an image of the inside of the refrigerator at 8:00 AM and sends it to a server. The server analyzes the image and recognizes that there are two apples and one piece of chicken inside. After that, when a user launches the app on their smartphone, the emotion analysis means analyzes the user's emotional state. If the user is feeling stressed, the generative AI model will receive the following prompt:

[0709] "Please suggest a meal that will help the user relax when they are feeling stressed. Below is the inventory in the refrigerator: 2 apples, 1 piece of chicken."

[0710] Based on this prompt, the generative AI model suggests recipes such as "Relaxed Apple and Chicken Salad." If other ingredients required for this recipe are missing, the food delivery ordering tool automatically places an order with the delivery service.

[0711] In this way, users can manage their refrigerator inventory, receive personalized meal suggestions based on their emotional state, and even have missing ingredients automatically ordered from a delivery service.

[0712] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0713] Step 1:

[0714] Photo taken inside the refrigerator

[0715] Input: Food status in the refrigerator, shooting schedule

[0716] Processing: The camera means installed in the terminal periodically takes pictures of the inside of the refrigerator. For example, take pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM.

[0717] Output: Image data of the inside of the refrigerator

[0718] Step 2:

[0719] Sending image data

[0720] Input: Image data of the inside of the refrigerator

[0721] Processing: The device sends the captured image data to a server via the Internet.

[0722] Output: Image data sent to the server

[0723] Step 3:

[0724] Image data analysis

[0725] Input: Submitted image data

[0726] Processing: The image analysis means installed on the server analyzes the transmitted image data and identifies each food item by category. For example, the food items are classified as apples, chicken, etc.

[0727] Output: Inventory data identified for each food category

[0728] Step 4:

[0729] Inventory management

[0730] Input: Identified inventory data

[0731] Processing: The inventory management means of the server stores the identified inventory data in a database and manages it as inventory information.

[0732] Output: Updated inventory information

[0733] Step 5:

[0734] User emotional state analysis

[0735] Input: User's facial expressions, voice, and biometric signals

[0736] Processing: When a user launches the app on their smartphone, the device's camera and microphone capture the user's emotional state and send it to the server, where the server's emotion analysis means analyzes it and generates the user's emotional state data.

[0737] Output: Emotional state data

[0738] Step 6:

[0739] Recipe generation and suggestions

[0740] Input: inventory information, emotional state data

[0741] Processing: The server's AI model generates a recipe suitable for the user based on inventory information and the user's emotional state data. For example, a recipe may be generated based on a prompt such as "a meal that will have a relaxing effect when the user is feeling stressed." The generated recipe is then suggested to the user through the recipe suggestion mechanism.

[0742] Output: Suggested recipe

[0743] Step 7:

[0744] Automatic ordering of missing ingredients

[0745] Input: Suggested recipes, inventory information

[0746] Processing: If there is a shortage of ingredients in stock based on the recipe suggested by the recipe suggestion means, the food delivery ordering means of the server automatically orders the necessary ingredients via a delivery service.

[0747] Output: Order confirmation notification

[0748] Step 8:

[0749] Advertisement notification

[0750] Input: Updated inventory information, user location information

[0751] Processing: The server's advertisement notification means collects advertisement information from the set nearby stores and transmits advertisement information to the user showing the most economical options based on the user's location information.

[0752] Output: Advertisement notification

[0753] Step 9:

[0754] User confirmation

[0755] Input: Suggested recipes, order confirmation notifications for missing ingredients, promotional product notifications

[0756] Processing: The user checks the suggested recipes, order confirmation notifications, and promotional product notifications on their smartphone, allowing them to stay informed of suggested dishes, purchased ingredients, and special promotional products.

[0757] Output: User verification data

[0758] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0759] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0760] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0761] [Third embodiment]

[0762] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0763] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0764] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0765] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0766] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0767] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0768] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0769] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0770] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0771] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0772] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0773] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0774] The present invention relates to a system for efficiently managing inventory in a refrigerator, allowing users to achieve optimal shopping and cooking. Detailed embodiments of the system will be described below.

[0775] System Overview

[0776] This system includes a camera installed inside the refrigerator, image analysis, inventory management, automatic e-commerce ordering, recipe suggestion, and advertisement notification. By linking these means, users can grasp the food inventory in their refrigerator in real time and receive suggestions for efficient shopping and cooking.

[0777] Photo taken inside the refrigerator

[0778] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, it takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. These pictures are used to record the types of food in the refrigerator and their arrangement.

[0779] Image data transmission and analysis

[0780] The captured image data is sent from the device (refrigerator camera) to a server via the Internet. The server analyzes the received images and uses an image recognition algorithm to identify the food by category. The analysis results are stored in a database and managed as inventory information.

[0781] Managing and checking inventory information

[0782] Users can check the inventory information in their refrigerator in real time through a dedicated app. The app retrieves the latest inventory information from the server and displays it to provide users with the current inventory status. For example, when a user opens the app, they can immediately see how many items are in the refrigerator.

[0783] Automatic EC ordering of missing items

[0784] Based on inventory information, food items that fall below a set threshold are automatically ordered online. The server periodically checks inventory information, and when the required food items become scarce, it automatically places an order with the registered e-commerce site. For example, if there is less than one bottle of milk, an order is automatically placed.

[0785] Recipe suggestions

[0786] Based on the inventory information, the server suggests recipes that can be made by the user. The server analyzes the inventory data, selects appropriate recipes, and sends them to the app. The user can check the suggested recipes on the app and start cooking right away. For example, if there is chicken and vegetables in the refrigerator, a recipe such as "stir-fried chicken and vegetables" will be suggested.

[0787] Advertisement Notice

[0788] The system also includes a function to collect advertised product information from selected nearby stores and notify users. The server obtains advertising information from local supermarkets and compares it with inventory information to identify products that users should be notified about. It then calculates the user's location and the distance to the store, and uses an algorithm to present the most economical option. For example, a LINE notification might be sent to the user saying, "Oranges are on sale at a nearby supermarket. It's a good deal if you go and buy some."

[0789] Example: User A's experience

[0790] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User A opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[0791] In the evening, Mr. A notices that there is little yogurt in the refrigerator, but the system has already automatically ordered it and notifies him that it will be delivered. Furthermore, he receives a LINE notification that a nearby supermarket is having a special sale on oranges, so he heads to the supermarket and does his shopping efficiently.

[0792] In this way, the present invention provides comprehensive support for the user's daily food management, shopping, and cooking, reducing waste and improving efficiency.

[0793] The processing flow will be explained below.

[0794] Step 1:

[0795] The device (refrigerator camera) takes pictures of the entire interior of the refrigerator according to a set schedule, for example, at 8:00 AM and 8:00 PM every day. These pictures are taken with a wide angle to capture the entire interior of the refrigerator.

[0796] Step 2:

[0797] The captured image data is compressed from the device (refrigerator camera) and sent to a server via the Internet, along with an acknowledgement message to confirm successful data transmission.

[0798] Step 3:

[0799] The server saves the received image data in a directory for analysis. If the save is successful, it records that fact in the log.

[0800] Step 4:

[0801] The server then analyzes the stored image data using image recognition algorithms, which identify the food items in the image by category and determine their type and quantity (for example, three apples, two tomatoes, etc.).

[0802] Step 5:

[0803] Based on the analysis results, the server updates the database with inventory information for each food category. The inventory information is updated in real time, ensuring that it is always up to date.

[0804] Step 6:

[0805] The user launches a dedicated app, which sends a request to the server to check stock availability.

[0806] Step 7:

[0807] The server retrieves the latest inventory information from the database and sends it back to the app, which displays the information to the user in real time through the app's UI.

[0808] Step 8:

[0809] The server periodically checks inventory to see if any food items are running low, and if they fall below a threshold, it generates an automatic order request for the item.

[0810] Step 9:

[0811] The server executes the automated order through the API of the e-commerce site, using shipping and payment information based on the user's pre-defined settings.

[0812] Step 10:

[0813] The server suggests recipes that can be made by the user based on inventory information, selects appropriate recipes, and sends the suggestions to the app.

[0814] Step 11:

[0815] Users can check the suggested recipes on the app, refer to the cooking instructions if necessary, and start cooking according to the recipe they select.

[0816] Step 12:

[0817] The server collects advertising information from local supermarkets and compares it with current inventory information to identify items that need to be purchased.

[0818] Step 13:

[0819] The server takes into account the user's location information, calculates travel distance and economic efficiency based on the displayed advertising information, and determines the most effective shopping method.

[0820] Step 14:

[0821] The server will provide users with advertisements for specific products and recommended store information for purchasing those products via LINE notifications, allowing users to shop in the most economical way.

[0822] Example 1

[0823] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0824] Conventional refrigerator inventory management requires manual effort and time from the user. There is also a high possibility of food shortages or waste due to duplicate purchases, making efficient food management difficult. Furthermore, there is no way to check inventory information in real time, making it inconvenient to be unable to check the inventory status of the refrigerator while shopping on the go. A new system is needed to solve these problems.

[0825] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0826] In this invention, the server includes an imaging means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the imaging means and classifying food items by category, a data management means for managing inventory information for each food category recognized by the image analysis means, an automatic e-commerce ordering means for automatically ordering missing items based on the data management means, a suggestion means for suggesting recipes that the user can use based on the data management means, an advertisement information notification means for collecting advertisement information from designated nearby stores and notifying the user, and a cloud data synchronization means for saving the inventory information saved in the data management means to the cloud and enabling the information to be checked in real time. This allows the user to efficiently manage their refrigerator inventory, prevent stockouts, and purchase and cook optimal ingredients.

[0827] "Imaging means" refers to a device for capturing images of the inside of the refrigerator, and includes cameras and other imaging devices for recording the types and arrangement of food items inside the refrigerator.

[0828] The "image analysis means" is a computer algorithm or program for analyzing image data acquired by the imaging means and identifying food items by category.

[0829] The "data management means" is a system for storing and managing inventory information for each food category recognized by the image analysis means, and includes a database and cloud storage.

[0830] "Automatic e-commerce ordering means" refers to a system that has the function of automatically ordering shortage items online based on data management means, and includes linkage with an e-commerce platform.

[0831] The "suggestion means" is a system that has the function of suggesting recipes that can be used by the user based on the data management means, and includes an algorithm that works in conjunction with the recipe database.

[0832] The "advertising information notification means" is a system that has the function of collecting advertising information from designated nearby stores and notifying the user, and includes an algorithm that works in conjunction with the user's location information.

[0833] The "cloud data synchronization means" is a system that stores inventory information stored in the data management means in the cloud, enabling information to be checked in real time.

[0834] The present invention relates to a system that efficiently manages food inventory in a refrigerator, enabling users to optimize their shopping and cooking. The system includes an imaging unit, an image analysis unit, a data management unit, an automatic e-commerce ordering unit, a suggestion unit, a promotional information notification unit, and a cloud data synchronization unit, all installed in the refrigerator.

[0835] Hardware or software configuration

[0836] Imaging means

[0837] The terminal (refrigerator camera) is a device for taking high-resolution images, and periodically takes pictures of the inside of the refrigerator, for example, every 24 hours. This camera has wireless communication capabilities and sends image data to a server via the Internet.

[0838] Image analysis methods

[0839] The server uses a high-performance image recognition algorithm (e.g., YOLO, ResNet) to analyze the received image data. This algorithm classifies food by category and identifies the food based on the extracted features.

[0840] Data Management Measures

[0841] The server stores the identified food information in a database, which includes information such as the food ingredient classification, quantity, and location, and manages it so that users can easily check the inventory status.

[0842] E-commerce automatic ordering tool

[0843] The server has the ability to automatically order food items online based on data management methods. For example, if there is less than one bottle of milk, a new order is placed via a pre-registered e-commerce site API.

[0844] Proposal means

[0845] The server proposes recipes that the user can cook based on the data management method. The server analyzes inventory data, selects appropriate recipes, and sends them to the app. The user can check the proposed recipes on the app and start cooking right away.

[0846] Promotional information notification means

[0847] The server collects promotional information from nearby stores and notifies the user. This function presents the most economical option based on the user's location and the distance to the store. For example, a LINE notification can be sent to the user to let them know about special sales at a nearby supermarket.

[0848] Cloud data synchronization method

[0849] The server stores inventory information on the cloud and updates it in real time, allowing users to check the inventory status of their refrigerators even when they are out and about.

[0850] Specific examples

[0851] User A's experience

[0852] A camera installed in the refrigerator takes an image of the inside of the refrigerator at 8:00 AM and sends it to a server. The server analyzes the image and determines that there are three apples and two tomatoes. User A opens a dedicated app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "apple and tomato salad."

[0853] In the evening, User A notices that there is little yogurt in the refrigerator, but is notified that new yogurt will arrive, as the system has already automatically placed an order. Furthermore, User A receives a LINE notification that a nearby supermarket is having a special sale on oranges, allowing User A to go to the supermarket and complete their shopping efficiently.

[0854] Example prompts to input to the generative AI model

[0855] "A camera installed inside the refrigerator takes pictures of the inside at 8:00 AM and sends them to a server. The server then analyzes the images and displays inventory information on an app. With this app, users can check the inventory in their refrigerator in real time and receive recipe suggestions, automatic ordering, and notifications of advertised products. Please explain this system."

[0856] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0857] Step 1: Photographing the inside of the refrigerator using imaging means

[0858] The device (refrigerator camera) takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. This camera is positioned to cover the entire inside of the refrigerator and captures high-resolution images. The input is the image data of the inside of the refrigerator taken by the camera, and the output is the image file. In this step, the types and arrangement of food in the refrigerator are recorded in detail.

[0859] Step 2: Sending image data

[0860] The device (refrigerator camera) sends the captured image data to a server via the Internet. Here, the input is the captured image data, and the output is the image data sent to the server. This step requires highly reliable data transmission, and data is transferred using protocols such as HTTP.

[0861] Step 3: Image analysis

[0862] The server analyzes the received image data using an image analysis algorithm (e.g., YOLO, ResNet). The input is the image data sent to the server, and the output is food data identified by category. In this step, food items in the image are identified by category and features such as size, shape, and color are extracted. For example, the server accurately recognizes foods such as apples, tomatoes, and milk from the image and registers them in a database.

[0863] Step 4: Update data management

[0864] The server stores the identified food information in a database and updates the inventory information. The input is the food identified by category, and the output is the updated database. In this step, information such as food classification, quantity, and location is recorded in the database and a timestamp is added to allow tracking of inventory fluctuations over time.

[0865] Step 5: Check inventory information

[0866] The user checks the inventory information in the refrigerator in real time through a dedicated app. The app periodically connects to the server to retrieve and display the latest inventory information. The input is the inventory information retrieved from the server, and the output is the inventory information displayed to the user. When the user opens the app, specific inventory information, such as "3 apples, 2 tomatoes," is immediately displayed.

[0867] Step 6: Automatically order missing items online

[0868] The server periodically checks the inventory information in the database and automatically places an order online for any food items that are running low. The input is the updated inventory data, and the output is the order data sent to the e-commerce platform. For example, if there is only one bottle of milk left, the server sends the necessary information to the e-commerce site API to place an automatic order.

[0869] Step 7: Recipe suggestions

[0870] The server suggests recipes that can be made by the user based on inventory information. The input is inventory information in the database, and the output is suggested recipe information. The server analyzes the inventory data, selects appropriate recipes from the database, and sends them to the app. The user can then view specific recipe suggestions, such as "apple and tomato salad," on the app.

[0871] Step 8: Promotional Information Notification

[0872] The server collects advertising information from nearby stores and notifies the user. The input is the collected advertising information, and the output is the advertising information notified to the user. The server obtains advertising data from local supermarkets and compares it with inventory information to identify products to notify the user about. Furthermore, the server calculates the travel distance to the optimal store based on the user's location information, and sends the user, for example, information about special sales at nearby supermarkets via LINE notifications.

[0873] (Application example 1)

[0874] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0875] Conventional refrigerator inventory management systems are limited to efficient inventory management and shopping optimization within the home, and have not been applied to inventory management or product recommendation efficiency in physical stores. In addition, there is a lack of a system for store clerks to check inventory information in real time and make optimal product recommendations to customers, so improving inventory management efficiency and customer satisfaction are issues. Furthermore, there is a need for a means to increase customer purchasing motivation by appropriately notifying customers of advertising information and promotions.

[0876] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0877] In this invention, the server includes an imaging means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the imaging means and classifying the food by category, an inventory management means for managing inventory information for each food category recognized by the image analysis means, an automatic e-commerce ordering means for automatically ordering missing items based on the inventory management means, a recipe suggestion means for suggesting recipes available to the user based on the inventory management means, an advertised item notification means for collecting advertised item information from selected nearby stores and notifying the user, a visualization means for displaying in-store inventory information in real time on a display device worn by the user and checking inventory and suggesting products, an automatic replenishment means for automatically replenishing in-store inventory when inventory falls below a certain threshold, and a notification means for notifying the display device of advertising information and promotional information. This improves the efficiency of inventory management in physical stores, enables store staff to suggest optimal products to customers in real time, and improves customer satisfaction by effectively notifying them of advertising information and promotional information.

[0878] The "imaging means" is a device that takes images of the inside of a refrigerator or a store.

[0879] The "image analysis means" is a device or software that has the function of analyzing captured image data and identifying food or product categories.

[0880] The "inventory management means" is a device or software that has the function of managing inventory information recognized by the image analysis means and updating it in real time.

[0881] An "automated electronic commerce ordering means" is a device or software that has the function of automatically ordering products that are in short supply based on inventory information.

[0882] The "recipe suggestion means" is a device or software that has the function of suggesting recipes that can be used by the user based on inventory information.

[0883] The "advertised item notification means" is a device or software that has the function of notifying the user of advertised item information collected from designated nearby stores.

[0884] A "display device" is a display device worn by a user to visually confirm information.

[0885] The "visualization means" is a device or software that displays inventory information in real time through a display device and has the function of checking inventory and making product suggestions.

[0886] An "automatic replenishment means" is a device or software that has the function of automatically replenishing products when inventory falls below a certain threshold.

[0887] The "notification means" is a device or software that has the function of notifying the display device of advertising information or promotional information.

[0888] The present invention relates to a system for improving the efficiency of inventory management and product recommendations in a physical store. Detailed embodiments of the present invention will be described below.

[0889] System Overview

[0890] This system is comprised of an imaging means, an image analysis means, an inventory management means, an automatic e-commerce ordering means, a recipe suggestion means, an advertised product notification means, a visualization means, an automatic replenishment means, and a notification means. By linking these means, efficient inventory management, product suggestions, and advertising information notifications in physical stores become possible.

[0891] Photography and image analysis

[0892] The server first periodically captures images of the product shelves in the store using an imaging device. This capture is performed automatically at regular intervals. The captured image data is sent to the server and analyzed by the image analysis device. The image analysis device then classifies the food and product by category and stores the data in a database as inventory information.

[0893] Inventory management and e-commerce automatic ordering

[0894] The server updates the inventory information recognized by the image analysis means in real time using the inventory management means. The user can wear a display device and instantly check the current inventory information using the visualization means. Based on this inventory information, if the inventory falls below a certain threshold, the e-commerce automatic ordering means automatically places a replenishment order.

[0895] Recipe suggestions and promotional product notifications

[0896] The server is equipped with a recipe suggestion unit that suggests recipes available to the user based on inventory information. Furthermore, an advertised product notification unit notifies the user of advertised product information collected from designated nearby stores. The server calculates the user's location information and the travel distance to the store, and uses an algorithm to present the most economical options.

[0897] In-store use

[0898] Store staff wear display devices and use visualization tools to check inventory information in real time. For example, if an item on a shelf runs low, an automatic replenishment tool is activated to immediately replenish it. It is also possible to check whether the product desired by the customer is in stock and present alternative options.

[0899] Notification means

[0900] The advertising information and promotion information are displayed on the display device using the notification means, allowing the store staff to provide the customer with appropriate product information.

[0901] Examples of concrete examples and prompts

[0902] For example, when a store clerk wears a display device and activates the "Check Stock" command, real-time inventory information is displayed on the display. For example, it may show "Milk: 10, Eggs: 5, Bread: 2." Also, when the "Auto Order" command is executed, inventory below a threshold (for example, bread) is automatically ordered.

[0903] Below are some example prompts for a generative AI model:

[0904] Can you give us an example of a system that streamlines refrigerator inventory management? A system that meets the following requirements: 1. Periodically takes pictures of the inside of the refrigerator using a camera. 2. Analyzes the images and updates inventory information. 3. Checks inventory information in real time using a smartphone or smart glasses. 4. Automatically places e-commerce orders when inventory is low. 5. Suggests recipes that can be made. 6. Notifies users of advertised product information.

[0905] The above is the details of the embodiment of the invention. This system is expected to improve the efficiency of inventory management in stores and increase customer satisfaction.

[0906] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0907] Step 1:

[0908] The server periodically captures images of the product shelves in the store using an imaging device. It receives the time interval and camera position as input and obtains the captured image data as output. The images captured by this imaging device are saved in a storage directory.

[0909] Step 2:

[0910] The server sends the captured image data to the image analysis means, which receives the image data as input and outputs the analysis results. The image analysis means uses machine learning algorithms to identify each item in the image by category and stores the results in a database.

[0911] Step 3:

[0912] The server retrieves inventory information from the database and manages the latest inventory information using inventory management means. It receives analysis result data as input and saves updated inventory information in the database as output. This is done in real time by the inventory management means.

[0913] Step 4:

[0914] The server uses visualization means to display the latest inventory information via a display device worn by the user. The latest inventory information is acquired as input, and the inventory information displayed on the display is obtained as output. This allows the user to check the current inventory status.

[0915] Step 5:

[0916] When inventory information falls below a certain threshold, the server automatically orders the missing items using an automatic e-commerce ordering system. The server receives inventory information and a set threshold as input, and obtains data related to the automatic order as output. This allows for a rapid response when inventory shortages occur.

[0917] Step 6:

[0918] The server activates a recipe suggestion means that suggests available recipes to the user based on the inventory information. The server receives the inventory information as input and generates suggested recipe information as output. The user can check these suggested recipes on the display device.

[0919] Step 7:

[0920] Using the advertising product notification means, the server notifies the user of advertising information collected from the set nearby stores. It receives advertising information and the user's location information as input and generates an optimal advertising notification as output. The user can learn about economical options through the display device.

[0921] Step 8:

[0922] The advertising information and promotion information are displayed on the display device by the notification means. The advertising information is used as input and the notification on the display is used as output. This allows the user and store clerk to instantly check the latest promotion information.

[0923] Step 9:

[0924] The user checks inventory and makes product suggestions via a display device, and replenishes products or makes suggestions to customers based on the information displayed on the display.The system receives instructions from the display device as input and generates specific product suggestions and replenishment actions as output.

[0925] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0926] The present invention is a system that provides a more personalized dining experience by suggesting menu items that take into account not only refrigerator inventory management but also the user's emotional state. Detailed embodiments of this system are described below.

[0927] System Overview

[0928] This system includes a camera installed inside the refrigerator, image analysis, inventory management, automatic e-commerce ordering, recipe suggestion, advertisement notification, and an emotion engine that recognizes the user's emotions. By linking these mechanisms, users can grasp the inventory in their refrigerator in real time and receive efficient shopping and cooking suggestions that match their emotional state.

[0929] Photo taken inside the refrigerator

[0930] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, it takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. These pictures are used to record the types of food in the refrigerator and their arrangement.

[0931] Image data transmission and analysis

[0932] The captured image data is sent from the device (refrigerator camera) to a server via the Internet. The server analyzes the received images and uses an image recognition algorithm to identify the food by category. The analysis results are stored in a database and managed as inventory information.

[0933] Managing and checking inventory information

[0934] Users can check the inventory information in their refrigerator in real time through a dedicated app. The app retrieves the latest inventory information from the server and displays it to provide users with the current inventory status. For example, when a user opens the app, they can immediately see how many items are in the refrigerator.

[0935] Automatic EC ordering of missing items

[0936] Based on inventory information, food items that fall below a set threshold are automatically ordered online. The server periodically checks inventory information, and when the required food items become scarce, it automatically places an order with the registered e-commerce site. For example, if there is less than one bottle of milk, an order is automatically placed.

[0937] Linking recipe suggestions with an emotion engine

[0938] Based on inventory information and the user's emotional state, the server suggests recipes that the user can make. The emotion engine analyzes the user's facial expressions, voice, and bio-signals to identify their emotional state. For example, if it detects that the user is feeling stressed, it will suggest simple recipes that have a relaxing effect. The user can check the suggested recipes on the app and proceed with selecting and preparing the food. For example, if there is chicken and vegetables in the refrigerator and the user is tired, it will suggest a recipe such as "easy stir-fried chicken and vegetables."

[0939] Advertisement Notice

[0940] This system also includes a function to collect information about products advertised in nearby stores and notify users. The server obtains advertising information from nearby stores and compares it with inventory information to identify products that should be notified to users. It then calculates the user's location and the distance to the store, and uses an algorithm to present the most economical option. For example, a LINE notification may be sent to the user saying, "Oranges are on sale at a nearby supermarket. It's a good deal if you go and buy some."

[0941] Example: User B's experience

[0942] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User B opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[0943] Furthermore, the emotion engine detects that B is tired at the time and suggests other easy-to-make recipes. By suggesting easy-to-make menus such as "Easy Tomato and Cheese Pizza," B's burden is reduced. In addition, information about items on sale at nearby supermarkets is sent via LINE notifications, allowing for economical shopping.

[0944] In this way, the present invention, including the emotion engine, provides comprehensive support for the user's daily food management, shopping, and cooking, reducing waste and improving efficiency.

[0945] The processing flow will be explained below.

[0946] Step 1:

[0947] The device (refrigerator camera) takes pictures of the inside of the refrigerator periodically according to a set schedule, for example, at 8:00 AM and 8:00 PM every day.

[0948] Step 2:

[0949] The captured image data is sent from the device (refrigerator camera) to the server via the Internet. Once the transmission is complete, an acknowledgement message is sent.

[0950] Step 3:

[0951] The server saves the received image data in a directory for analysis. If the save is successful, it records that fact in the log.

[0952] Step 4:

[0953] The server then analyzes the stored image data using an image recognition algorithm, which identifies the food items in the image by category and identifies their type and quantity (for example, three apples, two tomatoes, etc.).

[0954] Step 5:

[0955] Based on the analysis results, the server updates the database with inventory information for each food category. The inventory information is updated in real time, ensuring that it is always up to date.

[0956] Step 6:

[0957] The user launches a dedicated app, which sends a request to the server to check stock availability.

[0958] Step 7:

[0959] The server retrieves the latest inventory information from the database and sends it back to the app, which displays the information to the user in real time through the app's UI.

[0960] Step 8:

[0961] The server periodically checks inventory to see if any food items are running low, and if they fall below a threshold, it generates an automatic order request for the item.

[0962] Step 9:

[0963] The server executes the automated order through the API of the e-commerce site, using shipping and payment information based on the user's pre-defined settings.

[0964] Step 10:

[0965] Based on the user's refrigerator inventory and special requests, the app executes an emotion recognition program and sends emotional data to the server.

[0966] Step 11:

[0967] The server selects appropriate recipes based on the user's emotional state. For example, if it detects that the user is feeling stressed, it will select simple, relaxing recipes.

[0968] Step 12:

[0969] The server sends the selected recipe to the app and displays it to the user, who can then review the suggested recipes and refer to cooking instructions as needed.

[0970] Step 13:

[0971] The server collects advertising information from local supermarkets, compares it with current inventory information, and identifies items that need to be purchased.

[0972] Step 14:

[0973] The server takes into account the user's location information, calculates travel distance and economic efficiency based on the displayed advertising information, and determines the most effective shopping method.

[0974] Step 15:

[0975] The server sends LINE notifications to users, advertising specific products and providing them with recommended store information for purchasing them. For example, it can notify users that oranges are on sale at a nearby supermarket, helping them shop in the most economical way.

[0976] Example: User B's experience

[0977] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User B opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[0978] Furthermore, the emotion engine detects that B is tired at the time and suggests other easy-to-make recipes. By suggesting easy-to-make menus such as "Easy Tomato and Cheese Pizza," B's burden is reduced. In addition, information about items on sale at nearby supermarkets is sent via LINE notifications, allowing for efficient shopping.

[0979] Example 2

[0980] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0981] In modern life, efficient refrigerator inventory management and consumption are important challenges for many households. In particular, reducing food waste, replenishing needed ingredients in a timely manner, and providing users with recipes tailored to their needs are difficult. Furthermore, there is a need for systems that can provide meal suggestions based on the user's emotional state and economical shopping options, but no systems exist that can comprehensively address these challenges.

[0982] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a camera means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the camera means and classifying the food by category, an inventory management means for managing inventory information for each food category recognized by the image analysis means, an EC automatic ordering means for automatically ordering missing items, an emotion analysis means for recognizing the emotional state of the user, a recipe suggestion means for suggesting recipes that the user can use based on the inventory management means, and an advertised item notification means for collecting advertised item information from nearby stores and notifying the user. This makes it possible to efficiently manage inventory in the refrigerator and provide personalized meal suggestions and economical shopping options according to the user's emotional state.

[0983] The "camera means" is a device that takes pictures of the inside of the refrigerator.

[0984] The "image analysis means" is a module that analyzes the captured image data and identifies the food by category.

[0985] The "inventory management means" is a system that manages inventory information for each food category recognized by the image analysis means.

[0986] The "EC automatic ordering means" is a module that automatically places online orders for items that are in short supply based on the inventory management means.

[0987] The "recipe suggestion means" is a system that suggests recipes that can be used by the user based on the inventory management means.

[0988] The "advertised item notification means" is a module that collects advertised item information from designated nearby stores and notifies the user of the information.

[0989] An "emotion analysis means" is a system for recognizing a user's emotional state.

[0990] The present invention is a system that not only manages inventory in a refrigerator but also proposes menus that take into account the emotional state of the user, thereby providing a more personalized dining experience. Detailed embodiments of this system are described below.

[0991] System Overview

[0992] This system includes a camera installed inside the refrigerator, image analysis, inventory management, automatic e-commerce ordering, recipe suggestion, advertisement notification, and emotion analysis that recognizes the user's emotions. By linking these means, the user can grasp the inventory in the refrigerator in real time and receive efficient shopping and cooking suggestions that match their emotional state.

[0993] Photo taken inside the refrigerator

[0994] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, it takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. These pictures are used to record the types of food in the refrigerator and their arrangement.

[0995] Image data transmission and analysis

[0996] The captured image data is sent from the device (refrigerator camera) to a server via the Internet. The server analyzes the received images and uses an image recognition algorithm to identify the food by category. This analysis is performed using image recognition software such as TensorFlow and OpenCV. The results of this analysis are stored in a database and managed as inventory information.

[0997] Managing and checking inventory information

[0998] Users can check the inventory information in their refrigerator in real time through a dedicated app. The app retrieves the latest inventory information from the server and displays it to provide users with the current inventory status. For example, when a user opens the app, they can immediately see how many items are in the refrigerator.

[0999] Automatic EC ordering of missing items

[1000] Based on inventory information, food items that fall below a set threshold are automatically ordered online. The server periodically checks inventory information, and when the required food items become scarce, it automatically places an order with the registered e-commerce site (e.g., Amazon or Rakuten Ichiba). For example, an order is automatically placed when there is less than one bottle of milk left.

[1001] Linking recipe suggestions with an emotion engine

[1002] Based on inventory information and the user's emotional state, the server suggests recipes that the user can make. The emotion engine analyzes the user's facial expressions, voice, and bio-signals to identify their emotional state. For example, if it detects that the user is feeling stressed, it will suggest simple recipes that have a relaxing effect. The user can check the suggested recipes on the app and proceed with selecting and preparing the food. For example, if there is chicken and vegetables in the refrigerator and the user is tired, it will suggest a recipe such as "easy stir-fried chicken and vegetables."

[1003] Advertisement Notice

[1004] The server retrieves advertising information from nearby stores and compares it with inventory information to identify products that should be notified to the user. It then calculates the user's location and the distance to the store, and uses an algorithm to present the most economical option. For example, a LINE notification might be sent to the user saying, "Oranges are on sale at a nearby supermarket. It's a good deal if you stop by to buy some."

[1005] Example: User B's experience

[1006] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User B opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[1007] Furthermore, the emotion engine detects that B is tired at the time and suggests other easy-to-make recipes. By suggesting easy-to-make menus such as "Easy Tomato and Cheese Pizza," B's burden is reduced. In addition, information about items on sale at nearby supermarkets is sent via LINE notifications, allowing for economical shopping.

[1008] Example prompts to input to a generative AI model:

[1009] Please provide a detailed explanation of the specific processing steps and operations of a system that manages refrigerator inventory and suggests menu items while also taking into account the user's emotions. Please also mention what hardware and software are used in the process, and what data processing and calculations are performed.

[1010] In this way, the present invention, including the emotion engine, provides comprehensive support for the user's daily food management, shopping, and cooking, reducing waste and improving efficiency.

[1011] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1012] Step 1: Photographing the inside of the refrigerator

[1013] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, take pictures of the inside every day at 8:00 AM and 8:00 PM. The input to this step is the schedule setting information, and the output is the image data of the inside of the refrigerator. The camera operates and automatically records the food in the refrigerator at the specified time.

[1014] Step 2: Sending image data

[1015] The device (refrigerator camera) sends the captured image data to a server via the Internet. The input of this step is the captured image data, and the output is the image data sent to the server. The camera uses a network connection to upload the captured image data to the server.

[1016] Step 3: Receiving image data

[1017] The server receives image data sent via the Internet. The input of this step is the image data sent from the device, and the output is the storage of the received image data. The server receives the image data using a specific API endpoint.

[1018] Step 4: Image analysis

[1019] The server applies image analysis algorithms to the received images to recognize food in the image. The input of this step is the received image data, and the output is the recognized food data. Specifically, the server uses software such as TensorFlow and OpenCV to identify foods such as apples and tomatoes.

[1020] Step 5: Update inventory data

[1021] Based on the analysis results, the server updates the database with the food category and quantity. The input to this step is the recognized food data, and the output is the updated inventory information. For example, if the analysis finds that there are 3 apples and 2 tomatoes, this information is registered in the database.

[1022] Step 6: Get inventory information

[1023] The user opens the dedicated app and sends a request to the server to check the inventory information in the refrigerator. The input of this step is the user's request data, and the output is a request to obtain inventory information. The user operates the app interface to send the request to the server.

[1024] Step 7: Submit inventory information

[1025] The server retrieves the latest inventory information from the database and sends it to the user's app. The input to this step is a request to retrieve inventory information, and the output is the latest inventory information. The server queries the inventory data and sends the retrieved information to the user's app.

[1026] Step 8: View inventory information

[1027] The user checks the inventory information in the app. The input of this step is the inventory information sent from the server, and the output is the displayed inventory information. The user views the inventory information on the app screen.

[1028] Step 9: Check inventory

[1029] The server periodically checks inventory information and identifies foods that have fallen below a set threshold. The input to this step is the inventory information in the database, and the output is a list of items that are in short supply. The server scans the database and extracts the items that are in short supply.

[1030] Step 10: Execute automatic orders

[1031] When the server identifies missing items, it automatically places an order with the registered e-commerce site. The input for this step is a list of missing items, and the output is order confirmation information. The server calls the e-commerce site's API based on the list of missing items and performs the order procedure.

[1032] Step 11: Order confirmation

[1033] The user can check the automatically ordered food and its status on the app. The input of this step is the order confirmation information, and the output is the displayed order confirmation information. The user checks the order status on the app screen.

[1034] Step 12: Obtaining Emotion Data

[1035] The user's emotional state is acquired by a facial expression recognition camera or a biometric signal sensor and transmitted to the server via the device. The input of this step is emotional data, and the output is the emotional data transmitted to the server. The device detects the user's emotional data and transmits it to the server.

[1036] Step 13: Sentiment Analysis

[1037] The server analyzes the received emotion data to identify the user's current emotional state. The input of this step is the received emotion data, and the output is the identified emotional state. The server performs an operation to evaluate the user's emotional state using an emotion analysis algorithm.

[1038] Step 14: Generate recipe suggestions

[1039] The server uses a generative AI model to create an appropriate recipe based on inventory information and the user's emotional state. The inputs for this step are inventory information and the identified emotional state, and the output is a suggested recipe. The server uses the generative AI model to generate a personalized recipe.

[1040] Step 15: Serve the recipe

[1041] The user can view the suggested recipes in the app, select a dish, and prepare it. The input for this step is the suggested recipe, and the output is the displayed recipe information. The user browses the recipe on the app screen and proceeds with cooking.

[1042] Step 16: Collect advertising information

[1043] The server collects advertising information for nearby stores that is publicly available online. The input for this step is advertising information for nearby stores, and the output is the collected advertising data. The server uses scraping technology and APIs to collect advertising information.

[1044] Step 17: Check against inventory information

[1045] The server compares the collected advertising information with inventory information and identifies information that is useful to the user. The input for this step is the collected advertising information and inventory information, and the output is the advertising information to be notified. The server compares the inventory data with the advertising information and identifies the notification target.

[1046] Step 18: Execute Notification

[1047] The server identifies the most suitable store and product based on the user's location information and sends a notification. The input for this step is the advertising information to be notified and the user's location information, and the output is the sent notification information. The server provides useful advertising information to the user via LINE notifications, etc.

[1048] (Application example 2)

[1049] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1050] Today's busy consumers often have little time to manage the food in their refrigerators or purchase food, making it difficult to avoid food waste and irregular eating habits. Furthermore, the lack of a system that can provide appropriate meal suggestions based on users' emotional state is a problem, leading to a decline in the quality of users' diets. Furthermore, there is a need for efficient food management and economical shopping, such as notifications of low inventory in the refrigerator and advertised products at nearby stores.

[1051] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1052] In this invention, the server includes a camera means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the camera means and identifying food categories, an inventory management means for managing inventory information for each food category recognized by the image analysis means, an EC automatic ordering means for automatically ordering missing items based on the inventory management means, a recipe suggestion means for suggesting recipes available to the user based on the inventory management means, an emotion analysis means for analyzing the user's emotional state and suggesting recipes or meal plans based on that state, a food delivery ordering means for automatically placing an order with a food delivery service if ingredients needed for a suggested recipe or meal plan are missing, and an advertised item notification means for collecting advertised item information from designated nearby stores and notifying the user. This allows the user to efficiently manage inventory in the refrigerator, receive meal suggestions based on the user's emotional state, and automatically order necessary ingredients from a delivery service, thereby reducing waste and achieving efficient food management.

[1053] The "camera means" is a camera device for taking pictures of food in the refrigerator.

[1054] The "image analysis means" is a device or software for analyzing captured image data and identifying food items by category.

[1055] The "inventory management means" is a system for managing inventory information for each food category recognized by the image analysis means.

[1056] "EC automatic ordering means" means a device or software for automatically ordering shortage items via the Internet based on the inventory management means.

[1057] The "recipe suggestion means" is a device or software that suggests recipes that can be used by the user based on the inventory management means.

[1058] An "emotion analysis means" is a device or software that analyzes a user's emotional state and suggests recipes or meal plans based on that state.

[1059] "Food delivery ordering device" means a device or software that automatically places an order with a food delivery service when ingredients required for a suggested recipe or meal plan are missing.

[1060] The "advertised item notification means" is a device or software for collecting advertised item information from designated nearby stores and notifying the user of the information.

[1061] The present invention provides a food delivery system that takes into account the emotional state of the user in addition to the inventory management function in the refrigerator. Detailed embodiments of this system will be described below.

[1062] Hardware and software used

[1063] 1. Refrigerator camera means: A camera device for photographing food in the refrigerator.

[1064] 2. Smartphone: A device that allows users to receive emotion recognition and notifications.

[1065] 3. Server:

[1066] Image analysis means: A device that analyzes captured image data and identifies food categories.

[1067] Inventory management method: A system that manages analyzed inventory information.

[1068] Sentiment analysis means: A system that analyzes the user's emotional state and suggests recipes and meal plans based on that state.

[1069] Generative AI model: A model that generates recipes based on the user's emotional state and inventory information.

[1070] Food delivery ordering method: A system that automatically orders the necessary ingredients from a delivery service.

[1071] Advertised item notification means: A device that collects advertised item information from designated nearby stores and notifies the user.

[1072] System processing flow

[1073] The overall flow of the system is as follows: First, the camera means inside the refrigerator periodically takes pictures of the inside of the refrigerator. The captured image data is sent to a server via the Internet, where the image analysis means analyzes it and identifies the type and quantity of food stored inside. This data is then stored in the inventory management means.

[1074] When a user launches the application on their smartphone, the emotion analyzer captures and analyzes the user's emotional state using the camera and microphone. For example, if the user is feeling stressed, the emotion analyzer sends a prompt to the generative AI model to generate a "relaxing meal suitable for a stressed user."

[1075] The generative AI model generates an optimal recipe based on the prompts and the inventory information in the refrigerator. The recipe suggestion means suggests this recipe to the user, and if necessary ingredients are missing, the food delivery ordering means automatically places an order with a delivery service.

[1076] Users can review these suggestions and view inventory information and suggested recipes at any time. The advertising notification tool also collects advertising information from nearby stores, providing users with the latest economical options.

[1077] Specific examples

[1078] For example, a camera means installed inside a refrigerator takes an image of the inside of the refrigerator at 8:00 AM and sends it to a server. The server analyzes the image and recognizes that there are two apples and one piece of chicken inside. After that, when a user launches the app on their smartphone, the emotion analysis means analyzes the user's emotional state. If the user is feeling stressed, the generative AI model will receive the following prompt:

[1079] "Please suggest a meal that will help the user relax when they are feeling stressed. Below is the inventory in the refrigerator: 2 apples, 1 piece of chicken."

[1080] Based on this prompt, the generative AI model suggests recipes such as "Relaxed Apple and Chicken Salad." If other ingredients required for this recipe are missing, the food delivery ordering tool automatically places an order with the delivery service.

[1081] In this way, users can manage their refrigerator inventory, receive personalized meal suggestions based on their emotional state, and even have missing ingredients automatically ordered from a delivery service.

[1082] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1083] Step 1:

[1084] Photo taken inside the refrigerator

[1085] Input: Food status in the refrigerator, shooting schedule

[1086] Processing: The camera means installed in the terminal periodically takes pictures of the inside of the refrigerator. For example, take pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM.

[1087] Output: Image data of the inside of the refrigerator

[1088] Step 2:

[1089] Sending image data

[1090] Input: Image data of the inside of the refrigerator

[1091] Processing: The device sends the captured image data to a server via the Internet.

[1092] Output: Image data sent to the server

[1093] Step 3:

[1094] Image data analysis

[1095] Input: Submitted image data

[1096] Processing: The image analysis means installed on the server analyzes the transmitted image data and identifies each food item by category. For example, the food items are classified as apples, chicken, etc.

[1097] Output: Inventory data identified for each food category

[1098] Step 4:

[1099] Inventory management

[1100] Input: Identified inventory data

[1101] Processing: The inventory management means of the server stores the identified inventory data in a database and manages it as inventory information.

[1102] Output: Updated inventory information

[1103] Step 5:

[1104] User emotional state analysis

[1105] Input: User's facial expressions, voice, and biometric signals

[1106] Processing: When a user launches the app on their smartphone, the device's camera and microphone capture the user's emotional state and send it to the server, where the server's emotion analysis means analyzes it and generates the user's emotional state data.

[1107] Output: Emotional state data

[1108] Step 6:

[1109] Recipe generation and suggestions

[1110] Input: inventory information, emotional state data

[1111] Processing: The server's AI model generates a recipe suitable for the user based on inventory information and the user's emotional state data. For example, a recipe may be generated based on a prompt such as "a meal that will have a relaxing effect when the user is feeling stressed." The generated recipe is then suggested to the user through the recipe suggestion mechanism.

[1112] Output: Suggested recipe

[1113] Step 7:

[1114] Automatic ordering of missing ingredients

[1115] Input: Suggested recipes, inventory information

[1116] Processing: If there is a shortage of ingredients in stock based on the recipe suggested by the recipe suggestion means, the food delivery ordering means of the server automatically orders the necessary ingredients via a delivery service.

[1117] Output: Order confirmation notification

[1118] Step 8:

[1119] Advertisement notification

[1120] Input: Updated inventory information, user location information

[1121] Processing: The server's advertisement notification means collects advertisement information from the set nearby stores and transmits advertisement information to the user showing the most economical options based on the user's location information.

[1122] Output: Advertisement notification

[1123] Step 9:

[1124] User confirmation

[1125] Input: Suggested recipes, order confirmation notifications for missing ingredients, promotional product notifications

[1126] Processing: The user checks the suggested recipes, order confirmation notifications, and promotional product notifications on their smartphone, allowing them to stay informed of suggested dishes, purchased ingredients, and special promotional products.

[1127] Output: User verification data

[1128] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1129] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1130] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1131] [Fourth embodiment]

[1132] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1133] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1134] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1135] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1136] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1137] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1138] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1139] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1140] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1141] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1142] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1143] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1144] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1145] The present invention relates to a system for efficiently managing inventory in a refrigerator, allowing users to achieve optimal shopping and cooking. Detailed embodiments of the system will be described below.

[1146] System Overview

[1147] This system includes a camera installed inside the refrigerator, image analysis, inventory management, automatic e-commerce ordering, recipe suggestion, and advertisement notification. By linking these means, users can grasp the food inventory in their refrigerator in real time and receive suggestions for efficient shopping and cooking.

[1148] Photo taken inside the refrigerator

[1149] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, it takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. These pictures are used to record the types of food in the refrigerator and their arrangement.

[1150] Image data transmission and analysis

[1151] The captured image data is sent from the device (refrigerator camera) to a server via the Internet. The server analyzes the received images and uses an image recognition algorithm to identify the food by category. The analysis results are stored in a database and managed as inventory information.

[1152] Managing and checking inventory information

[1153] Users can check the inventory information in their refrigerator in real time through a dedicated app. The app retrieves the latest inventory information from the server and displays it to provide users with the current inventory status. For example, when a user opens the app, they can immediately see how many items are in the refrigerator.

[1154] Automatic EC ordering of missing items

[1155] Based on inventory information, food items that fall below a set threshold are automatically ordered online. The server periodically checks inventory information, and when the required food items become scarce, it automatically places an order with the registered e-commerce site. For example, if there is less than one bottle of milk, an order is automatically placed.

[1156] Recipe suggestions

[1157] Based on the inventory information, the server suggests recipes that can be made by the user. The server analyzes the inventory data, selects appropriate recipes, and sends them to the app. The user can check the suggested recipes on the app and start cooking right away. For example, if there is chicken and vegetables in the refrigerator, a recipe such as "stir-fried chicken and vegetables" will be suggested.

[1158] Advertisement Notice

[1159] The system also includes a function to collect advertised product information from selected nearby stores and notify users. The server obtains advertising information from local supermarkets and compares it with inventory information to identify products that users should be notified about. It then calculates the user's location and the distance to the store, and uses an algorithm to present the most economical option. For example, a LINE notification might be sent to the user saying, "Oranges are on sale at a nearby supermarket. It's a good deal if you go and buy some."

[1160] Example: User A's experience

[1161] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User A opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[1162] In the evening, Mr. A notices that there is little yogurt in the refrigerator, but the system has already automatically ordered it and notifies him that it will be delivered. Furthermore, he receives a LINE notification that a nearby supermarket is having a special sale on oranges, so he heads to the supermarket and does his shopping efficiently.

[1163] In this way, the present invention provides comprehensive support for the user's daily food management, shopping, and cooking, reducing waste and improving efficiency.

[1164] The processing flow will be explained below.

[1165] Step 1:

[1166] The device (refrigerator camera) takes pictures of the entire interior of the refrigerator according to a set schedule, for example, at 8:00 AM and 8:00 PM every day. These pictures are taken with a wide angle to capture the entire interior of the refrigerator.

[1167] Step 2:

[1168] The captured image data is compressed from the device (refrigerator camera) and sent to a server via the Internet, along with an acknowledgement message to confirm successful data transmission.

[1169] Step 3:

[1170] The server saves the received image data in a directory for analysis. If the save is successful, it records that fact in the log.

[1171] Step 4:

[1172] The server then analyzes the stored image data using image recognition algorithms, which identify the food items in the image by category and determine their type and quantity (for example, three apples, two tomatoes, etc.).

[1173] Step 5:

[1174] Based on the analysis results, the server updates the database with inventory information for each food category. The inventory information is updated in real time, ensuring that it is always up to date.

[1175] Step 6:

[1176] The user launches a dedicated app, which sends a request to the server to check stock availability.

[1177] Step 7:

[1178] The server retrieves the latest inventory information from the database and sends it back to the app, which displays the information to the user in real time through the app's UI.

[1179] Step 8:

[1180] The server periodically checks inventory to see if any food items are running low, and if they fall below a threshold, it generates an automatic order request for the item.

[1181] Step 9:

[1182] The server executes the automated order through the API of the e-commerce site, using shipping and payment information based on the user's pre-defined settings.

[1183] Step 10:

[1184] The server suggests recipes that can be made by the user based on inventory information, selects appropriate recipes, and sends the suggestions to the app.

[1185] Step 11:

[1186] Users can check the suggested recipes on the app, refer to the cooking instructions if necessary, and start cooking according to the recipe they select.

[1187] Step 12:

[1188] The server collects advertising information from local supermarkets and compares it with current inventory information to identify items that need to be purchased.

[1189] Step 13:

[1190] The server takes into account the user's location information, calculates travel distance and economic efficiency based on the displayed advertising information, and determines the most effective shopping method.

[1191] Step 14:

[1192] The server will provide users with advertisements for specific products and recommended store information for purchasing those products via LINE notifications, allowing users to shop in the most economical way.

[1193] Example 1

[1194] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1195] Conventional refrigerator inventory management requires manual effort and time from the user. There is also a high possibility of food shortages or waste due to duplicate purchases, making efficient food management difficult. Furthermore, there is no way to check inventory information in real time, making it inconvenient to be unable to check the inventory status of the refrigerator while shopping on the go. A new system is needed to solve these problems.

[1196] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1197] In this invention, the server includes an imaging means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the imaging means and classifying food items by category, a data management means for managing inventory information for each food category recognized by the image analysis means, an automatic e-commerce ordering means for automatically ordering missing items based on the data management means, a suggestion means for suggesting recipes that the user can use based on the data management means, an advertisement information notification means for collecting advertisement information from designated nearby stores and notifying the user, and a cloud data synchronization means for saving the inventory information saved in the data management means to the cloud and enabling the information to be checked in real time. This allows the user to efficiently manage their refrigerator inventory, prevent stockouts, and purchase and cook optimal ingredients.

[1198] "Imaging means" refers to a device for capturing images of the inside of the refrigerator, and includes cameras and other imaging devices for recording the types and arrangement of food items inside the refrigerator.

[1199] The "image analysis means" is a computer algorithm or program for analyzing image data acquired by the imaging means and identifying food items by category.

[1200] The "data management means" is a system for storing and managing inventory information for each food category recognized by the image analysis means, and includes a database and cloud storage.

[1201] "Automatic e-commerce ordering means" refers to a system that has the function of automatically ordering shortage items online based on data management means, and includes linkage with an e-commerce platform.

[1202] The "suggestion means" is a system that has the function of suggesting recipes that can be used by the user based on the data management means, and includes an algorithm that works in conjunction with the recipe database.

[1203] The "advertising information notification means" is a system that has the function of collecting advertising information from designated nearby stores and notifying the user, and includes an algorithm that works in conjunction with the user's location information.

[1204] The "cloud data synchronization means" is a system that stores inventory information stored in the data management means in the cloud, enabling information to be checked in real time.

[1205] The present invention relates to a system that efficiently manages food inventory in a refrigerator, enabling users to optimize their shopping and cooking. The system includes an imaging unit, an image analysis unit, a data management unit, an automatic e-commerce ordering unit, a suggestion unit, a promotional information notification unit, and a cloud data synchronization unit, all installed in the refrigerator.

[1206] Hardware or software configuration

[1207] Imaging means

[1208] The terminal (refrigerator camera) is a device for taking high-resolution images, and periodically takes pictures of the inside of the refrigerator, for example, every 24 hours. This camera has wireless communication capabilities and sends image data to a server via the Internet.

[1209] Image analysis methods

[1210] The server uses a high-performance image recognition algorithm (e.g., YOLO, ResNet) to analyze the received image data. This algorithm classifies food by category and identifies the food based on the extracted features.

[1211] Data Management Measures

[1212] The server stores the identified food information in a database, which includes information such as the food ingredient classification, quantity, and location, and manages it so that users can easily check the inventory status.

[1213] E-commerce automatic ordering tool

[1214] The server has the ability to automatically order food items online based on data management methods. For example, if there is less than one bottle of milk, a new order is placed via a pre-registered e-commerce site API.

[1215] Proposal means

[1216] The server proposes recipes that the user can cook based on the data management method. The server analyzes inventory data, selects appropriate recipes, and sends them to the app. The user can check the proposed recipes on the app and start cooking right away.

[1217] Promotional information notification means

[1218] The server collects promotional information from nearby stores and notifies the user. This function presents the most economical option based on the user's location and the distance to the store. For example, a LINE notification can be sent to the user to let them know about special sales at a nearby supermarket.

[1219] Cloud data synchronization method

[1220] The server stores inventory information on the cloud and updates it in real time, allowing users to check the inventory status of their refrigerators even when they are out and about.

[1221] Specific examples

[1222] User A's experience

[1223] A camera installed in the refrigerator takes an image of the inside of the refrigerator at 8:00 AM and sends it to a server. The server analyzes the image and determines that there are three apples and two tomatoes. User A opens a dedicated app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "apple and tomato salad."

[1224] In the evening, User A notices that there is little yogurt in the refrigerator, but is notified that new yogurt will arrive, as the system has already automatically placed an order. Furthermore, User A receives a LINE notification that a nearby supermarket is having a special sale on oranges, allowing User A to go to the supermarket and complete their shopping efficiently.

[1225] Example prompts to input to the generative AI model

[1226] "A camera installed inside the refrigerator takes pictures of the inside at 8:00 AM and sends them to a server. The server then analyzes the images and displays inventory information on an app. With this app, users can check the inventory in their refrigerator in real time and receive recipe suggestions, automatic ordering, and notifications of advertised products. Please explain this system."

[1227] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1228] Step 1: Photographing the inside of the refrigerator using imaging means

[1229] The device (refrigerator camera) takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. This camera is positioned to cover the entire inside of the refrigerator and captures high-resolution images. The input is the image data of the inside of the refrigerator taken by the camera, and the output is the image file. In this step, the types and arrangement of food in the refrigerator are recorded in detail.

[1230] Step 2: Sending image data

[1231] The device (refrigerator camera) sends the captured image data to a server via the Internet. Here, the input is the captured image data, and the output is the image data sent to the server. This step requires highly reliable data transmission, and data is transferred using protocols such as HTTP.

[1232] Step 3: Image analysis

[1233] The server analyzes the received image data using an image analysis algorithm (e.g., YOLO, ResNet). The input is the image data sent to the server, and the output is food data identified by category. In this step, food items in the image are identified by category and features such as size, shape, and color are extracted. For example, the server accurately recognizes foods such as apples, tomatoes, and milk from the image and registers them in a database.

[1234] Step 4: Update data management

[1235] The server stores the identified food information in a database and updates the inventory information. The input is the food identified by category, and the output is the updated database. In this step, information such as food classification, quantity, and location is recorded in the database and a timestamp is added to allow tracking of inventory fluctuations over time.

[1236] Step 5: Check inventory information

[1237] The user checks the inventory information in the refrigerator in real time through a dedicated app. The app periodically connects to the server to retrieve and display the latest inventory information. The input is the inventory information retrieved from the server, and the output is the inventory information displayed to the user. When the user opens the app, specific inventory information, such as "3 apples, 2 tomatoes," is immediately displayed.

[1238] Step 6: Automatically order missing items online

[1239] The server periodically checks the inventory information in the database and automatically places an order online for any food items that are running low. The input is the updated inventory data, and the output is the order data sent to the e-commerce platform. For example, if there is only one bottle of milk left, the server sends the necessary information to the e-commerce site API to place an automatic order.

[1240] Step 7: Recipe suggestions

[1241] The server suggests recipes that can be made by the user based on inventory information. The input is inventory information in the database, and the output is suggested recipe information. The server analyzes the inventory data, selects appropriate recipes from the database, and sends them to the app. The user can then view specific recipe suggestions, such as "apple and tomato salad," on the app.

[1242] Step 8: Promotional Information Notification

[1243] The server collects advertising information from nearby stores and notifies the user. The input is the collected advertising information, and the output is the advertising information notified to the user. The server obtains advertising data from local supermarkets and compares it with inventory information to identify products to notify the user about. Furthermore, the server calculates the travel distance to the optimal store based on the user's location information, and sends the user, for example, information about special sales at nearby supermarkets via LINE notifications.

[1244] (Application example 1)

[1245] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1246] Conventional refrigerator inventory management systems are limited to efficient inventory management and shopping optimization within the home, and have not been applied to inventory management or product recommendation efficiency in physical stores. In addition, there is a lack of a system for store clerks to check inventory information in real time and make optimal product recommendations to customers, so improving inventory management efficiency and customer satisfaction are issues. Furthermore, there is a need for a means to increase customer purchasing motivation by appropriately notifying customers of advertising information and promotions.

[1247] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1248] In this invention, the server includes an imaging means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the imaging means and classifying the food by category, an inventory management means for managing inventory information for each food category recognized by the image analysis means, an automatic e-commerce ordering means for automatically ordering missing items based on the inventory management means, a recipe suggestion means for suggesting recipes available to the user based on the inventory management means, an advertised item notification means for collecting advertised item information from selected nearby stores and notifying the user, a visualization means for displaying in-store inventory information in real time on a display device worn by the user and checking inventory and suggesting products, an automatic replenishment means for automatically replenishing in-store inventory when inventory falls below a certain threshold, and a notification means for notifying the display device of advertising information and promotional information. This improves the efficiency of inventory management in physical stores, enables store staff to suggest optimal products to customers in real time, and improves customer satisfaction by effectively notifying them of advertising information and promotional information.

[1249] The "imaging means" is a device that takes images of the inside of a refrigerator or a store.

[1250] The "image analysis means" is a device or software that has the function of analyzing captured image data and identifying food or product categories.

[1251] The "inventory management means" is a device or software that has the function of managing inventory information recognized by the image analysis means and updating it in real time.

[1252] An "automated electronic commerce ordering means" is a device or software that has the function of automatically ordering products that are in short supply based on inventory information.

[1253] The "recipe suggestion means" is a device or software that has the function of suggesting recipes that can be used by the user based on inventory information.

[1254] The "advertised item notification means" is a device or software that has the function of notifying the user of advertised item information collected from designated nearby stores.

[1255] A "display device" is a display device worn by a user to visually confirm information.

[1256] The "visualization means" is a device or software that displays inventory information in real time through a display device and has the function of checking inventory and making product suggestions.

[1257] An "automatic replenishment means" is a device or software that has the function of automatically replenishing products when inventory falls below a certain threshold.

[1258] The "notification means" is a device or software that has the function of notifying the display device of advertising information or promotional information.

[1259] The present invention relates to a system for improving the efficiency of inventory management and product recommendations in a physical store. Detailed embodiments of the present invention will be described below.

[1260] System Overview

[1261] This system is comprised of an imaging means, an image analysis means, an inventory management means, an automatic e-commerce ordering means, a recipe suggestion means, an advertised product notification means, a visualization means, an automatic replenishment means, and a notification means. By linking these means, efficient inventory management, product suggestions, and advertising information notifications in physical stores become possible.

[1262] Photography and image analysis

[1263] The server first periodically captures images of the product shelves in the store using an imaging device. This capture is performed automatically at regular intervals. The captured image data is sent to the server and analyzed by the image analysis device. The image analysis device then classifies the food and product by category and stores the data in a database as inventory information.

[1264] Inventory management and e-commerce automatic ordering

[1265] The server updates the inventory information recognized by the image analysis means in real time using the inventory management means. The user can wear a display device and instantly check the current inventory information using the visualization means. Based on this inventory information, if the inventory falls below a certain threshold, the e-commerce automatic ordering means automatically places a replenishment order.

[1266] Recipe suggestions and promotional product notifications

[1267] The server is equipped with a recipe suggestion unit that suggests recipes available to the user based on inventory information. Furthermore, an advertised product notification unit notifies the user of advertised product information collected from designated nearby stores. The server calculates the user's location information and the travel distance to the store, and uses an algorithm to present the most economical options.

[1268] In-store use

[1269] Store staff wear display devices and use visualization tools to check inventory information in real time. For example, if an item on a shelf runs low, an automatic replenishment tool is activated to immediately replenish it. It is also possible to check whether the product desired by the customer is in stock and present alternative options.

[1270] Notification means

[1271] The advertising information and promotion information are displayed on the display device using the notification means, allowing the store staff to provide the customer with appropriate product information.

[1272] Examples of concrete examples and prompts

[1273] For example, when a store clerk wears a display device and activates the "Check Stock" command, real-time inventory information is displayed on the display. For example, it may show "Milk: 10, Eggs: 5, Bread: 2." Also, when the "Auto Order" command is executed, inventory below a threshold (for example, bread) is automatically ordered.

[1274] Below are some example prompts for a generative AI model:

[1275] Can you give us an example of a system that streamlines refrigerator inventory management? A system that meets the following requirements: 1. Periodically takes pictures of the inside of the refrigerator using a camera. 2. Analyzes the images and updates inventory information. 3. Checks inventory information in real time using a smartphone or smart glasses. 4. Automatically places e-commerce orders when inventory is low. 5. Suggests recipes that can be made. 6. Notifies users of advertised product information.

[1276] The above is the details of the embodiment of the invention. This system is expected to improve the efficiency of inventory management in stores and increase customer satisfaction.

[1277] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1278] Step 1:

[1279] The server periodically captures images of the product shelves in the store using an imaging device. It receives the time interval and camera position as input and obtains the captured image data as output. The images captured by this imaging device are saved in a storage directory.

[1280] Step 2:

[1281] The server sends the captured image data to the image analysis means, which receives the image data as input and outputs the analysis results. The image analysis means uses machine learning algorithms to identify each item in the image by category and stores the results in a database.

[1282] Step 3:

[1283] The server retrieves inventory information from the database and manages the latest inventory information using inventory management means. It receives analysis result data as input and saves updated inventory information in the database as output. This is done in real time by the inventory management means.

[1284] Step 4:

[1285] The server uses visualization means to display the latest inventory information via a display device worn by the user. The latest inventory information is acquired as input, and the inventory information displayed on the display is obtained as output. This allows the user to check the current inventory status.

[1286] Step 5:

[1287] When inventory information falls below a certain threshold, the server automatically orders the missing items using an automatic e-commerce ordering system. The server receives inventory information and a set threshold as input, and obtains data related to the automatic order as output. This allows for a rapid response when inventory shortages occur.

[1288] Step 6:

[1289] The server activates a recipe suggestion means that suggests available recipes to the user based on the inventory information. The server receives the inventory information as input and generates suggested recipe information as output. The user can check these suggested recipes on the display device.

[1290] Step 7:

[1291] Using the advertising product notification means, the server notifies the user of advertising information collected from the set nearby stores. It receives advertising information and the user's location information as input and generates an optimal advertising notification as output. The user can learn about economical options through the display device.

[1292] Step 8:

[1293] The advertising information and promotion information are displayed on the display device by the notification means. The advertising information is used as input and the notification on the display is used as output. This allows the user and store clerk to instantly check the latest promotion information.

[1294] Step 9:

[1295] The user checks inventory and makes product suggestions via a display device, and replenishes products or makes suggestions to customers based on the information displayed on the display.The system receives instructions from the display device as input and generates specific product suggestions and replenishment actions as output.

[1296] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1297] The present invention is a system that provides a more personalized dining experience by suggesting menu items that take into account not only refrigerator inventory management but also the user's emotional state. Detailed embodiments of this system are described below.

[1298] System Overview

[1299] This system includes a camera installed inside the refrigerator, image analysis, inventory management, automatic e-commerce ordering, recipe suggestion, advertisement notification, and an emotion engine that recognizes the user's emotions. By linking these mechanisms, users can grasp the inventory in their refrigerator in real time and receive efficient shopping and cooking suggestions that match their emotional state.

[1300] Photo taken inside the refrigerator

[1301] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, it takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. These pictures are used to record the types of food in the refrigerator and their arrangement.

[1302] Image data transmission and analysis

[1303] The captured image data is sent from the device (refrigerator camera) to a server via the Internet. The server analyzes the received images and uses an image recognition algorithm to identify the food by category. The analysis results are stored in a database and managed as inventory information.

[1304] Managing and checking inventory information

[1305] Users can check the inventory information in their refrigerator in real time through a dedicated app. The app retrieves the latest inventory information from the server and displays it to provide users with the current inventory status. For example, when a user opens the app, they can immediately see how many items are in the refrigerator.

[1306] Automatic EC ordering of missing items

[1307] Based on inventory information, food items that fall below a set threshold are automatically ordered online. The server periodically checks inventory information, and when the required food items become scarce, it automatically places an order with the registered e-commerce site. For example, if there is less than one bottle of milk, an order is automatically placed.

[1308] Linking recipe suggestions with an emotion engine

[1309] Based on inventory information and the user's emotional state, the server suggests recipes that the user can make. The emotion engine analyzes the user's facial expressions, voice, and bio-signals to identify their emotional state. For example, if it detects that the user is feeling stressed, it will suggest simple recipes that have a relaxing effect. The user can check the suggested recipes on the app and proceed with selecting and preparing the food. For example, if there is chicken and vegetables in the refrigerator and the user is tired, it will suggest a recipe such as "easy stir-fried chicken and vegetables."

[1310] Advertisement Notice

[1311] This system also includes a function to collect information about products advertised in nearby stores and notify users. The server obtains advertising information from nearby stores and compares it with inventory information to identify products that should be notified to users. It then calculates the user's location and the distance to the store, and uses an algorithm to present the most economical option. For example, a LINE notification may be sent to the user saying, "Oranges are on sale at a nearby supermarket. It's a good deal if you go and buy some."

[1312] Example: User B's experience

[1313] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User B opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[1314] Furthermore, the emotion engine detects that B is tired at the time and suggests other easy-to-make recipes. By suggesting easy-to-make menus such as "Easy Tomato and Cheese Pizza," B's burden is reduced. In addition, information about items on sale at nearby supermarkets is sent via LINE notifications, allowing for economical shopping.

[1315] In this way, the present invention, including the emotion engine, provides comprehensive support for the user's daily food management, shopping, and cooking, reducing waste and improving efficiency.

[1316] The processing flow will be explained below.

[1317] Step 1:

[1318] The device (refrigerator camera) takes pictures of the inside of the refrigerator periodically according to a set schedule, for example, at 8:00 AM and 8:00 PM every day.

[1319] Step 2:

[1320] The captured image data is sent from the device (refrigerator camera) to the server via the Internet. Once the transmission is complete, an acknowledgement message is sent.

[1321] Step 3:

[1322] The server saves the received image data in a directory for analysis. If the save is successful, it records that fact in the log.

[1323] Step 4:

[1324] The server then analyzes the stored image data using an image recognition algorithm, which identifies the food items in the image by category and identifies their type and quantity (for example, three apples, two tomatoes, etc.).

[1325] Step 5:

[1326] Based on the analysis results, the server updates the database with inventory information for each food category. The inventory information is updated in real time, ensuring that it is always up to date.

[1327] Step 6:

[1328] The user launches a dedicated app, which sends a request to the server to check stock availability.

[1329] Step 7:

[1330] The server retrieves the latest inventory information from the database and sends it back to the app, which displays the information to the user in real time through the app's UI.

[1331] Step 8:

[1332] The server periodically checks inventory to see if any food items are running low, and if they fall below a threshold, it generates an automatic order request for the item.

[1333] Step 9:

[1334] The server executes the automated order through the API of the e-commerce site, using shipping and payment information based on the user's pre-defined settings.

[1335] Step 10:

[1336] Based on the user's refrigerator inventory and special requests, the app executes an emotion recognition program and sends emotional data to the server.

[1337] Step 11:

[1338] The server selects appropriate recipes based on the user's emotional state. For example, if it detects that the user is feeling stressed, it will select simple, relaxing recipes.

[1339] Step 12:

[1340] The server sends the selected recipe to the app and displays it to the user, who can then review the suggested recipes and refer to cooking instructions as needed.

[1341] Step 13:

[1342] The server collects advertising information from local supermarkets, compares it with current inventory information, and identifies items that need to be purchased.

[1343] Step 14:

[1344] The server takes into account the user's location information, calculates travel distance and economic efficiency based on the displayed advertising information, and determines the most effective shopping method.

[1345] Step 15:

[1346] The server sends LINE notifications to users, advertising specific products and providing them with recommended store information for purchasing them. For example, it can notify users that oranges are on sale at a nearby supermarket, helping them shop in the most economical way.

[1347] Example: User B's experience

[1348] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User B opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[1349] Furthermore, the emotion engine detects that B is tired at the time and suggests other easy-to-make recipes. By suggesting easy-to-make menus such as "Easy Tomato and Cheese Pizza," B's burden is reduced. In addition, information about items on sale at nearby supermarkets is sent via LINE notifications, allowing for efficient shopping.

[1350] Example 2

[1351] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1352] In modern life, efficient refrigerator inventory management and consumption are important challenges for many households. In particular, reducing food waste, replenishing needed ingredients in a timely manner, and providing users with recipes tailored to their needs are difficult. Furthermore, there is a need for systems that can provide meal suggestions based on the user's emotional state and economical shopping options, but no systems exist that can comprehensively address these challenges.

[1353] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a camera means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the camera means and classifying the food by category, an inventory management means for managing inventory information for each food category recognized by the image analysis means, an EC automatic ordering means for automatically ordering missing items, an emotion analysis means for recognizing the emotional state of the user, a recipe suggestion means for suggesting recipes that the user can use based on the inventory management means, and an advertised item notification means for collecting advertised item information from nearby stores and notifying the user. This makes it possible to efficiently manage inventory in the refrigerator and provide personalized meal suggestions and economical shopping options according to the user's emotional state.

[1354] The "camera means" is a device that takes pictures of the inside of the refrigerator.

[1355] The "image analysis means" is a module that analyzes the captured image data and identifies the food by category.

[1356] The "inventory management means" is a system that manages inventory information for each food category recognized by the image analysis means.

[1357] The "EC automatic ordering means" is a module that automatically places online orders for items that are in short supply based on the inventory management means.

[1358] The "recipe suggestion means" is a system that suggests recipes that can be used by the user based on the inventory management means.

[1359] The "advertised item notification means" is a module that collects advertised item information from designated nearby stores and notifies the user of the information.

[1360] An "emotion analysis means" is a system for recognizing a user's emotional state.

[1361] The present invention is a system that not only manages inventory in a refrigerator but also proposes menus that take into account the emotional state of the user, thereby providing a more personalized dining experience. Detailed embodiments of this system are described below.

[1362] System Overview

[1363] This system includes a camera installed inside the refrigerator, image analysis, inventory management, automatic e-commerce ordering, recipe suggestion, advertisement notification, and emotion analysis that recognizes the user's emotions. By linking these means, the user can grasp the inventory in the refrigerator in real time and receive efficient shopping and cooking suggestions that match their emotional state.

[1364] Photo taken inside the refrigerator

[1365] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, it takes pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM every day. These pictures are used to record the types of food in the refrigerator and their arrangement.

[1366] Image data transmission and analysis

[1367] The captured image data is sent from the device (refrigerator camera) to a server via the Internet. The server analyzes the received images and uses an image recognition algorithm to identify the food by category. This analysis is performed using image recognition software such as TensorFlow and OpenCV. The results of this analysis are stored in a database and managed as inventory information.

[1368] Managing and checking inventory information

[1369] Users can check the inventory information in their refrigerator in real time through a dedicated app. The app retrieves the latest inventory information from the server and displays it to provide users with the current inventory status. For example, when a user opens the app, they can immediately see how many items are in the refrigerator.

[1370] Automatic EC ordering of missing items

[1371] Based on inventory information, food items that fall below a set threshold are automatically ordered online. The server periodically checks inventory information, and when the required food items become scarce, it automatically places an order with the registered e-commerce site (e.g., Amazon or Rakuten Ichiba). For example, an order is automatically placed when there is less than one bottle of milk left.

[1372] Linking recipe suggestions with an emotion engine

[1373] Based on inventory information and the user's emotional state, the server suggests recipes that the user can make. The emotion engine analyzes the user's facial expressions, voice, and bio-signals to identify their emotional state. For example, if it detects that the user is feeling stressed, it will suggest simple recipes that have a relaxing effect. The user can check the suggested recipes on the app and proceed with selecting and preparing the food. For example, if there is chicken and vegetables in the refrigerator and the user is tired, it will suggest a recipe such as "easy stir-fried chicken and vegetables."

[1374] Advertisement Notice

[1375] The server retrieves advertising information from nearby stores and compares it with inventory information to identify products that should be notified to the user. It then calculates the user's location and the distance to the store, and uses an algorithm to present the most economical option. For example, a LINE notification might be sent to the user saying, "Oranges are on sale at a nearby supermarket. It's a good deal if you stop by to buy some."

[1376] Example: User B's experience

[1377] A camera installed in the refrigerator takes an image of the inside at 8:00 AM and sends it to the server. The server analyzes the image and determines that there are three apples and two tomatoes. User B opens the app to make lunch and checks the inventory information. The app displays "3 apples and 2 tomatoes" and suggests "Apple and Tomato Salad" as a recipe.

[1378] Furthermore, the emotion engine detects that B is tired at the time and suggests other easy-to-make recipes. By suggesting easy-to-make menus such as "Easy Tomato and Cheese Pizza," B's burden is reduced. In addition, information about items on sale at nearby supermarkets is sent via LINE notifications, allowing for economical shopping.

[1379] Example prompts to input to a generative AI model:

[1380] Please provide a detailed explanation of the specific processing steps and operations of a system that manages refrigerator inventory and suggests menu items while also taking into account the user's emotions. Please also mention what hardware and software are used in the process, and what data processing and calculations are performed.

[1381] In this way, the present invention, including the emotion engine, provides comprehensive support for the user's daily food management, shopping, and cooking, reducing waste and improving efficiency.

[1382] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1383] Step 1: Photographing the inside of the refrigerator

[1384] The device (refrigerator camera) periodically takes pictures of the inside of the refrigerator according to a set schedule. For example, take pictures of the inside every day at 8:00 AM and 8:00 PM. The input to this step is the schedule setting information, and the output is the image data of the inside of the refrigerator. The camera operates and automatically records the food in the refrigerator at the specified time.

[1385] Step 2: Sending image data

[1386] The device (refrigerator camera) sends the captured image data to a server via the Internet. The input of this step is the captured image data, and the output is the image data sent to the server. The camera uses a network connection to upload the captured image data to the server.

[1387] Step 3: Receiving image data

[1388] The server receives image data sent via the Internet. The input of this step is the image data sent from the device, and the output is the storage of the received image data. The server receives the image data using a specific API endpoint.

[1389] Step 4: Image analysis

[1390] The server applies image analysis algorithms to the received images to recognize food in the image. The input of this step is the received image data, and the output is the recognized food data. Specifically, the server uses software such as TensorFlow and OpenCV to identify foods such as apples and tomatoes.

[1391] Step 5: Update inventory data

[1392] Based on the analysis results, the server updates the database with the food category and quantity. The input to this step is the recognized food data, and the output is the updated inventory information. For example, if the analysis finds that there are 3 apples and 2 tomatoes, this information is registered in the database.

[1393] Step 6: Get inventory information

[1394] The user opens the dedicated app and sends a request to the server to check the inventory information in the refrigerator. The input of this step is the user's request data, and the output is a request to obtain inventory information. The user operates the app interface to send the request to the server.

[1395] Step 7: Submit inventory information

[1396] The server retrieves the latest inventory information from the database and sends it to the user's app. The input to this step is a request to retrieve inventory information, and the output is the latest inventory information. The server queries the inventory data and sends the retrieved information to the user's app.

[1397] Step 8: View inventory information

[1398] The user checks the inventory information in the app. The input of this step is the inventory information sent from the server, and the output is the displayed inventory information. The user views the inventory information on the app screen.

[1399] Step 9: Check inventory

[1400] The server periodically checks inventory information and identifies foods that have fallen below a set threshold. The input to this step is the inventory information in the database, and the output is a list of items that are in short supply. The server scans the database and extracts the items that are in short supply.

[1401] Step 10: Execute automatic orders

[1402] When the server identifies missing items, it automatically places an order with the registered e-commerce site. The input for this step is a list of missing items, and the output is order confirmation information. The server calls the e-commerce site's API based on the list of missing items and performs the order procedure.

[1403] Step 11: Order confirmation

[1404] The user can check the automatically ordered food and its status on the app. The input of this step is the order confirmation information, and the output is the displayed order confirmation information. The user checks the order status on the app screen.

[1405] Step 12: Obtaining Emotion Data

[1406] The user's emotional state is acquired by a facial expression recognition camera or a biometric signal sensor and transmitted to the server via the device. The input of this step is emotional data, and the output is the emotional data transmitted to the server. The device detects the user's emotional data and transmits it to the server.

[1407] Step 13: Sentiment Analysis

[1408] The server analyzes the received emotion data to identify the user's current emotional state. The input of this step is the received emotion data, and the output is the identified emotional state. The server performs an operation to evaluate the user's emotional state using an emotion analysis algorithm.

[1409] Step 14: Generate recipe suggestions

[1410] The server uses a generative AI model to create an appropriate recipe based on inventory information and the user's emotional state. The inputs for this step are inventory information and the identified emotional state, and the output is a suggested recipe. The server uses the generative AI model to generate a personalized recipe.

[1411] Step 15: Serve the recipe

[1412] The user can view the suggested recipes in the app, select a dish, and prepare it. The input for this step is the suggested recipe, and the output is the displayed recipe information. The user browses the recipe on the app screen and proceeds with cooking.

[1413] Step 16: Collect advertising information

[1414] The server collects advertising information for nearby stores that is publicly available online. The input for this step is advertising information for nearby stores, and the output is the collected advertising data. The server uses scraping technology and APIs to collect advertising information.

[1415] Step 17: Check against inventory information

[1416] The server compares the collected advertising information with inventory information and identifies information that is useful to the user. The input for this step is the collected advertising information and inventory information, and the output is the advertising information to be notified. The server compares the inventory data with the advertising information and identifies the notification target.

[1417] Step 18: Execute Notification

[1418] The server identifies the most suitable store and product based on the user's location information and sends a notification. The input for this step is the advertising information to be notified and the user's location information, and the output is the sent notification information. The server provides useful advertising information to the user via LINE notifications, etc.

[1419] (Application example 2)

[1420] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1421] Today's busy consumers often have little time to manage the food in their refrigerators or purchase food, making it difficult to avoid food waste and irregular eating habits. Furthermore, the lack of a system that can provide appropriate meal suggestions based on users' emotional state is a problem, leading to a decline in the quality of users' diets. Furthermore, there is a need for efficient food management and economical shopping, such as notifications of low inventory in the refrigerator and advertised products at nearby stores.

[1422] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1423] In this invention, the server includes a camera means for taking pictures of the inside of the refrigerator, an image analysis means for analyzing image data taken by the camera means and identifying food categories, an inventory management means for managing inventory information for each food category recognized by the image analysis means, an EC automatic ordering means for automatically ordering missing items based on the inventory management means, a recipe suggestion means for suggesting recipes available to the user based on the inventory management means, an emotion analysis means for analyzing the user's emotional state and suggesting recipes or meal plans based on that state, a food delivery ordering means for automatically placing an order with a food delivery service if ingredients needed for a suggested recipe or meal plan are missing, and an advertised item notification means for collecting advertised item information from designated nearby stores and notifying the user. This allows the user to efficiently manage inventory in the refrigerator, receive meal suggestions based on the user's emotional state, and automatically order necessary ingredients from a delivery service, thereby reducing waste and achieving efficient food management.

[1424] The "camera means" is a camera device for taking pictures of food in the refrigerator.

[1425] The "image analysis means" is a device or software for analyzing captured image data and identifying food items by category.

[1426] The "inventory management means" is a system for managing inventory information for each food category recognized by the image analysis means.

[1427] "EC automatic ordering means" means a device or software for automatically ordering shortage items via the Internet based on the inventory management means.

[1428] The "recipe suggestion means" is a device or software that suggests recipes that can be used by the user based on the inventory management means.

[1429] An "emotion analysis means" is a device or software that analyzes a user's emotional state and suggests recipes or meal plans based on that state.

[1430] "Food delivery ordering device" means a device or software that automatically places an order with a food delivery service when ingredients required for a suggested recipe or meal plan are missing.

[1431] The "advertised item notification means" is a device or software for collecting advertised item information from designated nearby stores and notifying the user of the information.

[1432] The present invention provides a food delivery system that takes into account the emotional state of the user in addition to the inventory management function in the refrigerator. Detailed embodiments of this system will be described below.

[1433] Hardware and software used

[1434] 1. Refrigerator camera means: A camera device for photographing food in the refrigerator.

[1435] 2. Smartphone: A device that allows users to receive emotion recognition and notifications.

[1436] 3. Server:

[1437] Image analysis means: A device that analyzes captured image data and identifies food categories.

[1438] Inventory management method: A system that manages analyzed inventory information.

[1439] Sentiment analysis means: A system that analyzes the user's emotional state and suggests recipes and meal plans based on that state.

[1440] Generative AI model: A model that generates recipes based on the user's emotional state and inventory information.

[1441] Food delivery ordering method: A system that automatically orders the necessary ingredients from a delivery service.

[1442] Advertised item notification means: A device that collects advertised item information from designated nearby stores and notifies the user.

[1443] System processing flow

[1444] The overall flow of the system is as follows: First, the camera means inside the refrigerator periodically takes pictures of the inside of the refrigerator. The captured image data is sent to a server via the Internet, where the image analysis means analyzes it and identifies the type and quantity of food stored inside. This data is then stored in the inventory management means.

[1445] When a user launches the application on their smartphone, the emotion analyzer captures and analyzes the user's emotional state using the camera and microphone. For example, if the user is feeling stressed, the emotion analyzer sends a prompt to the generative AI model to generate a "relaxing meal suitable for a stressed user."

[1446] The generative AI model generates an optimal recipe based on the prompts and the inventory information in the refrigerator. The recipe suggestion means suggests this recipe to the user, and if necessary ingredients are missing, the food delivery ordering means automatically places an order with a delivery service.

[1447] Users can review these suggestions and view inventory information and suggested recipes at any time. The advertising notification tool also collects advertising information from nearby stores, providing users with the latest economical options.

[1448] Specific examples

[1449] For example, a camera means installed inside a refrigerator takes an image of the inside of the refrigerator at 8:00 AM and sends it to a server. The server analyzes the image and recognizes that there are two apples and one piece of chicken inside. After that, when a user launches the app on their smartphone, the emotion analysis means analyzes the user's emotional state. If the user is feeling stressed, the generative AI model will receive the following prompt:

[1450] "Please suggest a meal that will help the user relax when they are feeling stressed. Below is the inventory in the refrigerator: 2 apples, 1 piece of chicken."

[1451] Based on this prompt, the generative AI model suggests recipes such as "Relaxed Apple and Chicken Salad." If other ingredients required for this recipe are missing, the food delivery ordering tool automatically places an order with the delivery service.

[1452] In this way, users can manage their refrigerator inventory, receive personalized meal suggestions based on their emotional state, and even have missing ingredients automatically ordered from a delivery service.

[1453] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1454] Step 1:

[1455] Photo taken inside the refrigerator

[1456] Input: Food status in the refrigerator, shooting schedule

[1457] Processing: The camera means installed in the terminal periodically takes pictures of the inside of the refrigerator. For example, take pictures of the inside of the refrigerator at 8:00 AM and 8:00 PM.

[1458] Output: Image data of the inside of the refrigerator

[1459] Step 2:

[1460] Sending image data

[1461] Input: Image data of the inside of the refrigerator

[1462] Processing: The device sends the captured image data to a server via the Internet.

[1463] Output: Image data sent to the server

[1464] Step 3:

[1465] Image data analysis

[1466] Input: Submitted image data

[1467] Processing: The image analysis means installed on the server analyzes the transmitted image data and identifies each food item by category. For example, the food items are classified as apples, chicken, etc.

[1468] Output: Inventory data identified for each food category

[1469] Step 4:

[1470] Inventory management

[1471] Input: Identified inventory data

[1472] Processing: The inventory management means of the server stores the identified inventory data in a database and manages it as inventory information.

[1473] Output: Updated inventory information

[1474] Step 5:

[1475] User emotional state analysis

[1476] Input: User's facial expressions, voice, and biometric signals

[1477] Processing: When a user launches the app on their smartphone, the device's camera and microphone capture the user's emotional state and send it to the server, where the server's emotion analysis means analyzes it and generates the user's emotional state data.

[1478] Output: Emotional state data

[1479] Step 6:

[1480] Recipe generation and suggestions

[1481] Input: inventory information, emotional state data

[1482] Processing: The server's AI model generates a recipe suitable for the user based on inventory information and the user's emotional state data. For example, a recipe may be generated based on a prompt such as "a meal that will have a relaxing effect when the user is feeling stressed." The generated recipe is then suggested to the user through the recipe suggestion mechanism.

[1483] Output: Suggested recipe

[1484] Step 7:

[1485] Automatic ordering of missing ingredients

[1486] Input: Suggested recipes, inventory information

[1487] Processing: If there is a shortage of ingredients in stock based on the recipe suggested by the recipe suggestion means, the food delivery ordering means of the server automatically orders the necessary ingredients via a delivery service.

[1488] Output: Order confirmation notification

[1489] Step 8:

[1490] Advertisement notification

[1491] Input: Updated inventory information, user location information

[1492] Processing: The server's advertisement notification means collects advertisement information from the set nearby stores and transmits advertisement information to the user showing the most economical options based on the user's location information.

[1493] Output: Advertisement notification

[1494] Step 9:

[1495] User confirmation

[1496] Input: Suggested recipes, order confirmation notifications for missing ingredients, promotional product notifications

[1497] Processing: The user checks the suggested recipes, order confirmation notifications, and promotional product notifications on their smartphone, allowing them to stay informed of suggested dishes, purchased ingredients, and special promotional products.

[1498] Output: User verification data

[1499] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1500] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1501] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1502] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1503] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1504] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1505] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1506] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1507] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1508] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1509] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server....

Claims

1. a camera means for taking pictures of the inside of the refrigerator; image analysis means for analyzing image data captured by the camera means and classifying the food items by category; an inventory management means for managing inventory information for each food category recognized by the image analysis means; an EC automatic ordering means for automatically ordering shortage items based on the inventory management means; a recipe suggestion means for suggesting recipes that can be used by the user based on the inventory management means; The system includes an advertisement product notification means for collecting advertisement product information from designated nearby stores and notifying the user of the information.

2. 2. The system according to claim 1, wherein the advertised product notification means includes an algorithm that calculates the user's location information and the distance traveled to the store, and presents the most economical option.

3. The system according to claim 1, wherein the inventory management means stores data on the cloud, enabling inventory information to be checked in real time.

Images