system

Abstract

Provide a system. A means for acquiring information about ingredients using a sensor installed inside the refrigerator; means for transmitting the ingredient information to a server; A means for the server to monitor the expiration date of ingredients and notify the user when it determines that the expiration date is approaching; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Food waste in modern households is a significant issue, particularly when excessive food stockpiles in refrigerators end up past their expiration dates. This problem arises from the time and effort required to manually manage expiration dates and oversights. Another problem is the lack of timely promotional information to promote efficient food use. Another issue is the lack of demand forecasting and supply chain optimization using data on food ingredients stored in refrigerators. The present invention aims to effectively solve these problems. [Means for solving the problem]

[0005] The present invention is a system that solves the above problems by the following means. First, a sensor installed in the refrigerator acquires food ingredient information and sends that information to a server. Next, the server monitors the expiration dates of food ingredients and notifies the user when it determines that the expiration date is approaching. Furthermore, the server acquires sales promotion information from affiliated retailers and notifies the user of sales promotion information related to food ingredients that are approaching their expiration date. Finally, the server analyzes the food ingredient data in the refrigerator, generates data on consumer behavior based on region and time of day, and provides this data to other companies to forecast demand and optimize the entire supply chain. This reduces food waste and makes it possible to use ingredients more efficiently.

[0006] A "sensor" is a device that automatically acquires information about ingredients inside a refrigerator.

[0007] The "server" is a central computer system that receives the food ingredient information sent from the refrigerator and manages and analyzes the data.

[0008] "Ingredient information" refers to data such as the name of the ingredient, expiration date, and quantity.

[0009] "Use by" refers to the date by which food can be safely consumed.

[0010] "Monitoring" is the process of constantly checking ingredient information based on certain rules.

[0011] A "notification" is an action to convey information to a user according to specified conditions.

[0012] "User" refers to a person who uses a refrigerator and receives notifications and information through an application or other means.

[0013] "Retailer" refers to a company or store that sells food ingredients.

[0014] "Promotional Information" refers to information about special offers and sales offered by retailers.

[0015] "Data" analysis is the process of collecting and analyzing information about ingredients in the refrigerator and converting it into valuable information.

[0016] "Demand forecasting" refers to predicting future consumption of food ingredients.

[0017] The "supply chain" refers to the entire process from the production of food ingredients to their delivery to consumers.

[0018] "Optimization" refers to the adjustment of resources and processes to maximize their efficiency. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0020] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0021] First, the terms used in the following description will be explained.

[0022] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0023] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0024] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0025] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0026] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0027] [First embodiment]

[0028] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0029] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0030] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0031] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0032] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0033] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0034] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0035] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0036] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0037] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0038] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0039] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0040] The following describes in detail an embodiment of the present invention. The system comprises a sensor installed in a refrigerator, a terminal linked to the refrigerator in each home, and a server.

[0041] Food Registration

[0042] First, we will explain the operation when a user stores new ingredients in the refrigerator. When a user places an ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode. This barcode information is sent to the terminal (a sensor system inside the refrigerator), and information such as the ingredient's name, expiration date, and quantity is automatically registered. The terminal saves this information in the refrigerator's internal memory and sends the registration data to a server.

[0043] Monitoring expiration dates of ingredients

[0044] The server periodically receives and stores information about ingredients sent from the refrigerator. It also monitors the expiration dates of ingredients and generates data to notify the user when the expiration date falls within a certain period of time (e.g., three days) from the set date.

[0045] Expiration date notice

[0046] The notification data generated by the server is immediately sent to the device (smartphone app). The device then notifies the user of a list of ingredients that are approaching their expiration date. This allows the user to avoid overlooking ingredients that are approaching their expiration date and consume them appropriately. As a result, food waste can be reduced.

[0047] Providing promotional information

[0048] The server also periodically receives and stores sales information from affiliated retailers. For food items nearing their expiration date, the server searches for related sales information and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and provides specific information, allowing the user to make purchases at an economically advantageous time.

[0049] Data provision and supply chain optimization

[0050] The server periodically collects and analyzes data on ingredients stored in the refrigerator, generating consumption patterns based on region and time of day. This consumption data is provided to other partner companies, who can use it to forecast demand and optimize the entire supply chain.

[0051] Specific examples

[0052] 1. Example of food ingredient registration

[0053] When a user buys milk and puts it in the refrigerator, they scan the barcode.

[0054] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[0055] 2. Example of expiration date notification

[0056] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[0057] The device displays a notification to the user through a smartphone app.

[0058] 3. Examples of sales promotion information provided

[0059] The server obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[0060] The device sends the user a sale notification such as "20% off milk."

[0061] 4. Data provision example

[0062] The server analyzes regional milk consumption data and identifies periods when consumption increases in specific regions.

[0063] The server provides the data to partner companies, which then adjust their supply plans.

[0064] Through the above-described embodiment, the present system can contribute to reducing food waste, optimizing consumption behavior, and optimizing the supply chain.

[0065] The processing flow will be explained below.

[0066] Food Registration

[0067] Step 1:

[0068] When a user places purchased ingredients in the refrigerator, they scan the barcode of the ingredients with a dedicated barcode scanner.

[0069] Step 2:

[0070] The terminal (a sensor system inside the refrigerator) receives the barcode and stores the food ingredient information (name, expiration date, quantity, etc.) in a database.

[0071] Step 3:

[0072] The terminal transmits the ingredient information to the server.

[0073] Monitoring expiration dates of ingredients

[0074] Step 1:

[0075] The server periodically (e.g., once a day) receives the latest ingredient information from each refrigerator.

[0076] Step 2:

[0077] The server monitors the expiration date of the ingredients received, and sets a "warning flag" if the expiration date falls within a certain period of time (e.g., three days) from the set date.

[0078] Expiration date notice

[0079] Step 1:

[0080] The server generates a list of ingredients whose expiration date is approaching and searches for ingredients that satisfy the notification conditions.

[0081] Step 2:

[0082] The server generates notification data and sends it to the device (smartphone app).

[0083] Step 3:

[0084] The device sends a notification to the user's smartphone and displays a list of ingredients that are approaching their expiration date.

[0085] Providing promotional information

[0086] Step 1:

[0087] The server periodically obtains sales information from affiliated stores and stores it in a database.

[0088] Step 2:

[0089] The server searches for sales promotion information related to food ingredients that are close to their expiration date.

[0090] Step 3:

[0091] The device notifies the user of promotional information. Specifically, the smartphone app notifies the user that "XX is on sale."

[0092] Data provision and supply chain optimization

[0093] Step 1:

[0094] The server periodically collects and analyzes data on ingredients in the refrigerator.

[0095] Step 2:

[0096] The server generates data on consumer behavior based on region and time of day.

[0097] Step 3:

[0098] The server provides the generated consumer behavior data to partner companies, who then use this data to forecast demand and optimize their supply chains.

[0099] Through the above processing steps, this system effectively manages ingredients, reduces food waste, and optimizes the supply chain.

[0100] Example 1

[0101] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0102] In modern society, managing food expiration dates is an important issue. However, many households do not adequately manage expiration dates, resulting in a large amount of food being wasted. In addition, appropriate promotional information is not provided for food that is approaching its expiration date, resulting in economic losses. Furthermore, data on consumer behavior is not being fully utilized, resulting in supply chain optimization.

[0103] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0104] In this invention, the server includes means for acquiring ingredient information using a detector installed in the food preservation device, means for transmitting the ingredient information to a computing device via a communication device, means for the computing device to monitor the expiration dates of ingredients and notify the user when it determines that the expiration date is approaching, means for the computing device to acquire sales promotion information from affiliated retailers and notify the user of sales promotion information related to ingredients that are approaching their expiration date, and means for the computing device to analyze the ingredient data in the food preservation device, generate data on consumer behavior based on region and time period, and provide the data to other companies. This makes it possible to reduce food waste and optimize consumer behavior.

[0105] A "food preservation device" is a device used to preserve food, such as a refrigerator.

[0106] A "detector" is a device such as a sensor or barcode scanner that obtains information about ingredients.

[0107] "Ingredient information" refers to information such as the name of the ingredient, expiration date, and quantity.

[0108] "Communication device" refers to a network device for transmitting data from a terminal to a server.

[0109] "Computing device" refers to a server or computer used to receive, store, and analyze ingredient information.

[0110] "Notification" refers to a means by which the server notifies users of food ingredients that are nearing their expiration date or promotional information.

[0111] "Affiliated retailer" refers to a store that sells ingredients or products that have a cooperative relationship with the system.

[0112] "Sales promotion information" refers to sales information and campaign information for sales promotion.

[0113] "Memory device" refers to data storage for saving acquired ingredient information and analysis results.

[0114] "Means of visually displaying information" refers to means of showing information to users via smartphone apps, displays, etc.

[0115] "Consumption behavior data" refers to food consumption patterns based on region and time of day.

[0116] "Means of providing data to other companies" refers to means of providing the consumer behavior data analyzed by the server to partner companies via APIs or other means.

[0117] The following describes in detail a specific embodiment of the present invention. The system is comprised of a detector installed in the food preservation device, a communication terminal connected to each household's food preservation device, and a computing device for data processing. This reduces food waste and optimizes consumption behavior.

[0118] Acquiring ingredient information

[0119] When a user places new ingredients in the food storage device, they use a dedicated barcode scanner to read the barcode of the ingredient. This operation obtains information about the ingredient, such as its name, expiration date, and quantity. The device receives this information via a detector in the food storage device.

[0120] Registering and saving food information

[0121] When the terminal receives the barcode information, it automatically retrieves the corresponding ingredient information from the database. The retrieved information is stored in the internal memory of the food storage device and then transmitted to the computing device via the Internet. The computing device then stores the received ingredient information in the database.

[0122] Expiration date monitoring

[0123] The computing device periodically monitors expiration dates based on the food ingredient information sent from the food storage device. When an ingredient approaches its expiration date, the computing device generates a notification message. This message is sent from the computing device to a terminal (smartphone app).

[0124] Expiration date notification

[0125] The notification data generated by the computing device is immediately sent to the terminal, which then notifies the user of a list of ingredients that are approaching their expiration dates, allowing the user to consume ingredients appropriately without missing the expiration dates.

[0126] Providing promotional information

[0127] The computing device periodically receives and stores sales information from affiliated retailers. It searches for sales information related to food items with approaching expiration dates and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and provides specific information.

[0128] Data analysis and provision

[0129] The computing device periodically collects and analyzes food data stored in the food storage device to generate consumption patterns based on region and time of day. This data is provided to partner companies, allowing them to optimize their supply chains based on demand forecasts.

[0130] Specific examples

[0131] 1. Example of food ingredient registration

[0132] When a user purchases milk and places it in the food storage device, they scan the barcode.

[0133] The terminal obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the computing device.

[0134] 2. Example of expiration date notification

[0135] A computing device monitors the expiration date of the milk and generates a notification on the seventh day saying, "Expires in two days."

[0136] The device displays a notification to the user through a smartphone app.

[0137] 3. Examples of sales promotion information provided

[0138] The computing device obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[0139] The device sends the user a sale notification such as "20% off milk."

[0140] 4. Data provision example

[0141] The computing device analyzes regional milk consumption data and identifies when consumption will increase in certain areas.

[0142] The computing device provides the data to partner companies, which then adjust their supply plans.

[0143] Prompt Sentence Examples

[0144] Please tell me in detail the steps by which the sensors in the refrigerator obtain food information.

[0145] What is the algorithm used by the server to generate expiration notifications?

[0146] Please tell us specifically how you will implement a system to obtain sale information and notify users.

[0147] By inputting these prompts into a generative AI model, we can obtain answers such as detailed explanations of each function and analysis of the algorithms.

[0148] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0149] Step 1:

[0150] When a user places a new ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode, which then acquires barcode information (input). The barcode information contains the ingredient's identification code.

[0151] Specific action: The user presses the button on the barcode scanner and reads the barcode on the milk.

[0152] Step 2:

[0153] The terminal receives the barcode information and retrieves detailed information such as the ingredient name, expiration date, and quantity from the database based on the ingredient's identification code (data processing). The retrieved ingredient information (output) is stored in the internal memory and simultaneously sent to the server.

[0154] Specific operation: The terminal receives the barcode "1234567890123", retrieves the information "Milk, expiration date: 10 days later, quantity: 1 liter" from the database, stores it in its internal memory, and then sends it to the server.

[0155] Step 3:

[0156] The server receives the food ingredient information sent from the terminal and stores it in a database (input). It periodically scans the stored data and monitors the expiration date (output).

[0157] What happens: The server stores the information "Milk, expiration date: 10 days later, quantity: 1 liter" in a database and runs a database query at 2 AM every day to scan for the expiration date.

[0158] Step 4:

[0159] The server detects ingredients whose expiration date is approaching (data calculation). If the expiration date is within a certain period of time (e.g., three days before) from the set date, it generates data to notify the user (output).

[0160] Specific operation: The server executes a database query to extract ingredients with expiration dates within three days. For example, it generates notification data such as "Milk expires in two days."

[0161] Step 5:

[0162] The server immediately sends the generated notification data to the device (smartphone app) (input). The device receives the notification data and displays it to the user (output).

[0163] Specific operation: The server sends notification data to the device's push notification API, stating that the milk's expiration date is in two days. The device displays the data on the smartphone app.

[0164] Step 6:

[0165] The server obtains sales information through the API of partner retailers and periodically saves it (input). It searches for sales information related to food items that are close to their expiration date and sends that information to the terminal (output).

[0166] Specific operation: The server retrieves the "20% off milk" sale information from the API of the partner retailer, associates it with information about milk that is nearing its expiration date, and notifies the user.

[0167] Step 7:

[0168] The server periodically collects consumption data on food items stored in the refrigerator and generates consumption behavior patterns based on region and time of day (data analysis: input). The generated consumption behavior data is provided to partner companies (output).

[0169] Specific operation: The server performs monthly data analysis and sends the results of "analysis of peak milk consumption in specific regions" to partner companies via API.

[0170] This allows the system to manage ingredients, notify expiration dates, provide economical purchasing timing, and contribute to supply chain optimization.

[0171] (Application example 1)

[0172] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0173] In managing food in refrigerators, there is a need to reduce the time and errors involved in manually entering food information, efficiently monitor expiration dates, and reduce food waste. It is also expected that the economic burden will be reduced by providing appropriate promotional information for food items approaching their expiration dates in a timely manner. Furthermore, there is a demand for providing companies with data on consumer behavior based on region and time of day to optimize supply plans.

[0174] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0175] In this invention, the server includes a means for acquiring information about ingredients using sensors installed in the refrigerator, a means for transmitting the ingredient information to the server, a means for the server to monitor the expiration dates of ingredients and notify the user when the expiration date is approaching, and a means for the user to scan barcodes using the camera on their smart device when storing ingredients in the refrigerator and automatically register the ingredient information. This eliminates the need to manually enter ingredient information, streamlines expiration date management, and reduces food waste. Furthermore, by providing promotional information at the appropriate time, it is possible to reduce financial burdens, and by providing companies with data on consumer behavior based on region and time of day, it is possible to optimize supply plans.

[0176] A "sensor installed inside a refrigerator" is a device that monitors the environment inside the refrigerator and obtains information about ingredients (e.g., location, temperature, weight, etc.).

[0177] "Ingredient information" is detailed data about ingredients, such as the name, quantity, expiration date, and barcode information of the ingredients.

[0178] A "server" is a computer system that communicates with external devices via a network and stores, processes, and analyzes data.

[0179] A "use-by date" is the date by which food is considered safe to eat, and quality and safety may not be guaranteed after that date.

[0180] A "notification" is a message or alert that the system uses to inform the user.

[0181] A "smart device" is an electronic device that is connected to the Internet and has advanced functions that allow it to run a variety of applications.

[0182] A "camera" is a device used to take images and videos, and is also used to read barcodes.

[0183] A "barcode" is a method of expressing information by arranging numbers and letters in a specific pattern that can be read by a machine.

[0184] "Automatic registration" is a process in which a machine or system automatically enters and stores information without human intervention.

[0185] The following describes in detail specific embodiments of the present invention. The system comprises a sensor installed in a refrigerator, a terminal (smart device) that connects to the refrigerator in each home, and a server.

[0186] Food Registration

[0187] When a user places new ingredients in the refrigerator, they use the camera on their smart device to read the barcode of the ingredient. This barcode information is sent to the terminal, and information such as the ingredient's name, expiration date, and quantity is automatically registered. The terminal stores this information in the refrigerator's internal memory and simultaneously sends it to the server. This eliminates the need for manual input and prevents input errors.

[0188] Monitoring expiration dates of ingredients

[0189] The server periodically receives and stores information about ingredients sent from the refrigerator. It also monitors the expiration dates of ingredients and generates data to notify the user when the expiration date falls within a certain period of time (e.g., three days) from the set date.

[0190] Expiration date notice

[0191] The notification data generated by the server is immediately sent to the device. The device displays a list of ingredients that are approaching their expiration date and provides audio and visual notifications to the user. This allows the user to avoid overlooking expiration dates and consume ingredients appropriately without wasting them.

[0192] Providing promotional information

[0193] The server also periodically receives and stores sales information from affiliated retailers. For food ingredients that are nearing their expiration date, the server searches for related sales information and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and displays detailed information. This allows the user to make purchases at an economically advantageous time.

[0194] Data provision and supply chain optimization

[0195] The server periodically collects and analyzes data on ingredients stored in the refrigerator, generating consumption patterns based on region and time of day. This consumption data is provided to other partner companies, who can use it to forecast demand and optimize the entire supply chain.

[0196] Hardware and software used

[0197] The following hardware and software are used to implement this system:

[0198] Smart Devices: Use your smartphone or smart glasses to scan the barcode with the built-in camera.

[0199] Server: A computer system for receiving, storing, analyzing, and notifying data.

[0200] Python: A scripting language used for data processing and analysis on the server.

[0201] OpenCV: A library for camera operation and barcode scanning.

[0202] Requests: A library for HTTP communication with the server.

[0203] Specific examples

[0204] Example of ingredient registration:

[0205] When a user purchases milk and puts it in the refrigerator, they scan the barcode using the camera on their smart device.

[0206] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[0207] Example of expiration date notification:

[0208] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[0209] The terminal displays a notification to the user on the smart device.

[0210] Examples of promotional information provided:

[0211] The server obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[0212] The device sends the user a sale notification such as "20% off milk."

[0213] Data provided example:

[0214] The server analyzes regional milk consumption data and identifies periods when consumption increases in specific regions.

[0215] The server provides the data to partner companies, which then adjust their supply plans.

[0216] Prompt Sentence Examples

[0217] "Please use the camera to read the barcode inside the refrigerator and register the new food information."

[0218] "Show me notifications about ingredients approaching their expiration date"

[0219] "Get relevant sales information and notifications on your smart device."

[0220] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0221] Step 1:

[0222] When a user places new ingredients in the refrigerator, they scan the barcode using the smart device's camera. At this time, the smart device receives the barcode image as input and extracts the barcode information. Specifically, the camera captures the image and uses the OpenCV library to detect and read the barcode. Once the barcode information is obtained, it is processed into ingredient information including the ingredient's name, expiration date, quantity, etc. and saved on the device.

[0223] Step 2:

[0224] The terminal sends the ingredient information acquired in step 1 to the server. The data sent includes the ingredient's barcode information, name, expiration date, quantity, etc. This information is sent to the server as an HTTP request, and the terminal processes to receive the response (success / failure status code) from the server.

[0225] Step 3:

[0226] The server stores the received ingredient information in a database. This database manages detailed information about ingredients stored in each household's refrigerator. The input ingredient information is added to the database, and the storage result is sent to the terminal as a response.

[0227] Step 4:

[0228] The server periodically monitors the food ingredient information in the database and checks the expiration date of each ingredient. When a specified period (e.g., three days before the expiration date) has passed, notification data for that ingredient is generated. The input food ingredient information is used as the source data, and an algorithm determines whether the expiration date is approaching and outputs the notification data.

[0229] Step 5:

[0230] The notification data generated by the server is immediately sent to the device. The device then displays this notification data to the user. For example, a message such as "The expiration date is in two days" can be displayed on the smart device's display to clearly communicate the information to the user. The notification message can be displayed as a voice notification or a visual pop-up.

[0231] Step 6:

[0232] The server periodically receives sales information from affiliated retailers and stores it in a database. Here, the input is the retailer's sales information (e.g., 20% off milk), and the server processes the sales information to add and update it in the database.

[0233] Step 7:

[0234] The server associates sale information with food ingredient information and searches the database for sale information related to food ingredients that are close to their expiration date. For example, if milk is close to its expiration date, the server searches for related "20% off milk" information and generates it as notification data.

[0235] Step 8:

[0236] The server generates sales promotion information and sends it to the terminal, which then displays it to the user. The user can see the sales promotion information on the smart device display, such as "There are related products on sale," and is encouraged to make purchases at economically advantageous times.

[0237] Step 9:

[0238] The server periodically collects and analyzes food data from the refrigerator and generates consumption behavior patterns based on region and time of day. Here, food information data is used as input and consumption patterns are output using statistical analysis and machine learning models. The generated consumption behavior patterns are stored on the server.

[0239] Step 10:

[0240] The server provides the generated consumer behavior patterns to partner companies and other related organizations. Partner companies can use this data to make demand forecasts and optimize their entire supply chains. Data is provided to companies in the form of automatic transmissions via APIs and regular reports.

[0241] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0242] A specific embodiment of the present invention will be described in detail below. The system comprises a sensor installed in a refrigerator, a terminal linked to each household refrigerator, a server, and an emotion engine that recognizes the user's emotions.

[0243] Food Registration

[0244] First, we will explain the operation when a user stores new ingredients in the refrigerator. When a user places an ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode. This barcode information is sent to the terminal (a sensor system inside the refrigerator), and information such as the ingredient's name, expiration date, and quantity is automatically registered. The terminal saves this information in the refrigerator's internal memory and sends the registration data to a server.

[0245] Monitoring expiration dates of ingredients

[0246] The server periodically receives and stores information about ingredients sent from the refrigerator. It also monitors the expiration dates of ingredients and generates data to notify the user when the expiration date falls within a certain period of time (e.g., three days) from the set date.

[0247] Expiration date notice

[0248] The notification data generated by the server is immediately sent to the device (smartphone app). The device then notifies the user of a list of ingredients that are approaching their expiration date. This allows the user to avoid overlooking ingredients that are approaching their expiration date and consume them appropriately. As a result, food waste can be reduced.

[0249] Providing promotional information

[0250] The server also periodically receives and stores sales information from affiliated retailers. For food items nearing their expiration date, the server searches for related sales information and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and provides specific information, allowing the user to make purchases at an economically advantageous time.

[0251] Emotion engine integration

[0252] The emotion engine collects user emotion data and transmits it to a server in real time. The emotion engine is installed in devices such as smartphones and smart speakers and recognizes emotions from the user's tone of voice and facial expressions.

[0253] Optimizing notifications based on emotions

[0254] The server analyzes the emotion data received from the emotion engine and optimizes the content and timing of notifications based on the user's current emotional state. For example, if the user is stressed, a reminder notification will be sent in a gentle tone, while if the user is relaxed, a notification prompting a specific action will be sent.

[0255] Customize promotional information based on emotions

[0256] The server detects the user's emotional data and personalizes relevant promotional information. For example, if the user is happy, it can provide healthy recipes and discount coupons related to ingredients that are close to their expiration date, encouraging the user to make a purchase.

[0257] Sentiment Data Analysis and Supply Chain Optimization

[0258] The server periodically collects and analyzes data on ingredients and emotions in the refrigerator, and reflects this data in consumer behavior patterns based on region and time of day. This data is provided to partner companies, who can use this information to forecast demand and optimize the entire supply chain.

[0259] Specific examples

[0260] 1. Example of food ingredient registration

[0261] When a user buys milk and puts it in the refrigerator, they scan the barcode.

[0262] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[0263] 2. Example of expiration date notification

[0264] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[0265] The device displays a notification to the user through a smartphone app.

[0266] 3. Example of Emotion-Based Notification

[0267] The emotion engine recognizes that the user is feeling stressed from their tone of voice.

[0268] Instead of the usual reminder notification, the server notifies you with a soft tone message saying "Your milk is about to expire."

[0269] 4. Examples of sales promotion information provided

[0270] The server obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[0271] The device sends the user a sale notification such as "20% off milk."

[0272] 5. Data provision example

[0273] The server analyzes regional milk consumption data and identifies periods when consumption increases in specific regions.

[0274] The server provides the data to partner companies, which then adjust their supply plans.

[0275] Through the above-described embodiments, the system contributes to reducing food waste, optimizing consumer behavior, and optimizing the supply chain. Furthermore, by integrating an emotion engine, the system can provide a more personalized user experience and support efficient food ingredient management.

[0276] The processing flow will be explained below.

[0277] Food Registration

[0278] Step 1:

[0279] When a user places purchased ingredients in the refrigerator, they scan the barcode of the ingredients with a dedicated barcode scanner.

[0280] Step 2:

[0281] The terminal (a sensor system inside the refrigerator) receives the barcode and stores the food ingredient information (name, expiration date, quantity, etc.) in a database.

[0282] Step 3:

[0283] The food ingredient information acquired by the terminal is transmitted to the server.

[0284] Monitoring expiration dates of ingredients

[0285] Step 1:

[0286] The server periodically (e.g., once a day) receives the latest ingredient information from each refrigerator.

[0287] Step 2:

[0288] The server monitors the expiration date of the ingredients received, and sets a "warning flag" if the expiration date falls within a certain period of time (e.g., three days) from the set date.

[0289] Expiration date notice

[0290] Step 1:

[0291] The server generates a list of ingredients whose expiration date is approaching and searches for ingredients that satisfy the notification conditions.

[0292] Step 2:

[0293] The server generates notification data and sends it to the device (smartphone app).

[0294] Step 3:

[0295] The device sends a notification to the user's smartphone and displays a list of ingredients that are approaching their expiration date.

[0296] Providing promotional information

[0297] Step 1:

[0298] The server periodically obtains sales information from affiliated stores and stores it in a database.

[0299] Step 2:

[0300] The server searches for sales promotion information related to food ingredients that are close to their expiration date.

[0301] Step 3:

[0302] The device notifies the user of promotional information. Specifically, the smartphone app notifies the user that "XX is on sale."

[0303] Emotion engine integration

[0304] Step 1:

[0305] When a user is in front of the refrigerator or in the kitchen, the emotion engine collects the user's emotional data through the microphone and camera on their smartphone or smart speaker.

[0306] Step 2:

[0307] The emotion engine recognizes emotions from the user's tone of voice and facial expressions, and sends the data to the server in real time.

[0308] Optimizing notifications based on emotions

[0309] Step 1:

[0310] The server analyzes the emotion data received from the emotion engine.

[0311] Step 2:

[0312] The server adjusts the content and timing of notifications based on the user's current emotional state.

[0313] Step 3:

[0314] The device will notify the user with a message tailored to their emotional state, such as "The milk's expiration date is approaching."

[0315] Customize promotional information based on emotions

[0316] Step 1:

[0317] The server personalizes relevant promotional information based on the user's emotional data.

[0318] Step 2:

[0319] The server generates promotional information that matches the emotional state of the user.

[0320] Step 3:

[0321] The device sends notifications appropriate to the user's emotional state, such as "There are related products on sale."

[0322] Data provision and supply chain optimization

[0323] Step 1:

[0324] The server periodically collects and analyzes food data and emotional data from the refrigerator.

[0325] Step 2:

[0326] The server generates consumption behavior data based on region and time period.

[0327] Step 3:

[0328] The server provides the generated consumer behavior data to partner companies, who use this data to forecast demand and optimize their supply chains.

[0329] Through the above processing steps, this system effectively manages ingredients, reduces food waste, and optimizes the supply chain. In addition, the integration of an emotion engine provides a more personalized user experience and supports efficient ingredient management.

[0330] Example 2

[0331] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0332] Currently, managing food expiration dates is a hassle for many households, leading to food waste. Furthermore, notifications sent without considering the user's emotional state can increase their stress. Furthermore, there is a lack of effective ways to utilize consumer behavior data, preventing demand forecasting and supply chain optimization.

[0333] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for acquiring food information using a detection device installed in the refrigerator, a means for transmitting the food information to the server, a means for the server to monitor the food's expiration date and notify the user when it determines that the expiration date is approaching, an emotion recognition device for collecting the user's emotional data and transmitting it to the server, and a means for the server to optimize the content and timing of notifications based on the user's emotional data. This enables users to efficiently manage food expiration dates and reduce food waste. Furthermore, personalized notifications based on the user's emotional state reduce stress and provide a more comfortable user experience. Furthermore, analyzing and providing consumer behavior data enables demand forecasting and supply chain optimization.

[0334] A "detection device" is a device such as a sensor that is installed inside a refrigerator to obtain information about food.

[0335] "Food information" refers to data about food, such as the name, expiration date, and quantity of the food.

[0336] "Server" means a central system that receives, stores, analyzes food information and other related data, and provides notifications and information.

[0337] An "emotion recognition device" is a device that collects emotional data from the user's tone of voice, facial expressions, etc.

[0338] "Emotion data" is information that reflects the user's emotional state and is used to optimize the content and timing of notifications.

[0339] "Notification means" refers to the means used by the server to provide information to the user, and includes, for example, notification via a smartphone app.

[0340] "Sales promotion information" is information that provides economic benefits to users, such as sale information and coupons provided by affiliated sales organizations.

[0341] "Consumption behavior data" is data that analyzes food consumption patterns based on region and time of day, and is used for demand forecasting and supply chain optimization.

[0342] The present invention is a system for managing food in a refrigerator and personalizing the user experience, and consists of the following components:

[0343] System Configuration

[0344] Detection device

[0345] The detector installed in the refrigerator includes a sensor for reading the barcodes of food items. This sensor acquires the barcode information of the food items and transmits it to a terminal for storage in the refrigerator's internal memory. For example, when the barcode of milk is read, the information is transmitted to the terminal and data such as the name of the food, expiration date, and quantity are recorded.

[0346] Terminal

[0347] The terminal is a device connected to the refrigerator, which processes the food information received from the detector, stores it in its internal memory, and transmits the information to the server.

[0348] server

[0349] The server receives food information from the device and stores it in a database. Based on this information, the server monitors food expiration dates and generates notifications when the expiration date approaches. The server also obtains sales promotion information from affiliated sales organizations and notifies users of information related to foods that are approaching their expiration date.

[0350] emotion recognition device

[0351] An emotion recognition device is a device that analyzes a user's tone of voice and facial expressions to collect emotional data. This device is embedded in smartphones, smart speakers, etc. and transmits the emotional data to a server in real time.

[0352] Server Features

[0353] Expiration date monitoring and notification

[0354] The server monitors the food information in the refrigerator and generates a notification a certain period of time (e.g., three days) before the expiration date. For example, when the expiration date of milk approaches, the server creates a notification stating, "Milk expires in two days," and sends it to the user via the terminal.

[0355] Providing promotional information

[0356] The server periodically receives and stores promotional information from affiliated distributors, searches for promotional information related to food products that are close to their expiration date, and sends notifications such as "20% off milk" to the terminal.

[0357] Optimizing notifications based on emotions

[0358] The server analyzes the emotion data received from the emotion recognition device and optimizes the content and timing of notifications based on the user's current emotional state. For example, if the user is feeling stressed, the server may send a reminder with a soft-toned message.

[0359] Analysis and provision of consumer behavior data

[0360] The server periodically analyzes the food and sentiment data stored in the refrigerator to generate patterns of consumer behavior by region and time of day, which is then provided to partner companies for the purposes of demand forecasting and supply chain optimization.

[0361] Specific examples

[0362] 1. Example of food ingredient registration

[0363] When a user buys milk and puts it in the refrigerator, they scan the barcode.

[0364] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[0365] 2. Example of expiration date notification

[0366] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[0367] The device displays a notification to the user through a smartphone app.

[0368] 3. Example of Emotion-Based Notification

[0369] The emotion recognition device recognizes that the user is feeling stressed from the tone of their voice.

[0370] Instead of the usual reminder notification, the server notifies you with a soft tone message saying "Your milk is about to expire."

[0371] 4. Examples of sales promotion information provided

[0372] The server acquires milk sale information from affiliated sales organizations and associates it with milk whose expiration date is approaching.

[0373] The device sends the user a sale notification such as "20% off milk."

[0374] This system allows users to efficiently manage food expiration dates and reduce food waste. It also provides notifications based on the user's emotional state, reducing stress and providing a better user experience. Furthermore, by analyzing and providing consumer behavior data, companies can forecast demand and optimize their supply chains.

[0375] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0376] Step 1:

[0377] When a user puts a new ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode. The input is the ingredient's barcode, and the output is the barcode information. A detection device reads this barcode information and sends it to a terminal. For example, when a user puts milk in the refrigerator, the scanner reads the milk's barcode and sends the information to the terminal.

[0378] Step 2:

[0379] Based on the barcode information received by the terminal from the detector, detailed information about the ingredients (such as name, expiration date, and quantity) is obtained from the food database. The input is the barcode information, and the output is detailed information about the ingredients. This detailed information is stored in the terminal's internal memory and sent to the server. For example, when the terminal receives barcode information about milk, it obtains the data "Milk, expiration date: 10 days later, quantity: 1 liter" and sends it to the server.

[0380] Step 3:

[0381] The server receives the ingredient information sent from the terminal and saves it in its own database. The input is the ingredient information sent from the terminal, and the output is the saved data. The server uses this data to monitor the expiration date in detail. For example, the server receives data such as "Milk, expiration date: 10 days later, quantity: 1 liter" and saves it in its own database.

[0382] Step 4:

[0383] The server monitors food expiration dates based on stored data. The input is the stored food ingredient information, and the output is notification data generated based on the expiration date. A notification is generated a certain period of time (e.g., three days) before the expiration date is set, and sent to the terminal. For example, the server monitors the expiration date of milk, and when seven days have passed, it generates a notification stating, "The expiration date is two days later."

[0384] Step 5:

[0385] The device receives the notification data sent from the server and displays the notification to the user through the smartphone app. The input is the notification data from the server, and the output is the notification displayed to the user. For example, the device displays a notification on the smartphone app saying, "The expiration date for the milk is in two days."

[0386] Step 6:

[0387] The server periodically receives and stores promotional information from affiliated sales organizations. The input is the promotional information from the sales organization, and the output is the stored promotional information. For example, the server receives and stores sale information about "20% off milk" from an affiliated supermarket.

[0388] Step 7:

[0389] The server searches for promotional information related to food ingredients that are close to their expiration date and sends that information to the terminal. The input is information about food ingredients that are close to their expiration date and stored promotional information, and the output is notification data for the related promotional information. For example, the server searches for sale information about "20% off milk" related to milk that is close to its expiration date and sends that information to the terminal.

[0390] Step 8:

[0391] The emotion recognition device collects the user's tone of voice and facial expression data and sends it to the server in real time. The input is the user's emotional data, and the output is the emotional data sent to the server. For example, the emotion recognition device recognizes that the user is feeling stressed from the tone of their voice and sends that data to the server.

[0392] Step 9:

[0393] The server analyzes the received emotion data and optimizes the content and timing of notifications. The input is emotion data, and the output is optimized notification data. For example, if the user is feeling stressed, the server generates a soft-toned message saying, "Your milk is about to expire."

[0394] Step 10:

[0395] The server periodically analyzes food ingredient data and emotion data stored in the refrigerator to generate consumption behavior patterns by region and time period. The input is food ingredient data and emotion data, and the output is consumption behavior pattern data. This data is provided to partner companies and used for demand forecasting and supply chain optimization. For example, the server analyzes milk consumption data by region and provides that data to companies to adjust their supply plans.

[0396] (Application example 2)

[0397] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0398] The main functions of conventional food management systems are to notify users of expiration dates and provide promotional information, but they are unable to optimize notifications that take into account the user's emotional state or make meal suggestions based on their emotions, which means that they are unable to reduce users' stress or personalize their experience.In particular, in food delivery services, the lack of suggestions based on the user's emotional state and ingredients that are close to their expiration date has led to issues that result in lower user satisfaction and food waste.

[0399] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring ingredient information using a sensor installed in the refrigerator, means for transmitting the ingredient information to the server, means for monitoring the expiration dates of ingredients and notifying the user when it is determined that the expiration date is approaching, means for acquiring emotion data using an emotion engine that recognizes the user's emotional state, and means for optimizing the content and timing of the notification based on the emotion data. This enables expiration date notifications and meal suggestions that take the user's emotional state into consideration, making it possible to reduce ingredient waste and provide a more personalized food delivery experience.

[0400] A "sensor installed inside a refrigerator" is a device that is attached inside a refrigerator and detects information about food ingredients (such as name, expiration date, quantity, etc.).

[0401] A "server" is a computer system that receives, stores, and processes ingredient information and emotion data.

[0402] "Ingredient information" refers to data such as the name, expiration date, and quantity of each ingredient stored in the refrigerator.

[0403] The "means for monitoring expiration dates" is a function that continuously checks the expiration dates of ingredients and generates an alert when the expiration date approaches.

[0404] "Means for notifying users" refers to a function that sends information about the expiration date of ingredients or promotional information about specific ingredients to the user's device (e.g., smartphone).

[0405] The "emotion engine" is a system that analyzes the user's voice and facial expression data and recognizes the user's emotional state in real time.

[0406] "Emotion data" is data obtained by the emotion engine that represents the user's emotional state (e.g., joy, stress, relaxation, etc.).

[0407] "Means for optimizing notification content and timing" refers to a function that adjusts the content and timing of expiration date notifications and meal suggestions based on emotional data, providing information in the most optimal form for the user.

[0408] "Sales promotion information" refers to various types of marketing information provided to consumers, such as sale information and discount information provided by affiliated retailers.

[0409] The following describes in detail specific embodiments of the present invention. The present invention comprises a sensor installed in a refrigerator, a server that manages ingredient information, an emotion engine, and a user terminal.

[0410] First, a sensor installed in the refrigerator acquires information such as the name, expiration date, and quantity of each ingredient. Specifically, when a user places new ingredients in the refrigerator, a barcode scanner is used to read the ingredient's barcode. This information is sent to the sensor system inside the refrigerator, and the ingredient information is automatically registered. The ingredient information is then sent to the server and stored in the server's database.

[0411] Next, the server periodically monitors the ingredient information. The server continuously checks the expiration date of the ingredients, and when the expiration date approaches, it generates notification data and sends it to the user's device (e.g., smartphone). This allows the user to consume ingredients appropriately without missing any ingredients that are approaching their expiration date.

[0412] The emotion engine analyzes the user's voice and facial expressions to obtain emotional data. For example, it recognizes the user's emotional state, such as whether they are stressed or relaxed, from their voice tone and facial expression. This emotional data is sent to the server in real time.

[0413] The server optimizes the content and timing of notifications based on the acquired emotional data. For example, if the user is stressed, the notification will be delivered in a softer tone, while if the user is relaxed, the notification will prompt specific actions.

[0414] In addition, the server obtains sales promotion information from affiliated retailers. For food items nearing their expiration date, the server searches for related sales promotion information and sends that information to the user's device. This allows the user to purchase food items at an economically advantageous time.

[0415] Consider the following scenario as a concrete example. When a user scans the barcode of milk when putting it in the refrigerator, a sensor inside the refrigerator acquires the milk's information and sends it to a server. The server monitors the expiration date and, when the expiration date approaches, notifies the user's smartphone that "The milk's expiration date is approaching." If the user is feeling stressed, the notification will be made in a softer tone. The server also obtains milk sale information from affiliated stores and notifies the user.

[0416] An example of how a generative AI model can be used is the following prompt:

[0417] Example prompt:

[0418] Generate meal suggestions suitable for a stressed user based on the expiration dates of ingredients in the refrigerator and the user's emotional data. For example, suggest a relaxing dessert or a stress-relieving menu.

[0419] As a result, this system improves the efficiency of food management in the refrigerator and provides optimal information and experience for users.

[0420] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0421] Step 1:

[0422] When a user puts an ingredient in the refrigerator, they scan the barcode. The input is the ingredient's barcode data. The barcode scanner is used to process the data, obtaining information such as the ingredient's name, expiration date, and quantity. The output is the obtained ingredient information.

[0423] Step 2:

[0424] The sensor system inside the refrigerator sends the acquired ingredient information to a server. The input is the ingredient information obtained from the sensor system. The sensor system performs data calculations to transfer this data to a designated server. The output is the ingredient information sent to the server.

[0425] Step 3:

[0426] The server stores the received ingredient information in a database. The input is the ingredient information sent from the sensor system. The server processes the data by adding new ingredient information to the database. The output is the stored ingredient information.

[0427] Step 4:

[0428] The server periodically monitors the ingredient information in the database. The input is the ingredient information in the database. The server performs data calculations to detect the expiration date of each ingredient and generate notification data when the expiration date approaches. The output is the generated notification data.

[0429] Step 5:

[0430] The server sends the generated notification data to the user's device (e.g., smartphone). The input is the notification data. The server performs data calculations to transfer the notification data from the database. The output is the notification sent to the smartphone.

[0431] Step 6:

[0432] The emotion engine analyzes the user's voice and facial expression data to generate emotion data. The input is the user's voice data and facial expression data. The emotion engine performs data analysis to recognize the user's emotional state. The output is emotion data.

[0433] Step 7:

[0434] Emotion data is sent to the server. The input is emotion data provided by the emotion engine. The server stores this in a database and performs data calculations to optimize the content and timing of notifications. The output is optimized notification data.

[0435] Step 8:

[0436] The server obtains promotional information from affiliated retailers and searches for promotional information related to food ingredients with an approaching expiration date. The input is the promotional information from the retailer and the food ingredient database information. The server performs data operations to search for and integrate related promotional information. The output is notification data including the promotional information.

[0437] Step 9:

[0438] The optimized notification data and promotional information are sent to the user's terminal. The input is the notification data and promotional information sent from the server. The user's terminal processes the data and displays it as a notification. The output is the user's receipt of the notification.

[0439] The above are the specific processing steps of the system program that realizes this application example. The specific operations performed in each step smoothly manage food ingredient information, notify expiration dates, and optimize emotional state recognition and notification.

[0440] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0441] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0442] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0443] [Second embodiment]

[0444] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0445] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0446] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0447] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0448] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0449] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0450] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0451] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0452] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0453] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0454] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0455] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0456] The following describes in detail an embodiment of the present invention. The system comprises a sensor installed in a refrigerator, a terminal linked to the refrigerator in each home, and a server.

[0457] Food Registration

[0458] First, we will explain the operation when a user stores new ingredients in the refrigerator. When a user places an ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode. This barcode information is sent to the terminal (a sensor system inside the refrigerator), and information such as the ingredient's name, expiration date, and quantity is automatically registered. The terminal saves this information in the refrigerator's internal memory and sends the registration data to a server.

[0459] Monitoring expiration dates of ingredients

[0460] The server periodically receives and stores information about ingredients sent from the refrigerator. It also monitors the expiration dates of ingredients and generates data to notify the user when the expiration date falls within a certain period of time (e.g., three days) from the set date.

[0461] Expiration date notice

[0462] The notification data generated by the server is immediately sent to the device (smartphone app). The device then notifies the user of a list of ingredients that are approaching their expiration date. This allows the user to avoid overlooking ingredients that are approaching their expiration date and consume them appropriately. As a result, food waste can be reduced.

[0463] Providing promotional information

[0464] The server also periodically receives and stores sales information from affiliated retailers. For food items nearing their expiration date, the server searches for related sales information and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and provides specific information, allowing the user to make purchases at an economically advantageous time.

[0465] Data provision and supply chain optimization

[0466] The server periodically collects and analyzes data on ingredients stored in the refrigerator, generating consumption patterns based on region and time of day. This consumption data is provided to other partner companies, who can use it to forecast demand and optimize the entire supply chain.

[0467] Specific examples

[0468] 1. Example of food ingredient registration

[0469] When a user buys milk and puts it in the refrigerator, they scan the barcode.

[0470] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[0471] 2. Example of expiration date notification

[0472] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[0473] The device displays a notification to the user through a smartphone app.

[0474] 3. Examples of sales promotion information provided

[0475] The server obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[0476] The device sends the user a sale notification such as "20% off milk."

[0477] 4. Data provision example

[0478] The server analyzes regional milk consumption data and identifies periods when consumption increases in specific regions.

[0479] The server provides the data to partner companies, which then adjust their supply plans.

[0480] Through the above-described embodiment, the present system can contribute to reducing food waste, optimizing consumption behavior, and optimizing the supply chain.

[0481] The processing flow will be explained below.

[0482] Food Registration

[0483] Step 1:

[0484] When a user places purchased ingredients in the refrigerator, they scan the barcode of the ingredients with a dedicated barcode scanner.

[0485] Step 2:

[0486] The terminal (a sensor system inside the refrigerator) receives the barcode and stores the food ingredient information (name, expiration date, quantity, etc.) in a database.

[0487] Step 3:

[0488] The terminal transmits the ingredient information to the server.

[0489] Monitoring expiration dates of ingredients

[0490] Step 1:

[0491] The server periodically (e.g., once a day) receives the latest ingredient information from each refrigerator.

[0492] Step 2:

[0493] The server monitors the expiration date of the ingredients received, and sets a "warning flag" if the expiration date falls within a certain period of time (e.g., three days) from the set date.

[0494] Expiration date notice

[0495] Step 1:

[0496] The server generates a list of ingredients whose expiration date is approaching and searches for ingredients that satisfy the notification conditions.

[0497] Step 2:

[0498] The server generates notification data and sends it to the device (smartphone app).

[0499] Step 3:

[0500] The device sends a notification to the user's smartphone and displays a list of ingredients that are approaching their expiration date.

[0501] Providing promotional information

[0502] Step 1:

[0503] The server periodically obtains sales information from affiliated stores and stores it in a database.

[0504] Step 2:

[0505] The server searches for sales promotion information related to food ingredients that are close to their expiration date.

[0506] Step 3:

[0507] The device notifies the user of promotional information. Specifically, the smartphone app notifies the user that "XX is on sale."

[0508] Data provision and supply chain optimization

[0509] Step 1:

[0510] The server periodically collects and analyzes data on ingredients in the refrigerator.

[0511] Step 2:

[0512] The server generates data on consumer behavior based on region and time of day.

[0513] Step 3:

[0514] The server provides the generated consumer behavior data to partner companies, who then use this data to forecast demand and optimize their supply chains.

[0515] Through the above processing steps, this system effectively manages ingredients, reduces food waste, and optimizes the supply chain.

[0516] Example 1

[0517] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0518] In modern society, managing food expiration dates is an important issue. However, many households do not adequately manage expiration dates, resulting in a large amount of food being wasted. In addition, appropriate promotional information is not provided for food that is approaching its expiration date, resulting in economic losses. Furthermore, data on consumer behavior is not being fully utilized, resulting in supply chain optimization.

[0519] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0520] In this invention, the server includes means for acquiring ingredient information using a detector installed in the food preservation device, means for transmitting the ingredient information to a computing device via a communication device, means for the computing device to monitor the expiration dates of ingredients and notify the user when it determines that the expiration date is approaching, means for the computing device to acquire sales promotion information from affiliated retailers and notify the user of sales promotion information related to ingredients that are approaching their expiration date, and means for the computing device to analyze the ingredient data in the food preservation device, generate data on consumer behavior based on region and time period, and provide the data to other companies. This makes it possible to reduce food waste and optimize consumer behavior.

[0521] A "food preservation device" is a device used to preserve food, such as a refrigerator.

[0522] A "detector" is a device such as a sensor or barcode scanner that obtains information about ingredients.

[0523] "Ingredient information" refers to information such as the name of the ingredient, expiration date, and quantity.

[0524] "Communication device" refers to a network device for transmitting data from a terminal to a server.

[0525] "Computing device" refers to a server or computer used to receive, store, and analyze ingredient information.

[0526] "Notification" refers to a means by which the server notifies users of food ingredients that are nearing their expiration date or promotional information.

[0527] "Affiliated retailer" refers to a store that sells ingredients or products that have a cooperative relationship with the system.

[0528] "Sales promotion information" refers to sales information and campaign information for sales promotion.

[0529] "Memory device" refers to data storage for saving acquired ingredient information and analysis results.

[0530] "Means of visually displaying information" refers to means of showing information to users via smartphone apps, displays, etc.

[0531] "Consumption behavior data" refers to food consumption patterns based on region and time of day.

[0532] "Means of providing data to other companies" refers to means of providing the consumer behavior data analyzed by the server to partner companies via APIs or other means.

[0533] The following describes in detail a specific embodiment of the present invention. The system is comprised of a detector installed in the food preservation device, a communication terminal connected to each household's food preservation device, and a computing device for data processing. This reduces food waste and optimizes consumption behavior.

[0534] Acquiring ingredient information

[0535] When a user places new ingredients in the food storage device, they use a dedicated barcode scanner to read the barcode of the ingredient. This operation obtains information about the ingredient, such as its name, expiration date, and quantity. The device receives this information via a detector in the food storage device.

[0536] Registering and saving food information

[0537] When the terminal receives the barcode information, it automatically retrieves the corresponding ingredient information from the database. The retrieved information is stored in the internal memory of the food storage device and then transmitted to the computing device via the Internet. The computing device then stores the received ingredient information in the database.

[0538] Expiration date monitoring

[0539] The computing device periodically monitors expiration dates based on the food ingredient information sent from the food storage device. When an ingredient approaches its expiration date, the computing device generates a notification message. This message is sent from the computing device to a terminal (smartphone app).

[0540] Expiration date notification

[0541] The notification data generated by the computing device is immediately sent to the terminal, which then notifies the user of a list of ingredients that are approaching their expiration dates, allowing the user to consume ingredients appropriately without missing the expiration dates.

[0542] Providing promotional information

[0543] The computing device periodically receives and stores sales information from affiliated retailers. It searches for sales information related to food items with approaching expiration dates and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and provides specific information.

[0544] Data analysis and provision

[0545] The computing device periodically collects and analyzes food data stored in the food storage device to generate consumption patterns based on region and time of day. This data is provided to partner companies, allowing them to optimize their supply chains based on demand forecasts.

[0546] Specific examples

[0547] 1. Example of food ingredient registration

[0548] When a user purchases milk and places it in the food storage device, they scan the barcode.

[0549] The terminal obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the computing device.

[0550] 2. Example of expiration date notification

[0551] A computing device monitors the expiration date of the milk and generates a notification on the seventh day saying, "Expires in two days."

[0552] The device displays a notification to the user through a smartphone app.

[0553] 3. Examples of sales promotion information provided

[0554] The computing device obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[0555] The device sends the user a sale notification such as "20% off milk."

[0556] 4. Data provision example

[0557] The computing device analyzes regional milk consumption data and identifies when consumption will increase in certain areas.

[0558] The computing device provides the data to partner companies, which then adjust their supply plans.

[0559] Prompt Sentence Examples

[0560] Please tell me in detail the steps by which the sensors in the refrigerator obtain food information.

[0561] What is the algorithm used by the server to generate expiration notifications?

[0562] Please tell us specifically how you will implement a system to obtain sale information and notify users.

[0563] By inputting these prompts into a generative AI model, we can obtain answers such as detailed explanations of each function and analysis of the algorithms.

[0564] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0565] Step 1:

[0566] When a user places a new ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode, which then acquires barcode information (input). The barcode information contains the ingredient's identification code.

[0567] Specific action: The user presses the button on the barcode scanner and reads the barcode on the milk.

[0568] Step 2:

[0569] The terminal receives the barcode information and retrieves detailed information such as the ingredient name, expiration date, and quantity from the database based on the ingredient's identification code (data processing). The retrieved ingredient information (output) is stored in the internal memory and simultaneously sent to the server.

[0570] Specific operation: The terminal receives the barcode "1234567890123", retrieves the information "Milk, expiration date: 10 days later, quantity: 1 liter" from the database, stores it in its internal memory, and then sends it to the server.

[0571] Step 3:

[0572] The server receives the food ingredient information sent from the terminal and stores it in a database (input). It periodically scans the stored data and monitors the expiration date (output).

[0573] What happens: The server stores the information "Milk, expiration date: 10 days later, quantity: 1 liter" in a database and runs a database query at 2 AM every day to scan for the expiration date.

[0574] Step 4:

[0575] The server detects ingredients whose expiration date is approaching (data calculation). If the expiration date is within a certain period of time (e.g., three days before) from the set date, it generates data to notify the user (output).

[0576] Specific operation: The server executes a database query to extract ingredients with expiration dates within three days. For example, it generates notification data such as "Milk expires in two days."

[0577] Step 5:

[0578] The server immediately sends the generated notification data to the device (smartphone app) (input). The device receives the notification data and displays it to the user (output).

[0579] Specific operation: The server sends notification data to the device's push notification API, stating that the milk's expiration date is in two days. The device displays the data on the smartphone app.

[0580] Step 6:

[0581] The server obtains sales information through the API of partner retailers and periodically saves it (input). It searches for sales information related to food items that are close to their expiration date and sends that information to the terminal (output).

[0582] Specific operation: The server retrieves the "20% off milk" sale information from the API of the partner retailer, associates it with information about milk that is nearing its expiration date, and notifies the user.

[0583] Step 7:

[0584] The server periodically collects consumption data on food items stored in the refrigerator and generates consumption behavior patterns based on region and time of day (data analysis: input). The generated consumption behavior data is provided to partner companies (output).

[0585] Specific operation: The server performs monthly data analysis and sends the results of "analysis of peak milk consumption in specific regions" to partner companies via API.

[0586] This allows the system to manage ingredients, notify expiration dates, provide economical purchasing timing, and contribute to supply chain optimization.

[0587] (Application example 1)

[0588] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0589] In managing food in refrigerators, there is a need to reduce the time and errors involved in manually entering food information, efficiently monitor expiration dates, and reduce food waste. It is also expected that the economic burden will be reduced by providing appropriate promotional information for food items approaching their expiration dates in a timely manner. Furthermore, there is a demand for providing companies with data on consumer behavior based on region and time of day to optimize supply plans.

[0590] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0591] In this invention, the server includes a means for acquiring information about ingredients using sensors installed in the refrigerator, a means for transmitting the ingredient information to the server, a means for the server to monitor the expiration dates of ingredients and notify the user when the expiration date is approaching, and a means for the user to scan barcodes using the camera on their smart device when storing ingredients in the refrigerator and automatically register the ingredient information. This eliminates the need to manually enter ingredient information, streamlines expiration date management, and reduces food waste. Furthermore, by providing promotional information at the appropriate time, it is possible to reduce financial burdens, and by providing companies with data on consumer behavior based on region and time of day, it is possible to optimize supply plans.

[0592] A "sensor installed inside a refrigerator" is a device that monitors the environment inside the refrigerator and obtains information about ingredients (e.g., location, temperature, weight, etc.).

[0593] "Ingredient information" is detailed data about ingredients, such as the name, quantity, expiration date, and barcode information of the ingredients.

[0594] A "server" is a computer system that communicates with external devices via a network and stores, processes, and analyzes data.

[0595] A "use-by date" is the date by which food is considered safe to eat, and quality and safety may not be guaranteed after that date.

[0596] A "notification" is a message or alert that the system uses to inform the user.

[0597] A "smart device" is an electronic device that is connected to the Internet and has advanced functions that allow it to run a variety of applications.

[0598] A "camera" is a device used to take images and videos, and is also used to read barcodes.

[0599] A "barcode" is a method of expressing information by arranging numbers and letters in a specific pattern that can be read by a machine.

[0600] "Automatic registration" is a process in which a machine or system automatically enters and stores information without human intervention.

[0601] The following describes in detail specific embodiments of the present invention. The system comprises a sensor installed in a refrigerator, a terminal (smart device) that connects to the refrigerator in each home, and a server.

[0602] Food Registration

[0603] When a user places new ingredients in the refrigerator, they use the camera on their smart device to read the barcode of the ingredient. This barcode information is sent to the terminal, and information such as the ingredient's name, expiration date, and quantity is automatically registered. The terminal stores this information in the refrigerator's internal memory and simultaneously sends it to the server. This eliminates the need for manual input and prevents input errors.

[0604] Monitoring expiration dates of ingredients

[0605] The server periodically receives and stores information about ingredients sent from the refrigerator. It also monitors the expiration dates of ingredients and generates data to notify the user when the expiration date falls within a certain period of time (e.g., three days) from the set date.

[0606] Expiration date notice

[0607] The notification data generated by the server is immediately sent to the device. The device displays a list of ingredients that are approaching their expiration date and provides audio and visual notifications to the user. This allows the user to avoid overlooking expiration dates and consume ingredients appropriately without wasting them.

[0608] Providing promotional information

[0609] The server also periodically receives and stores sales information from affiliated retailers. For food ingredients that are nearing their expiration date, the server searches for related sales information and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and displays detailed information. This allows the user to make purchases at an economically advantageous time.

[0610] Data provision and supply chain optimization

[0611] The server periodically collects and analyzes data on ingredients stored in the refrigerator, generating consumption patterns based on region and time of day. This consumption data is provided to other partner companies, who can use it to forecast demand and optimize the entire supply chain.

[0612] Hardware and software used

[0613] The following hardware and software are used to implement this system:

[0614] Smart Devices: Use your smartphone or smart glasses to scan the barcode with the built-in camera.

[0615] Server: A computer system for receiving, storing, analyzing, and notifying data.

[0616] Python: A scripting language used for data processing and analysis on the server.

[0617] OpenCV: A library for camera operation and barcode scanning.

[0618] Requests: A library for HTTP communication with the server.

[0619] Specific examples

[0620] Example of ingredient registration:

[0621] When a user purchases milk and puts it in the refrigerator, they scan the barcode using the camera on their smart device.

[0622] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[0623] Example of expiration date notification:

[0624] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[0625] The terminal displays a notification to the user on the smart device.

[0626] Examples of promotional information provided:

[0627] The server obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[0628] The device sends the user a sale notification such as "20% off milk."

[0629] Data provided example:

[0630] The server analyzes regional milk consumption data and identifies periods when consumption increases in specific regions.

[0631] The server provides the data to partner companies, which then adjust their supply plans.

[0632] Prompt Sentence Examples

[0633] "Please use the camera to read the barcode inside the refrigerator and register the new food information."

[0634] "Show me notifications about ingredients approaching their expiration date"

[0635] "Get relevant sales information and notifications on your smart device."

[0636] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0637] Step 1:

[0638] When a user places new ingredients in the refrigerator, they scan the barcode using the smart device's camera. At this time, the smart device receives the barcode image as input and extracts the barcode information. Specifically, the camera captures the image and uses the OpenCV library to detect and read the barcode. Once the barcode information is obtained, it is processed into ingredient information including the ingredient's name, expiration date, quantity, etc. and saved on the device.

[0639] Step 2:

[0640] The terminal sends the ingredient information acquired in step 1 to the server. The data sent includes the ingredient's barcode information, name, expiration date, quantity, etc. This information is sent to the server as an HTTP request, and the terminal processes to receive the response (success / failure status code) from the server.

[0641] Step 3:

[0642] The server stores the received ingredient information in a database. This database manages detailed information about ingredients stored in each household's refrigerator. The input ingredient information is added to the database, and the storage result is sent to the terminal as a response.

[0643] Step 4:

[0644] The server periodically monitors the food ingredient information in the database and checks the expiration date of each ingredient. When a specified period (e.g., three days before the expiration date) has passed, notification data for that ingredient is generated. The input food ingredient information is used as the source data, and an algorithm determines whether the expiration date is approaching and outputs the notification data.

[0645] Step 5:

[0646] The notification data generated by the server is immediately sent to the device. The device then displays this notification data to the user. For example, a message such as "The expiration date is in two days" can be displayed on the smart device's display to clearly communicate the information to the user. The notification message can be displayed as a voice notification or a visual pop-up.

[0647] Step 6:

[0648] The server periodically receives sales information from affiliated retailers and stores it in a database. Here, the input is the retailer's sales information (e.g., 20% off milk), and the server processes the sales information to add and update it in the database.

[0649] Step 7:

[0650] The server associates sale information with food ingredient information and searches the database for sale information related to food ingredients that are close to their expiration date. For example, if milk is close to its expiration date, the server searches for related "20% off milk" information and generates it as notification data.

[0651] Step 8:

[0652] The server generates sales promotion information and sends it to the terminal, which then displays it to the user. The user can see the sales promotion information on the smart device display, such as "There are related products on sale," and is encouraged to make purchases at economically advantageous times.

[0653] Step 9:

[0654] The server periodically collects and analyzes food data from the refrigerator and generates consumption behavior patterns based on region and time of day. Here, food information data is used as input and consumption patterns are output using statistical analysis and machine learning models. The generated consumption behavior patterns are stored on the server.

[0655] Step 10:

[0656] The server provides the generated consumer behavior patterns to partner companies and other related organizations. Partner companies can use this data to make demand forecasts and optimize their entire supply chains. Data is provided to companies in the form of automatic transmissions via APIs and regular reports.

[0657] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0658] A specific embodiment of the present invention will be described in detail below. The system comprises a sensor installed in a refrigerator, a terminal linked to each household refrigerator, a server, and an emotion engine that recognizes the user's emotions.

[0659] Food Registration

[0660] First, we will explain the operation when a user stores new ingredients in the refrigerator. When a user places an ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode. This barcode information is sent to the terminal (a sensor system inside the refrigerator), and information such as the ingredient's name, expiration date, and quantity is automatically registered. The terminal saves this information in the refrigerator's internal memory and sends the registration data to a server.

[0661] Monitoring expiration dates of ingredients

[0662] The server periodically receives and stores information about ingredients sent from the refrigerator. It also monitors the expiration dates of ingredients and generates data to notify the user when the expiration date falls within a certain period of time (e.g., three days) from the set date.

[0663] Expiration date notice

[0664] The notification data generated by the server is immediately sent to the device (smartphone app). The device then notifies the user of a list of ingredients that are approaching their expiration date. This allows the user to avoid overlooking ingredients that are approaching their expiration date and consume them appropriately. As a result, food waste can be reduced.

[0665] Providing promotional information

[0666] The server also periodically receives and stores sales information from affiliated retailers. For food items nearing their expiration date, the server searches for related sales information and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and provides specific information, allowing the user to make purchases at an economically advantageous time.

[0667] Emotion engine integration

[0668] The emotion engine collects user emotion data and transmits it to a server in real time. The emotion engine is installed in devices such as smartphones and smart speakers and recognizes emotions from the user's tone of voice and facial expressions.

[0669] Optimizing notifications based on emotions

[0670] The server analyzes the emotion data received from the emotion engine and optimizes the content and timing of notifications based on the user's current emotional state. For example, if the user is stressed, a reminder notification will be sent in a gentle tone, while if the user is relaxed, a notification prompting a specific action will be sent.

[0671] Customize promotional information based on emotions

[0672] The server detects the user's emotional data and personalizes relevant promotional information. For example, if the user is happy, it can provide healthy recipes and discount coupons related to ingredients that are close to their expiration date, encouraging the user to make a purchase.

[0673] Sentiment Data Analysis and Supply Chain Optimization

[0674] The server periodically collects and analyzes data on ingredients and emotions in the refrigerator, and reflects this data in consumer behavior patterns based on region and time of day. This data is provided to partner companies, who can use this information to forecast demand and optimize the entire supply chain.

[0675] Specific examples

[0676] 1. Example of food ingredient registration

[0677] When a user buys milk and puts it in the refrigerator, they scan the barcode.

[0678] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[0679] 2. Example of expiration date notification

[0680] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[0681] The device displays a notification to the user through a smartphone app.

[0682] 3. Example of Emotion-Based Notification

[0683] The emotion engine recognizes that the user is feeling stressed from their tone of voice.

[0684] Instead of the usual reminder notification, the server notifies you with a soft tone message saying "Your milk is about to expire."

[0685] 4. Examples of sales promotion information provided

[0686] The server obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[0687] The device sends the user a sale notification such as "20% off milk."

[0688] 5. Data provision example

[0689] The server analyzes regional milk consumption data and identifies periods when consumption increases in specific regions.

[0690] The server provides the data to partner companies, which then adjust their supply plans.

[0691] Through the above-described embodiments, the system contributes to reducing food waste, optimizing consumer behavior, and optimizing the supply chain. Furthermore, by integrating an emotion engine, the system can provide a more personalized user experience and support efficient food ingredient management.

[0692] The processing flow will be explained below.

[0693] Food Registration

[0694] Step 1:

[0695] When a user places purchased ingredients in the refrigerator, they scan the barcode of the ingredients with a dedicated barcode scanner.

[0696] Step 2:

[0697] The terminal (a sensor system inside the refrigerator) receives the barcode and stores the food ingredient information (name, expiration date, quantity, etc.) in a database.

[0698] Step 3:

[0699] The food ingredient information acquired by the terminal is transmitted to the server.

[0700] Monitoring expiration dates of ingredients

[0701] Step 1:

[0702] The server periodically (e.g., once a day) receives the latest ingredient information from each refrigerator.

[0703] Step 2:

[0704] The server monitors the expiration date of the ingredients received, and sets a "warning flag" if the expiration date falls within a certain period of time (e.g., three days) from the set date.

[0705] Expiration date notice

[0706] Step 1:

[0707] The server generates a list of ingredients whose expiration date is approaching and searches for ingredients that satisfy the notification conditions.

[0708] Step 2:

[0709] The server generates notification data and sends it to the device (smartphone app).

[0710] Step 3:

[0711] The device sends a notification to the user's smartphone and displays a list of ingredients that are approaching their expiration date.

[0712] Providing promotional information

[0713] Step 1:

[0714] The server periodically obtains sales information from affiliated stores and stores it in a database.

[0715] Step 2:

[0716] The server searches for sales promotion information related to food ingredients that are close to their expiration date.

[0717] Step 3:

[0718] The device notifies the user of promotional information. Specifically, the smartphone app notifies the user that "XX is on sale."

[0719] Emotion engine integration

[0720] Step 1:

[0721] When a user is in front of the refrigerator or in the kitchen, the emotion engine collects the user's emotional data through the microphone and camera on their smartphone or smart speaker.

[0722] Step 2:

[0723] The emotion engine recognizes emotions from the user's tone of voice and facial expressions, and sends the data to the server in real time.

[0724] Optimizing notifications based on emotions

[0725] Step 1:

[0726] The server analyzes the emotion data received from the emotion engine.

[0727] Step 2:

[0728] The server adjusts the content and timing of notifications based on the user's current emotional state.

[0729] Step 3:

[0730] The device will notify the user with a message tailored to their emotional state, such as "The milk's expiration date is approaching."

[0731] Customize promotional information based on emotions

[0732] Step 1:

[0733] The server personalizes relevant promotional information based on the user's emotional data.

[0734] Step 2:

[0735] The server generates promotional information that matches the emotional state of the user.

[0736] Step 3:

[0737] The device sends notifications appropriate to the user's emotional state, such as "There are related products on sale."

[0738] Data provision and supply chain optimization

[0739] Step 1:

[0740] The server periodically collects and analyzes food data and emotional data from the refrigerator.

[0741] Step 2:

[0742] The server generates consumption behavior data based on region and time period.

[0743] Step 3:

[0744] The server provides the generated consumer behavior data to partner companies, who use this data to forecast demand and optimize their supply chains.

[0745] Through the above processing steps, this system effectively manages ingredients, reduces food waste, and optimizes the supply chain. In addition, the integration of an emotion engine provides a more personalized user experience and supports efficient ingredient management.

[0746] Example 2

[0747] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0748] Currently, managing food expiration dates is a hassle for many households, leading to food waste. Furthermore, notifications sent without considering the user's emotional state can increase their stress. Furthermore, there is a lack of effective ways to utilize consumer behavior data, preventing demand forecasting and supply chain optimization.

[0749] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for acquiring food information using a detection device installed in the refrigerator, a means for transmitting the food information to the server, a means for the server to monitor the food's expiration date and notify the user when it determines that the expiration date is approaching, an emotion recognition device for collecting the user's emotional data and transmitting it to the server, and a means for the server to optimize the content and timing of notifications based on the user's emotional data. This enables users to efficiently manage food expiration dates and reduce food waste. Furthermore, personalized notifications based on the user's emotional state reduce stress and provide a more comfortable user experience. Furthermore, analyzing and providing consumer behavior data enables demand forecasting and supply chain optimization.

[0750] A "detection device" is a device such as a sensor that is installed inside a refrigerator to obtain information about food.

[0751] "Food information" refers to data about food, such as the name, expiration date, and quantity of the food.

[0752] "Server" means a central system that receives, stores, analyzes food information and other related data, and provides notifications and information.

[0753] An "emotion recognition device" is a device that collects emotional data from the user's tone of voice, facial expressions, etc.

[0754] "Emotion data" is information that reflects the user's emotional state and is used to optimize the content and timing of notifications.

[0755] "Notification means" refers to the means used by the server to provide information to the user, and includes, for example, notification via a smartphone app.

[0756] "Sales promotion information" is information that provides economic benefits to users, such as sale information and coupons provided by affiliated sales organizations.

[0757] "Consumption behavior data" is data that analyzes food consumption patterns based on region and time of day, and is used for demand forecasting and supply chain optimization.

[0758] The present invention is a system for managing food in a refrigerator and personalizing the user experience, and consists of the following components:

[0759] System Configuration

[0760] Detection device

[0761] The detector installed in the refrigerator includes a sensor for reading the barcodes of food items. This sensor acquires the barcode information of the food items and transmits it to a terminal for storage in the refrigerator's internal memory. For example, when the barcode of milk is read, the information is transmitted to the terminal and data such as the name of the food, expiration date, and quantity are recorded.

[0762] Terminal

[0763] The terminal is a device connected to the refrigerator, which processes the food information received from the detector, stores it in its internal memory, and transmits the information to the server.

[0764] server

[0765] The server receives food information from the device and stores it in a database. Based on this information, the server monitors food expiration dates and generates notifications when the expiration date approaches. The server also obtains sales promotion information from affiliated sales organizations and notifies users of information related to foods that are approaching their expiration date.

[0766] emotion recognition device

[0767] An emotion recognition device is a device that analyzes a user's tone of voice and facial expressions to collect emotional data. This device is embedded in smartphones, smart speakers, etc. and transmits the emotional data to a server in real time.

[0768] Server Features

[0769] Expiration date monitoring and notification

[0770] The server monitors the food information in the refrigerator and generates a notification a certain period of time (e.g., three days) before the expiration date. For example, when the expiration date of milk approaches, the server creates a notification stating, "Milk expires in two days," and sends it to the user via the terminal.

[0771] Providing promotional information

[0772] The server periodically receives and stores promotional information from affiliated distributors, searches for promotional information related to food products that are close to their expiration date, and sends notifications such as "20% off milk" to the terminal.

[0773] Optimizing notifications based on emotions

[0774] The server analyzes the emotion data received from the emotion recognition device and optimizes the content and timing of notifications based on the user's current emotional state. For example, if the user is feeling stressed, the server may send a reminder with a soft-toned message.

[0775] Analysis and provision of consumer behavior data

[0776] The server periodically analyzes the food and sentiment data stored in the refrigerator to generate patterns of consumer behavior by region and time of day, which is then provided to partner companies for the purposes of demand forecasting and supply chain optimization.

[0777] Specific examples

[0778] 1. Example of food ingredient registration

[0779] When a user buys milk and puts it in the refrigerator, they scan the barcode.

[0780] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[0781] 2. Example of expiration date notification

[0782] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[0783] The device displays a notification to the user through a smartphone app.

[0784] 3. Example of Emotion-Based Notification

[0785] The emotion recognition device recognizes that the user is feeling stressed from the tone of their voice.

[0786] Instead of the usual reminder notification, the server notifies you with a soft tone message saying "Your milk is about to expire."

[0787] 4. Examples of sales promotion information provided

[0788] The server acquires milk sale information from affiliated sales organizations and associates it with milk whose expiration date is approaching.

[0789] The device sends the user a sale notification such as "20% off milk."

[0790] This system allows users to efficiently manage food expiration dates and reduce food waste. It also provides notifications based on the user's emotional state, reducing stress and providing a better user experience. Furthermore, by analyzing and providing consumer behavior data, companies can forecast demand and optimize their supply chains.

[0791] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0792] Step 1:

[0793] When a user puts a new ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode. The input is the ingredient's barcode, and the output is the barcode information. A detection device reads this barcode information and sends it to a terminal. For example, when a user puts milk in the refrigerator, the scanner reads the milk's barcode and sends the information to the terminal.

[0794] Step 2:

[0795] Based on the barcode information received by the terminal from the detector, detailed information about the ingredients (such as name, expiration date, and quantity) is obtained from the food database. The input is the barcode information, and the output is detailed information about the ingredients. This detailed information is stored in the terminal's internal memory and sent to the server. For example, when the terminal receives barcode information about milk, it obtains the data "Milk, expiration date: 10 days later, quantity: 1 liter" and sends it to the server.

[0796] Step 3:

[0797] The server receives the ingredient information sent from the terminal and saves it in its own database. The input is the ingredient information sent from the terminal, and the output is the saved data. The server uses this data to monitor the expiration date in detail. For example, the server receives data such as "Milk, expiration date: 10 days later, quantity: 1 liter" and saves it in its own database.

[0798] Step 4:

[0799] The server monitors food expiration dates based on stored data. The input is the stored food ingredient information, and the output is notification data generated based on the expiration date. A notification is generated a certain period of time (e.g., three days) before the expiration date is set, and sent to the terminal. For example, the server monitors the expiration date of milk, and when seven days have passed, it generates a notification stating, "The expiration date is two days later."

[0800] Step 5:

[0801] The device receives the notification data sent from the server and displays the notification to the user through the smartphone app. The input is the notification data from the server, and the output is the notification displayed to the user. For example, the device displays a notification on the smartphone app saying, "The expiration date for the milk is in two days."

[0802] Step 6:

[0803] The server periodically receives and stores promotional information from affiliated sales organizations. The input is the promotional information from the sales organization, and the output is the stored promotional information. For example, the server receives and stores sale information about "20% off milk" from an affiliated supermarket.

[0804] Step 7:

[0805] The server searches for promotional information related to food ingredients that are close to their expiration date and sends that information to the terminal. The input is information about food ingredients that are close to their expiration date and stored promotional information, and the output is notification data for the related promotional information. For example, the server searches for sale information about "20% off milk" related to milk that is close to its expiration date and sends that information to the terminal.

[0806] Step 8:

[0807] The emotion recognition device collects the user's tone of voice and facial expression data and sends it to the server in real time. The input is the user's emotional data, and the output is the emotional data sent to the server. For example, the emotion recognition device recognizes that the user is feeling stressed from the tone of their voice and sends that data to the server.

[0808] Step 9:

[0809] The server analyzes the received emotion data and optimizes the content and timing of notifications. The input is emotion data, and the output is optimized notification data. For example, if the user is feeling stressed, the server generates a soft-toned message saying, "Your milk is about to expire."

[0810] Step 10:

[0811] The server periodically analyzes food ingredient data and emotion data stored in the refrigerator to generate consumption behavior patterns by region and time period. The input is food ingredient data and emotion data, and the output is consumption behavior pattern data. This data is provided to partner companies and used for demand forecasting and supply chain optimization. For example, the server analyzes milk consumption data by region and provides that data to companies to adjust their supply plans.

[0812] (Application example 2)

[0813] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0814] The main functions of conventional food management systems are to notify users of expiration dates and provide promotional information, but they are unable to optimize notifications that take into account the user's emotional state or make meal suggestions based on their emotions, which means that they are unable to reduce users' stress or personalize their experience.In particular, in food delivery services, the lack of suggestions based on the user's emotional state and ingredients that are close to their expiration date has led to issues that result in lower user satisfaction and food waste.

[0815] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring ingredient information using a sensor installed in the refrigerator, means for transmitting the ingredient information to the server, means for monitoring the expiration dates of ingredients and notifying the user when it is determined that the expiration date is approaching, means for acquiring emotion data using an emotion engine that recognizes the user's emotional state, and means for optimizing the content and timing of the notification based on the emotion data. This enables expiration date notifications and meal suggestions that take the user's emotional state into consideration, making it possible to reduce ingredient waste and provide a more personalized food delivery experience.

[0816] A "sensor installed inside a refrigerator" is a device that is attached inside a refrigerator and detects information about food ingredients (such as name, expiration date, quantity, etc.).

[0817] A "server" is a computer system that receives, stores, and processes ingredient information and emotion data.

[0818] "Ingredient information" refers to data such as the name, expiration date, and quantity of each ingredient stored in the refrigerator.

[0819] The "means for monitoring expiration dates" is a function that continuously checks the expiration dates of ingredients and generates an alert when the expiration date approaches.

[0820] "Means for notifying users" refers to a function that sends information about the expiration date of ingredients or promotional information about specific ingredients to the user's device (e.g., smartphone).

[0821] The "emotion engine" is a system that analyzes the user's voice and facial expression data and recognizes the user's emotional state in real time.

[0822] "Emotion data" is data obtained by the emotion engine that represents the user's emotional state (e.g., joy, stress, relaxation, etc.).

[0823] "Means for optimizing notification content and timing" refers to a function that adjusts the content and timing of expiration date notifications and meal suggestions based on emotional data, providing information in the most optimal form for the user.

[0824] "Sales promotion information" refers to various types of marketing information provided to consumers, such as sale information and discount information provided by affiliated retailers.

[0825] The following describes in detail specific embodiments of the present invention. The present invention comprises a sensor installed in a refrigerator, a server that manages ingredient information, an emotion engine, and a user terminal.

[0826] First, a sensor installed in the refrigerator acquires information such as the name, expiration date, and quantity of each ingredient. Specifically, when a user places new ingredients in the refrigerator, a barcode scanner is used to read the ingredient's barcode. This information is sent to the sensor system inside the refrigerator, and the ingredient information is automatically registered. The ingredient information is then sent to the server and stored in the server's database.

[0827] Next, the server periodically monitors the ingredient information. The server continuously checks the expiration date of the ingredients, and when the expiration date approaches, it generates notification data and sends it to the user's device (e.g., smartphone). This allows the user to consume ingredients appropriately without missing any ingredients that are approaching their expiration date.

[0828] The emotion engine analyzes the user's voice and facial expressions to obtain emotional data. For example, it recognizes the user's emotional state, such as whether they are stressed or relaxed, from their voice tone and facial expression. This emotional data is sent to the server in real time.

[0829] The server optimizes the content and timing of notifications based on the acquired emotional data. For example, if the user is stressed, the notification will be delivered in a softer tone, while if the user is relaxed, the notification will prompt specific actions.

[0830] In addition, the server obtains sales promotion information from affiliated retailers. For food items nearing their expiration date, the server searches for related sales promotion information and sends that information to the user's device. This allows the user to purchase food items at an economically advantageous time.

[0831] Consider the following scenario as a concrete example. When a user scans the barcode of milk when putting it in the refrigerator, a sensor inside the refrigerator acquires the milk's information and sends it to a server. The server monitors the expiration date and, when the expiration date approaches, notifies the user's smartphone that "The milk's expiration date is approaching." If the user is feeling stressed, the notification will be made in a softer tone. The server also obtains milk sale information from affiliated stores and notifies the user.

[0832] An example of how a generative AI model can be used is the following prompt:

[0833] Example prompt:

[0834] Generate meal suggestions suitable for a stressed user based on the expiration dates of ingredients in the refrigerator and the user's emotional data. For example, suggest a relaxing dessert or a stress-relieving menu.

[0835] As a result, this system improves the efficiency of food management in the refrigerator and provides optimal information and experience for users.

[0836] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0837] Step 1:

[0838] When a user puts an ingredient in the refrigerator, they scan the barcode. The input is the ingredient's barcode data. The barcode scanner is used to process the data, obtaining information such as the ingredient's name, expiration date, and quantity. The output is the obtained ingredient information.

[0839] Step 2:

[0840] The sensor system inside the refrigerator sends the acquired ingredient information to a server. The input is the ingredient information obtained from the sensor system. The sensor system performs data calculations to transfer this data to a designated server. The output is the ingredient information sent to the server.

[0841] Step 3:

[0842] The server stores the received ingredient information in a database. The input is the ingredient information sent from the sensor system. The server processes the data by adding new ingredient information to the database. The output is the stored ingredient information.

[0843] Step 4:

[0844] The server periodically monitors the ingredient information in the database. The input is the ingredient information in the database. The server performs data calculations to detect the expiration date of each ingredient and generate notification data when the expiration date approaches. The output is the generated notification data.

[0845] Step 5:

[0846] The server sends the generated notification data to the user's device (e.g., smartphone). The input is the notification data. The server performs data calculations to transfer the notification data from the database. The output is the notification sent to the smartphone.

[0847] Step 6:

[0848] The emotion engine analyzes the user's voice and facial expression data to generate emotion data. The input is the user's voice data and facial expression data. The emotion engine performs data analysis to recognize the user's emotional state. The output is emotion data.

[0849] Step 7:

[0850] Emotion data is sent to the server. The input is emotion data provided by the emotion engine. The server stores this in a database and performs data calculations to optimize the content and timing of notifications. The output is optimized notification data.

[0851] Step 8:

[0852] The server obtains promotional information from affiliated retailers and searches for promotional information related to food ingredients with an approaching expiration date. The input is the promotional information from the retailer and the food ingredient database information. The server performs data operations to search for and integrate related promotional information. The output is notification data including the promotional information.

[0853] Step 9:

[0854] The optimized notification data and promotional information are sent to the user's terminal. The input is the notification data and promotional information sent from the server. The user's terminal processes the data and displays it as a notification. The output is the user's receipt of the notification.

[0855] The above are the specific processing steps of the system program that realizes this application example. The specific operations performed in each step smoothly manage food ingredient information, notify expiration dates, and optimize emotional state recognition and notification.

[0856] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0857] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0858] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0859] [Third embodiment]

[0860] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0861] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0862] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0863] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0864] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0865] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0866] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0867] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0868] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0869] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0870] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0871] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0872] The following describes in detail an embodiment of the present invention. The system comprises a sensor installed in a refrigerator, a terminal linked to the refrigerator in each home, and a server.

[0873] Food Registration

[0874] First, we will explain the operation when a user stores new ingredients in the refrigerator. When a user places an ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode. This barcode information is sent to the terminal (a sensor system inside the refrigerator), and information such as the ingredient's name, expiration date, and quantity is automatically registered. The terminal saves this information in the refrigerator's internal memory and sends the registration data to a server.

[0875] Monitoring expiration dates of ingredients

[0876] The server periodically receives and stores information about ingredients sent from the refrigerator. It also monitors the expiration dates of ingredients and generates data to notify the user when the expiration date falls within a certain period of time (e.g., three days) from the set date.

[0877] Expiration date notice

[0878] The notification data generated by the server is immediately sent to the device (smartphone app). The device then notifies the user of a list of ingredients that are approaching their expiration date. This allows the user to avoid overlooking ingredients that are approaching their expiration date and consume them appropriately. As a result, food waste can be reduced.

[0879] Providing promotional information

[0880] The server also periodically receives and stores sales information from affiliated retailers. For food items nearing their expiration date, the server searches for related sales information and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and provides specific information, allowing the user to make purchases at an economically advantageous time.

[0881] Data provision and supply chain optimization

[0882] The server periodically collects and analyzes data on ingredients stored in the refrigerator, generating consumption patterns based on region and time of day. This consumption data is provided to other partner companies, who can use it to forecast demand and optimize the entire supply chain.

[0883] Specific examples

[0884] 1. Example of food ingredient registration

[0885] When a user buys milk and puts it in the refrigerator, they scan the barcode.

[0886] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[0887] 2. Example of expiration date notification

[0888] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[0889] The device displays a notification to the user through a smartphone app.

[0890] 3. Examples of sales promotion information provided

[0891] The server obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[0892] The device sends the user a sale notification such as "20% off milk."

[0893] 4. Data provision example

[0894] The server analyzes regional milk consumption data and identifies periods when consumption increases in specific regions.

[0895] The server provides the data to partner companies, which then adjust their supply plans.

[0896] Through the above-described embodiment, the present system can contribute to reducing food waste, optimizing consumption behavior, and optimizing the supply chain.

[0897] The processing flow will be explained below.

[0898] Food Registration

[0899] Step 1:

[0900] When a user places purchased ingredients in the refrigerator, they scan the barcode of the ingredients with a dedicated barcode scanner.

[0901] Step 2:

[0902] The terminal (a sensor system inside the refrigerator) receives the barcode and stores the food ingredient information (name, expiration date, quantity, etc.) in a database.

[0903] Step 3:

[0904] The terminal transmits the ingredient information to the server.

[0905] Monitoring expiration dates of ingredients

[0906] Step 1:

[0907] The server periodically (e.g., once a day) receives the latest ingredient information from each refrigerator.

[0908] Step 2:

[0909] The server monitors the expiration date of the ingredients received, and sets a "warning flag" if the expiration date falls within a certain period of time (e.g., three days) from the set date.

[0910] Expiration date notice

[0911] Step 1:

[0912] The server generates a list of ingredients whose expiration date is approaching and searches for ingredients that satisfy the notification conditions.

[0913] Step 2:

[0914] The server generates notification data and sends it to the device (smartphone app).

[0915] Step 3:

[0916] The device sends a notification to the user's smartphone and displays a list of ingredients that are approaching their expiration date.

[0917] Providing promotional information

[0918] Step 1:

[0919] The server periodically obtains sales information from affiliated stores and stores it in a database.

[0920] Step 2:

[0921] The server searches for sales promotion information related to food ingredients that are close to their expiration date.

[0922] Step 3:

[0923] The device notifies the user of promotional information. Specifically, the smartphone app notifies the user that "XX is on sale."

[0924] Data provision and supply chain optimization

[0925] Step 1:

[0926] The server periodically collects and analyzes data on ingredients in the refrigerator.

[0927] Step 2:

[0928] The server generates data on consumer behavior based on region and time of day.

[0929] Step 3:

[0930] The server provides the generated consumer behavior data to partner companies, who then use this data to forecast demand and optimize their supply chains.

[0931] Through the above processing steps, this system effectively manages ingredients, reduces food waste, and optimizes the supply chain.

[0932] Example 1

[0933] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0934] In modern society, managing food expiration dates is an important issue. However, many households do not adequately manage expiration dates, resulting in a large amount of food being wasted. In addition, appropriate promotional information is not provided for food that is approaching its expiration date, resulting in economic losses. Furthermore, data on consumer behavior is not being fully utilized, resulting in supply chain optimization.

[0935] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0936] In this invention, the server includes means for acquiring ingredient information using a detector installed in the food preservation device, means for transmitting the ingredient information to a computing device via a communication device, means for the computing device to monitor the expiration dates of ingredients and notify the user when it determines that the expiration date is approaching, means for the computing device to acquire sales promotion information from affiliated retailers and notify the user of sales promotion information related to ingredients that are approaching their expiration date, and means for the computing device to analyze the ingredient data in the food preservation device, generate data on consumer behavior based on region and time period, and provide the data to other companies. This makes it possible to reduce food waste and optimize consumer behavior.

[0937] A "food preservation device" is a device used to preserve food, such as a refrigerator.

[0938] A "detector" is a device such as a sensor or barcode scanner that obtains information about ingredients.

[0939] "Ingredient information" refers to information such as the name of the ingredient, expiration date, and quantity.

[0940] "Communication device" refers to a network device for transmitting data from a terminal to a server.

[0941] "Computing device" refers to a server or computer used to receive, store, and analyze ingredient information.

[0942] "Notification" refers to a means by which the server notifies users of food ingredients that are nearing their expiration date or promotional information.

[0943] "Affiliated retailer" refers to a store that sells ingredients or products that have a cooperative relationship with the system.

[0944] "Sales promotion information" refers to sales information and campaign information for sales promotion.

[0945] "Memory device" refers to data storage for saving acquired ingredient information and analysis results.

[0946] "Means of visually displaying information" refers to means of showing information to users via smartphone apps, displays, etc.

[0947] "Consumption behavior data" refers to food consumption patterns based on region and time of day.

[0948] "Means of providing data to other companies" refers to means of providing the consumer behavior data analyzed by the server to partner companies via APIs or other means.

[0949] The following describes in detail a specific embodiment of the present invention. The system is comprised of a detector installed in the food preservation device, a communication terminal connected to each household's food preservation device, and a computing device for data processing. This reduces food waste and optimizes consumption behavior.

[0950] Acquiring ingredient information

[0951] When a user places new ingredients in the food storage device, they use a dedicated barcode scanner to read the barcode of the ingredient. This operation obtains information about the ingredient, such as its name, expiration date, and quantity. The device receives this information via a detector in the food storage device.

[0952] Registering and saving food information

[0953] When the terminal receives the barcode information, it automatically retrieves the corresponding ingredient information from the database. The retrieved information is stored in the internal memory of the food storage device and then transmitted to the computing device via the Internet. The computing device then stores the received ingredient information in the database.

[0954] Expiration date monitoring

[0955] The computing device periodically monitors expiration dates based on the food ingredient information sent from the food storage device. When an ingredient approaches its expiration date, the computing device generates a notification message. This message is sent from the computing device to a terminal (smartphone app).

[0956] Expiration date notification

[0957] The notification data generated by the computing device is immediately sent to the terminal, which then notifies the user of a list of ingredients that are approaching their expiration dates, allowing the user to consume ingredients appropriately without missing the expiration dates.

[0958] Providing promotional information

[0959] The computing device periodically receives and stores sales information from affiliated retailers. It searches for sales information related to food items with approaching expiration dates and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and provides specific information.

[0960] Data analysis and provision

[0961] The computing device periodically collects and analyzes food data stored in the food storage device to generate consumption patterns based on region and time of day. This data is provided to partner companies, allowing them to optimize their supply chains based on demand forecasts.

[0962] Specific examples

[0963] 1. Example of food ingredient registration

[0964] When a user purchases milk and places it in the food storage device, they scan the barcode.

[0965] The terminal obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the computing device.

[0966] 2. Example of expiration date notification

[0967] A computing device monitors the expiration date of the milk and generates a notification on the seventh day saying, "Expires in two days."

[0968] The device displays a notification to the user through a smartphone app.

[0969] 3. Examples of sales promotion information provided

[0970] The computing device obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[0971] The device sends the user a sale notification such as "20% off milk."

[0972] 4. Data provision example

[0973] The computing device analyzes regional milk consumption data and identifies when consumption will increase in certain areas.

[0974] The computing device provides the data to partner companies, which then adjust their supply plans.

[0975] Prompt Sentence Examples

[0976] Please tell me in detail the steps by which the sensors in the refrigerator obtain food information.

[0977] What is the algorithm used by the server to generate expiration notifications?

[0978] Please tell us specifically how you will implement a system to obtain sale information and notify users.

[0979] By inputting these prompts into a generative AI model, we can obtain answers such as detailed explanations of each function and analysis of the algorithms.

[0980] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0981] Step 1:

[0982] When a user places a new ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode, which then acquires barcode information (input). The barcode information contains the ingredient's identification code.

[0983] Specific action: The user presses the button on the barcode scanner and reads the barcode on the milk.

[0984] Step 2:

[0985] The terminal receives the barcode information and retrieves detailed information such as the ingredient name, expiration date, and quantity from the database based on the ingredient's identification code (data processing). The retrieved ingredient information (output) is stored in the internal memory and simultaneously sent to the server.

[0986] Specific operation: The terminal receives the barcode "1234567890123", retrieves the information "Milk, expiration date: 10 days later, quantity: 1 liter" from the database, stores it in its internal memory, and then sends it to the server.

[0987] Step 3:

[0988] The server receives the food ingredient information sent from the terminal and stores it in a database (input). It periodically scans the stored data and monitors the expiration date (output).

[0989] What happens: The server stores the information "Milk, expiration date: 10 days later, quantity: 1 liter" in a database and runs a database query at 2 AM every day to scan for the expiration date.

[0990] Step 4:

[0991] The server detects ingredients whose expiration date is approaching (data calculation). If the expiration date is within a certain period of time (e.g., three days before) from the set date, it generates data to notify the user (output).

[0992] Specific operation: The server executes a database query to extract ingredients with expiration dates within three days. For example, it generates notification data such as "Milk expires in two days."

[0993] Step 5:

[0994] The server immediately sends the generated notification data to the device (smartphone app) (input). The device receives the notification data and displays it to the user (output).

[0995] Specific operation: The server sends notification data to the device's push notification API, stating that the milk's expiration date is in two days. The device displays the data on the smartphone app.

[0996] Step 6:

[0997] The server obtains sales information through the API of partner retailers and periodically saves it (input). It searches for sales information related to food items that are close to their expiration date and sends that information to the terminal (output).

[0998] Specific operation: The server retrieves the "20% off milk" sale information from the API of the partner retailer, associates it with information about milk that is nearing its expiration date, and notifies the user.

[0999] Step 7:

[1000] The server periodically collects consumption data on food items stored in the refrigerator and generates consumption behavior patterns based on region and time of day (data analysis: input). The generated consumption behavior data is provided to partner companies (output).

[1001] Specific operation: The server performs monthly data analysis and sends the results of "analysis of peak milk consumption in specific regions" to partner companies via API.

[1002] This allows the system to manage ingredients, notify expiration dates, provide economical purchasing timing, and contribute to supply chain optimization.

[1003] (Application example 1)

[1004] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1005] In managing food in refrigerators, there is a need to reduce the time and errors involved in manually entering food information, efficiently monitor expiration dates, and reduce food waste. It is also expected that the economic burden will be reduced by providing appropriate promotional information for food items approaching their expiration dates in a timely manner. Furthermore, there is a demand for providing companies with data on consumer behavior based on region and time of day to optimize supply plans.

[1006] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1007] In this invention, the server includes a means for acquiring information about ingredients using sensors installed in the refrigerator, a means for transmitting the ingredient information to the server, a means for the server to monitor the expiration dates of ingredients and notify the user when the expiration date is approaching, and a means for the user to scan barcodes using the camera on their smart device when storing ingredients in the refrigerator and automatically register the ingredient information. This eliminates the need to manually enter ingredient information, streamlines expiration date management, and reduces food waste. Furthermore, by providing promotional information at the appropriate time, it is possible to reduce financial burdens, and by providing companies with data on consumer behavior based on region and time of day, it is possible to optimize supply plans.

[1008] A "sensor installed inside a refrigerator" is a device that monitors the environment inside the refrigerator and obtains information about ingredients (e.g., location, temperature, weight, etc.).

[1009] "Ingredient information" is detailed data about ingredients, such as the name, quantity, expiration date, and barcode information of the ingredients.

[1010] A "server" is a computer system that communicates with external devices via a network and stores, processes, and analyzes data.

[1011] A "use-by date" is the date by which food is considered safe to eat, and quality and safety may not be guaranteed after that date.

[1012] A "notification" is a message or alert that the system uses to inform the user.

[1013] A "smart device" is an electronic device that is connected to the Internet and has advanced functions that allow it to run a variety of applications.

[1014] A "camera" is a device used to take images and videos, and is also used to read barcodes.

[1015] A "barcode" is a method of expressing information by arranging numbers and letters in a specific pattern that can be read by a machine.

[1016] "Automatic registration" is a process in which a machine or system automatically enters and stores information without human intervention.

[1017] The following describes in detail specific embodiments of the present invention. The system comprises a sensor installed in a refrigerator, a terminal (smart device) that connects to the refrigerator in each home, and a server.

[1018] Food Registration

[1019] When a user places new ingredients in the refrigerator, they use the camera on their smart device to read the barcode of the ingredient. This barcode information is sent to the terminal, and information such as the ingredient's name, expiration date, and quantity is automatically registered. The terminal stores this information in the refrigerator's internal memory and simultaneously sends it to the server. This eliminates the need for manual input and prevents input errors.

[1020] Monitoring expiration dates of ingredients

[1021] The server periodically receives and stores information about ingredients sent from the refrigerator. It also monitors the expiration dates of ingredients and generates data to notify the user when the expiration date falls within a certain period of time (e.g., three days) from the set date.

[1022] Expiration date notice

[1023] The notification data generated by the server is immediately sent to the device. The device displays a list of ingredients that are approaching their expiration date and provides audio and visual notifications to the user. This allows the user to avoid overlooking expiration dates and consume ingredients appropriately without wasting them.

[1024] Providing promotional information

[1025] The server also periodically receives and stores sales information from affiliated retailers. For food ingredients that are nearing their expiration date, the server searches for related sales information and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and displays detailed information. This allows the user to make purchases at an economically advantageous time.

[1026] Data provision and supply chain optimization

[1027] The server periodically collects and analyzes data on ingredients stored in the refrigerator, generating consumption patterns based on region and time of day. This consumption data is provided to other partner companies, who can use it to forecast demand and optimize the entire supply chain.

[1028] Hardware and software used

[1029] The following hardware and software are used to implement this system:

[1030] Smart Devices: Use your smartphone or smart glasses to scan the barcode with the built-in camera.

[1031] Server: A computer system for receiving, storing, analyzing, and notifying data.

[1032] Python: A scripting language used for data processing and analysis on the server.

[1033] OpenCV: A library for camera operation and barcode scanning.

[1034] Requests: A library for HTTP communication with the server.

[1035] Specific examples

[1036] Example of ingredient registration:

[1037] When a user purchases milk and puts it in the refrigerator, they scan the barcode using the camera on their smart device.

[1038] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[1039] Example of expiration date notification:

[1040] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[1041] The terminal displays a notification to the user on the smart device.

[1042] Examples of promotional information provided:

[1043] The server obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[1044] The device sends the user a sale notification such as "20% off milk."

[1045] Data provided example:

[1046] The server analyzes regional milk consumption data and identifies periods when consumption increases in specific regions.

[1047] The server provides the data to partner companies, which then adjust their supply plans.

[1048] Prompt Sentence Examples

[1049] "Please use the camera to read the barcode inside the refrigerator and register the new food information."

[1050] "Show me notifications about ingredients approaching their expiration date"

[1051] "Get relevant sales information and notifications on your smart device."

[1052] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1053] Step 1:

[1054] When a user places new ingredients in the refrigerator, they scan the barcode using the smart device's camera. At this time, the smart device receives the barcode image as input and extracts the barcode information. Specifically, the camera captures the image and uses the OpenCV library to detect and read the barcode. Once the barcode information is obtained, it is processed into ingredient information including the ingredient's name, expiration date, quantity, etc. and saved on the device.

[1055] Step 2:

[1056] The terminal sends the ingredient information acquired in step 1 to the server. The data sent includes the ingredient's barcode information, name, expiration date, quantity, etc. This information is sent to the server as an HTTP request, and the terminal processes to receive the response (success / failure status code) from the server.

[1057] Step 3:

[1058] The server stores the received ingredient information in a database. This database manages detailed information about ingredients stored in each household's refrigerator. The input ingredient information is added to the database, and the storage result is sent to the terminal as a response.

[1059] Step 4:

[1060] The server periodically monitors the food ingredient information in the database and checks the expiration date of each ingredient. When a specified period (e.g., three days before the expiration date) has passed, notification data for that ingredient is generated. The input food ingredient information is used as the source data, and an algorithm determines whether the expiration date is approaching and outputs the notification data.

[1061] Step 5:

[1062] The notification data generated by the server is immediately sent to the device. The device then displays this notification data to the user. For example, a message such as "The expiration date is in two days" can be displayed on the smart device's display to clearly communicate the information to the user. The notification message can be displayed as a voice notification or a visual pop-up.

[1063] Step 6:

[1064] The server periodically receives sales information from affiliated retailers and stores it in a database. Here, the input is the retailer's sales information (e.g., 20% off milk), and the server processes the sales information to add and update it in the database.

[1065] Step 7:

[1066] The server associates sale information with food ingredient information and searches the database for sale information related to food ingredients that are close to their expiration date. For example, if milk is close to its expiration date, the server searches for related "20% off milk" information and generates it as notification data.

[1067] Step 8:

[1068] The server generates sales promotion information and sends it to the terminal, which then displays it to the user. The user can see the sales promotion information on the smart device display, such as "There are related products on sale," and is encouraged to make purchases at economically advantageous times.

[1069] Step 9:

[1070] The server periodically collects and analyzes food data from the refrigerator and generates consumption behavior patterns based on region and time of day. Here, food information data is used as input and consumption patterns are output using statistical analysis and machine learning models. The generated consumption behavior patterns are stored on the server.

[1071] Step 10:

[1072] The server provides the generated consumer behavior patterns to partner companies and other related organizations. Partner companies can use this data to make demand forecasts and optimize their entire supply chains. Data is provided to companies in the form of automatic transmissions via APIs and regular reports.

[1073] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1074] A specific embodiment of the present invention will be described in detail below. The system comprises a sensor installed in a refrigerator, a terminal linked to each household refrigerator, a server, and an emotion engine that recognizes the user's emotions.

[1075] Food Registration

[1076] First, we will explain the operation when a user stores new ingredients in the refrigerator. When a user places an ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode. This barcode information is sent to the terminal (a sensor system inside the refrigerator), and information such as the ingredient's name, expiration date, and quantity is automatically registered. The terminal saves this information in the refrigerator's internal memory and sends the registration data to a server.

[1077] Monitoring expiration dates of ingredients

[1078] The server periodically receives and stores information about ingredients sent from the refrigerator. It also monitors the expiration dates of ingredients and generates data to notify the user when the expiration date falls within a certain period of time (e.g., three days) from the set date.

[1079] Expiration date notice

[1080] The notification data generated by the server is immediately sent to the device (smartphone app). The device then notifies the user of a list of ingredients that are approaching their expiration date. This allows the user to avoid overlooking ingredients that are approaching their expiration date and consume them appropriately. As a result, food waste can be reduced.

[1081] Providing promotional information

[1082] The server also periodically receives and stores sales information from affiliated retailers. For food items nearing their expiration date, the server searches for related sales information and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and provides specific information, allowing the user to make purchases at an economically advantageous time.

[1083] Emotion engine integration

[1084] The emotion engine collects user emotion data and transmits it to a server in real time. The emotion engine is installed in devices such as smartphones and smart speakers and recognizes emotions from the user's tone of voice and facial expressions.

[1085] Optimizing notifications based on emotions

[1086] The server analyzes the emotion data received from the emotion engine and optimizes the content and timing of notifications based on the user's current emotional state. For example, if the user is stressed, a reminder notification will be sent in a gentle tone, while if the user is relaxed, a notification prompting a specific action will be sent.

[1087] Customize promotional information based on emotions

[1088] The server detects the user's emotional data and personalizes relevant promotional information. For example, if the user is happy, it can provide healthy recipes and discount coupons related to ingredients that are close to their expiration date, encouraging the user to make a purchase.

[1089] Sentiment Data Analysis and Supply Chain Optimization

[1090] The server periodically collects and analyzes data on ingredients and emotions in the refrigerator, and reflects this data in consumer behavior patterns based on region and time of day. This data is provided to partner companies, who can use this information to forecast demand and optimize the entire supply chain.

[1091] Specific examples

[1092] 1. Example of food ingredient registration

[1093] When a user buys milk and puts it in the refrigerator, they scan the barcode.

[1094] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[1095] 2. Example of expiration date notification

[1096] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[1097] The device displays a notification to the user through a smartphone app.

[1098] 3. Example of Emotion-Based Notification

[1099] The emotion engine recognizes that the user is feeling stressed from their tone of voice.

[1100] Instead of the usual reminder notification, the server notifies you with a soft tone message saying "Your milk is about to expire."

[1101] 4. Examples of sales promotion information provided

[1102] The server obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[1103] The device sends the user a sale notification such as "20% off milk."

[1104] 5. Data provision example

[1105] The server analyzes regional milk consumption data and identifies periods when consumption increases in specific regions.

[1106] The server provides the data to partner companies, which then adjust their supply plans.

[1107] Through the above-described embodiments, the system contributes to reducing food waste, optimizing consumer behavior, and optimizing the supply chain. Furthermore, by integrating an emotion engine, the system can provide a more personalized user experience and support efficient food ingredient management.

[1108] The processing flow will be explained below.

[1109] Food Registration

[1110] Step 1:

[1111] When a user places purchased ingredients in the refrigerator, they scan the barcode of the ingredients with a dedicated barcode scanner.

[1112] Step 2:

[1113] The terminal (a sensor system inside the refrigerator) receives the barcode and stores the food ingredient information (name, expiration date, quantity, etc.) in a database.

[1114] Step 3:

[1115] The food ingredient information acquired by the terminal is transmitted to the server.

[1116] Monitoring expiration dates of ingredients

[1117] Step 1:

[1118] The server periodically (e.g., once a day) receives the latest ingredient information from each refrigerator.

[1119] Step 2:

[1120] The server monitors the expiration date of the ingredients received, and sets a "warning flag" if the expiration date falls within a certain period of time (e.g., three days) from the set date.

[1121] Expiration date notice

[1122] Step 1:

[1123] The server generates a list of ingredients whose expiration date is approaching and searches for ingredients that satisfy the notification conditions.

[1124] Step 2:

[1125] The server generates notification data and sends it to the device (smartphone app).

[1126] Step 3:

[1127] The device sends a notification to the user's smartphone and displays a list of ingredients that are approaching their expiration date.

[1128] Providing promotional information

[1129] Step 1:

[1130] The server periodically obtains sales information from affiliated stores and stores it in a database.

[1131] Step 2:

[1132] The server searches for sales promotion information related to food ingredients that are close to their expiration date.

[1133] Step 3:

[1134] The device notifies the user of promotional information. Specifically, the smartphone app notifies the user that "XX is on sale."

[1135] Emotion engine integration

[1136] Step 1:

[1137] When a user is in front of the refrigerator or in the kitchen, the emotion engine collects the user's emotional data through the microphone and camera on their smartphone or smart speaker.

[1138] Step 2:

[1139] The emotion engine recognizes emotions from the user's tone of voice and facial expressions, and sends the data to the server in real time.

[1140] Optimizing notifications based on emotions

[1141] Step 1:

[1142] The server analyzes the emotion data received from the emotion engine.

[1143] Step 2:

[1144] The server adjusts the content and timing of notifications based on the user's current emotional state.

[1145] Step 3:

[1146] The device will notify the user with a message tailored to their emotional state, such as "The milk's expiration date is approaching."

[1147] Customize promotional information based on emotions

[1148] Step 1:

[1149] The server personalizes relevant promotional information based on the user's emotional data.

[1150] Step 2:

[1151] The server generates promotional information that matches the emotional state of the user.

[1152] Step 3:

[1153] The device sends notifications appropriate to the user's emotional state, such as "There are related products on sale."

[1154] Data provision and supply chain optimization

[1155] Step 1:

[1156] The server periodically collects and analyzes food data and emotional data from the refrigerator.

[1157] Step 2:

[1158] The server generates consumption behavior data based on region and time period.

[1159] Step 3:

[1160] The server provides the generated consumer behavior data to partner companies, who use this data to forecast demand and optimize their supply chains.

[1161] Through the above processing steps, this system effectively manages ingredients, reduces food waste, and optimizes the supply chain. In addition, the integration of an emotion engine provides a more personalized user experience and supports efficient ingredient management.

[1162] Example 2

[1163] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1164] Currently, managing food expiration dates is a hassle for many households, leading to food waste. Furthermore, notifications sent without considering the user's emotional state can increase their stress. Furthermore, there is a lack of effective ways to utilize consumer behavior data, preventing demand forecasting and supply chain optimization.

[1165] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for acquiring food information using a detection device installed in the refrigerator, a means for transmitting the food information to the server, a means for the server to monitor the food's expiration date and notify the user when it determines that the expiration date is approaching, an emotion recognition device for collecting the user's emotional data and transmitting it to the server, and a means for the server to optimize the content and timing of notifications based on the user's emotional data. This enables users to efficiently manage food expiration dates and reduce food waste. Furthermore, personalized notifications based on the user's emotional state reduce stress and provide a more comfortable user experience. Furthermore, analyzing and providing consumer behavior data enables demand forecasting and supply chain optimization.

[1166] A "detection device" is a device such as a sensor that is installed inside a refrigerator to obtain information about food.

[1167] "Food information" refers to data about food, such as the name, expiration date, and quantity of the food.

[1168] "Server" means a central system that receives, stores, analyzes food information and other related data, and provides notifications and information.

[1169] An "emotion recognition device" is a device that collects emotional data from the user's tone of voice, facial expressions, etc.

[1170] "Emotion data" is information that reflects the user's emotional state and is used to optimize the content and timing of notifications.

[1171] "Notification means" refers to the means used by the server to provide information to the user, and includes, for example, notification via a smartphone app.

[1172] "Sales promotion information" is information that provides economic benefits to users, such as sale information and coupons provided by affiliated sales organizations.

[1173] "Consumption behavior data" is data that analyzes food consumption patterns based on region and time of day, and is used for demand forecasting and supply chain optimization.

[1174] The present invention is a system for managing food in a refrigerator and personalizing the user experience, and consists of the following components:

[1175] System Configuration

[1176] Detection device

[1177] The detector installed in the refrigerator includes a sensor for reading the barcodes of food items. This sensor acquires the barcode information of the food items and transmits it to a terminal for storage in the refrigerator's internal memory. For example, when the barcode of milk is read, the information is transmitted to the terminal and data such as the name of the food, expiration date, and quantity are recorded.

[1178] Terminal

[1179] The terminal is a device connected to the refrigerator, which processes the food information received from the detector, stores it in its internal memory, and transmits the information to the server.

[1180] server

[1181] The server receives food information from the device and stores it in a database. Based on this information, the server monitors food expiration dates and generates notifications when the expiration date approaches. The server also obtains sales promotion information from affiliated sales organizations and notifies users of information related to foods that are approaching their expiration date.

[1182] emotion recognition device

[1183] An emotion recognition device is a device that analyzes a user's tone of voice and facial expressions to collect emotional data. This device is embedded in smartphones, smart speakers, etc. and transmits the emotional data to a server in real time.

[1184] Server Features

[1185] Expiration date monitoring and notification

[1186] The server monitors the food information in the refrigerator and generates a notification a certain period of time (e.g., three days) before the expiration date. For example, when the expiration date of milk approaches, the server creates a notification stating, "Milk expires in two days," and sends it to the user via the terminal.

[1187] Providing promotional information

[1188] The server periodically receives and stores promotional information from affiliated distributors, searches for promotional information related to food products that are close to their expiration date, and sends notifications such as "20% off milk" to the terminal.

[1189] Optimizing notifications based on emotions

[1190] The server analyzes the emotion data received from the emotion recognition device and optimizes the content and timing of notifications based on the user's current emotional state. For example, if the user is feeling stressed, the server may send a reminder with a soft-toned message.

[1191] Analysis and provision of consumer behavior data

[1192] The server periodically analyzes the food and sentiment data stored in the refrigerator to generate patterns of consumer behavior by region and time of day, which is then provided to partner companies for the purposes of demand forecasting and supply chain optimization.

[1193] Specific examples

[1194] 1. Example of food ingredient registration

[1195] When a user buys milk and puts it in the refrigerator, they scan the barcode.

[1196] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[1197] 2. Example of expiration date notification

[1198] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[1199] The device displays a notification to the user through a smartphone app.

[1200] 3. Example of Emotion-Based Notification

[1201] The emotion recognition device recognizes that the user is feeling stressed from the tone of their voice.

[1202] Instead of the usual reminder notification, the server notifies you with a soft tone message saying "Your milk is about to expire."

[1203] 4. Examples of sales promotion information provided

[1204] The server acquires milk sale information from affiliated sales organizations and associates it with milk whose expiration date is approaching.

[1205] The device sends the user a sale notification such as "20% off milk."

[1206] This system allows users to efficiently manage food expiration dates and reduce food waste. It also provides notifications based on the user's emotional state, reducing stress and providing a better user experience. Furthermore, by analyzing and providing consumer behavior data, companies can forecast demand and optimize their supply chains.

[1207] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1208] Step 1:

[1209] When a user puts a new ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode. The input is the ingredient's barcode, and the output is the barcode information. A detection device reads this barcode information and sends it to a terminal. For example, when a user puts milk in the refrigerator, the scanner reads the milk's barcode and sends the information to the terminal.

[1210] Step 2:

[1211] Based on the barcode information received by the terminal from the detector, detailed information about the ingredients (such as name, expiration date, and quantity) is obtained from the food database. The input is the barcode information, and the output is detailed information about the ingredients. This detailed information is stored in the terminal's internal memory and sent to the server. For example, when the terminal receives barcode information about milk, it obtains the data "Milk, expiration date: 10 days later, quantity: 1 liter" and sends it to the server.

[1212] Step 3:

[1213] The server receives the ingredient information sent from the terminal and saves it in its own database. The input is the ingredient information sent from the terminal, and the output is the saved data. The server uses this data to monitor the expiration date in detail. For example, the server receives data such as "Milk, expiration date: 10 days later, quantity: 1 liter" and saves it in its own database.

[1214] Step 4:

[1215] The server monitors food expiration dates based on stored data. The input is the stored food ingredient information, and the output is notification data generated based on the expiration date. A notification is generated a certain period of time (e.g., three days) before the expiration date is set, and sent to the terminal. For example, the server monitors the expiration date of milk, and when seven days have passed, it generates a notification stating, "The expiration date is two days later."

[1216] Step 5:

[1217] The device receives the notification data sent from the server and displays the notification to the user through the smartphone app. The input is the notification data from the server, and the output is the notification displayed to the user. For example, the device displays a notification on the smartphone app saying, "The expiration date for the milk is in two days."

[1218] Step 6:

[1219] The server periodically receives and stores promotional information from affiliated sales organizations. The input is the promotional information from the sales organization, and the output is the stored promotional information. For example, the server receives and stores sale information about "20% off milk" from an affiliated supermarket.

[1220] Step 7:

[1221] The server searches for promotional information related to food ingredients that are close to their expiration date and sends that information to the terminal. The input is information about food ingredients that are close to their expiration date and stored promotional information, and the output is notification data for the related promotional information. For example, the server searches for sale information about "20% off milk" related to milk that is close to its expiration date and sends that information to the terminal.

[1222] Step 8:

[1223] The emotion recognition device collects the user's tone of voice and facial expression data and sends it to the server in real time. The input is the user's emotional data, and the output is the emotional data sent to the server. For example, the emotion recognition device recognizes that the user is feeling stressed from the tone of their voice and sends that data to the server.

[1224] Step 9:

[1225] The server analyzes the received emotion data and optimizes the content and timing of notifications. The input is emotion data, and the output is optimized notification data. For example, if the user is feeling stressed, the server generates a soft-toned message saying, "Your milk is about to expire."

[1226] Step 10:

[1227] The server periodically analyzes food ingredient data and emotion data stored in the refrigerator to generate consumption behavior patterns by region and time period. The input is food ingredient data and emotion data, and the output is consumption behavior pattern data. This data is provided to partner companies and used for demand forecasting and supply chain optimization. For example, the server analyzes milk consumption data by region and provides that data to companies to adjust their supply plans.

[1228] (Application example 2)

[1229] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1230] The main functions of conventional food management systems are to notify users of expiration dates and provide promotional information, but they are unable to optimize notifications that take into account the user's emotional state or make meal suggestions based on their emotions, which means that they are unable to reduce users' stress or personalize their experience.In particular, in food delivery services, the lack of suggestions based on the user's emotional state and ingredients that are close to their expiration date has led to issues that result in lower user satisfaction and food waste.

[1231] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring ingredient information using a sensor installed in the refrigerator, means for transmitting the ingredient information to the server, means for monitoring the expiration dates of ingredients and notifying the user when it is determined that the expiration date is approaching, means for acquiring emotion data using an emotion engine that recognizes the user's emotional state, and means for optimizing the content and timing of the notification based on the emotion data. This enables expiration date notifications and meal suggestions that take the user's emotional state into consideration, making it possible to reduce ingredient waste and provide a more personalized food delivery experience.

[1232] A "sensor installed inside a refrigerator" is a device that is attached inside a refrigerator and detects information about food ingredients (such as name, expiration date, quantity, etc.).

[1233] A "server" is a computer system that receives, stores, and processes ingredient information and emotion data.

[1234] "Ingredient information" refers to data such as the name, expiration date, and quantity of each ingredient stored in the refrigerator.

[1235] The "means for monitoring expiration dates" is a function that continuously checks the expiration dates of ingredients and generates an alert when the expiration date approaches.

[1236] "Means for notifying users" refers to a function that sends information about the expiration date of ingredients or promotional information about specific ingredients to the user's device (e.g., smartphone).

[1237] The "emotion engine" is a system that analyzes the user's voice and facial expression data and recognizes the user's emotional state in real time.

[1238] "Emotion data" is data obtained by the emotion engine that represents the user's emotional state (e.g., joy, stress, relaxation, etc.).

[1239] "Means for optimizing notification content and timing" refers to a function that adjusts the content and timing of expiration date notifications and meal suggestions based on emotional data, providing information in the most optimal form for the user.

[1240] "Sales promotion information" refers to various types of marketing information provided to consumers, such as sale information and discount information provided by affiliated retailers.

[1241] The following describes in detail specific embodiments of the present invention. The present invention comprises a sensor installed in a refrigerator, a server that manages ingredient information, an emotion engine, and a user terminal.

[1242] First, a sensor installed in the refrigerator acquires information such as the name, expiration date, and quantity of each ingredient. Specifically, when a user places new ingredients in the refrigerator, a barcode scanner is used to read the ingredient's barcode. This information is sent to the sensor system inside the refrigerator, and the ingredient information is automatically registered. The ingredient information is then sent to the server and stored in the server's database.

[1243] Next, the server periodically monitors the ingredient information. The server continuously checks the expiration date of the ingredients, and when the expiration date approaches, it generates notification data and sends it to the user's device (e.g., smartphone). This allows the user to consume ingredients appropriately without missing any ingredients that are approaching their expiration date.

[1244] The emotion engine analyzes the user's voice and facial expressions to obtain emotional data. For example, it recognizes the user's emotional state, such as whether they are stressed or relaxed, from their voice tone and facial expression. This emotional data is sent to the server in real time.

[1245] The server optimizes the content and timing of notifications based on the acquired emotional data. For example, if the user is stressed, the notification will be delivered in a softer tone, while if the user is relaxed, the notification will prompt specific actions.

[1246] In addition, the server obtains sales promotion information from affiliated retailers. For food items nearing their expiration date, the server searches for related sales promotion information and sends that information to the user's device. This allows the user to purchase food items at an economically advantageous time.

[1247] Consider the following scenario as a concrete example. When a user scans the barcode of milk when putting it in the refrigerator, a sensor inside the refrigerator acquires the milk's information and sends it to a server. The server monitors the expiration date and, when the expiration date approaches, notifies the user's smartphone that "The milk's expiration date is approaching." If the user is feeling stressed, the notification will be made in a softer tone. The server also obtains milk sale information from affiliated stores and notifies the user.

[1248] An example of how a generative AI model can be used is the following prompt:

[1249] Example prompt:

[1250] Generate meal suggestions suitable for a stressed user based on the expiration dates of ingredients in the refrigerator and the user's emotional data. For example, suggest a relaxing dessert or a stress-relieving menu.

[1251] As a result, this system improves the efficiency of food management in the refrigerator and provides optimal information and experience for users.

[1252] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1253] Step 1:

[1254] When a user puts an ingredient in the refrigerator, they scan the barcode. The input is the ingredient's barcode data. The barcode scanner is used to process the data, obtaining information such as the ingredient's name, expiration date, and quantity. The output is the obtained ingredient information.

[1255] Step 2:

[1256] The sensor system inside the refrigerator sends the acquired ingredient information to a server. The input is the ingredient information obtained from the sensor system. The sensor system performs data calculations to transfer this data to a designated server. The output is the ingredient information sent to the server.

[1257] Step 3:

[1258] The server stores the received ingredient information in a database. The input is the ingredient information sent from the sensor system. The server processes the data by adding new ingredient information to the database. The output is the stored ingredient information.

[1259] Step 4:

[1260] The server periodically monitors the ingredient information in the database. The input is the ingredient information in the database. The server performs data calculations to detect the expiration date of each ingredient and generate notification data when the expiration date approaches. The output is the generated notification data.

[1261] Step 5:

[1262] The server sends the generated notification data to the user's device (e.g., smartphone). The input is the notification data. The server performs data calculations to transfer the notification data from the database. The output is the notification sent to the smartphone.

[1263] Step 6:

[1264] The emotion engine analyzes the user's voice and facial expression data to generate emotion data. The input is the user's voice data and facial expression data. The emotion engine performs data analysis to recognize the user's emotional state. The output is emotion data.

[1265] Step 7:

[1266] Emotion data is sent to the server. The input is emotion data provided by the emotion engine. The server stores this in a database and performs data calculations to optimize the content and timing of notifications. The output is optimized notification data.

[1267] Step 8:

[1268] The server obtains promotional information from affiliated retailers and searches for promotional information related to food ingredients with an approaching expiration date. The input is the promotional information from the retailer and the food ingredient database information. The server performs data operations to search for and integrate related promotional information. The output is notification data including the promotional information.

[1269] Step 9:

[1270] The optimized notification data and promotional information are sent to the user's terminal. The input is the notification data and promotional information sent from the server. The user's terminal processes the data and displays it as a notification. The output is the user's receipt of the notification.

[1271] The above are the specific processing steps of the system program that realizes this application example. The specific operations performed in each step smoothly manage food ingredient information, notify expiration dates, and optimize emotional state recognition and notification.

[1272] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1273] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1274] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1275] [Fourth embodiment]

[1276] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1277] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1278] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1279] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1280] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1281] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1282] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1283] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1284] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1285] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1286] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1287] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1288] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1289] The following describes in detail an embodiment of the present invention. The system comprises a sensor installed in a refrigerator, a terminal linked to the refrigerator in each home, and a server.

[1290] Food Registration

[1291] First, we will explain the operation when a user stores new ingredients in the refrigerator. When a user places an ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode. This barcode information is sent to the terminal (a sensor system inside the refrigerator), and information such as the ingredient's name, expiration date, and quantity is automatically registered. The terminal saves this information in the refrigerator's internal memory and sends the registration data to a server.

[1292] Monitoring expiration dates of ingredients

[1293] The server periodically receives and stores information about ingredients sent from the refrigerator. It also monitors the expiration dates of ingredients and generates data to notify the user when the expiration date falls within a certain period of time (e.g., three days) from the set date.

[1294] Expiration date notice

[1295] The notification data generated by the server is immediately sent to the device (smartphone app). The device then notifies the user of a list of ingredients that are approaching their expiration date. This allows the user to avoid overlooking ingredients that are approaching their expiration date and consume them appropriately. As a result, food waste can be reduced.

[1296] Providing promotional information

[1297] The server also periodically receives and stores sales information from affiliated retailers. For food items nearing their expiration date, the server searches for related sales information and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and provides specific information, allowing the user to make purchases at an economically advantageous time.

[1298] Data provision and supply chain optimization

[1299] The server periodically collects and analyzes data on ingredients stored in the refrigerator, generating consumption patterns based on region and time of day. This consumption data is provided to other partner companies, who can use it to forecast demand and optimize the entire supply chain.

[1300] Specific examples

[1301] 1. Example of food ingredient registration

[1302] When a user buys milk and puts it in the refrigerator, they scan the barcode.

[1303] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[1304] 2. Example of expiration date notification

[1305] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[1306] The device displays a notification to the user through a smartphone app.

[1307] 3. Examples of sales promotion information provided

[1308] The server obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[1309] The device sends the user a sale notification such as "20% off milk."

[1310] 4. Data provision example

[1311] The server analyzes regional milk consumption data and identifies periods when consumption increases in specific regions.

[1312] The server provides the data to partner companies, which then adjust their supply plans.

[1313] Through the above-described embodiment, the present system can contribute to reducing food waste, optimizing consumption behavior, and optimizing the supply chain.

[1314] The processing flow will be explained below.

[1315] Food Registration

[1316] Step 1:

[1317] When a user places purchased ingredients in the refrigerator, they scan the barcode of the ingredients with a dedicated barcode scanner.

[1318] Step 2:

[1319] The terminal (a sensor system inside the refrigerator) receives the barcode and stores the food ingredient information (name, expiration date, quantity, etc.) in a database.

[1320] Step 3:

[1321] The terminal transmits the ingredient information to the server.

[1322] Monitoring expiration dates of ingredients

[1323] Step 1:

[1324] The server periodically (e.g., once a day) receives the latest ingredient information from each refrigerator.

[1325] Step 2:

[1326] The server monitors the expiration date of the ingredients received, and sets a "warning flag" if the expiration date falls within a certain period of time (e.g., three days) from the set date.

[1327] Expiration date notice

[1328] Step 1:

[1329] The server generates a list of ingredients whose expiration date is approaching and searches for ingredients that satisfy the notification conditions.

[1330] Step 2:

[1331] The server generates notification data and sends it to the device (smartphone app).

[1332] Step 3:

[1333] The device sends a notification to the user's smartphone and displays a list of ingredients that are approaching their expiration date.

[1334] Providing promotional information

[1335] Step 1:

[1336] The server periodically obtains sales information from affiliated stores and stores it in a database.

[1337] Step 2:

[1338] The server searches for sales promotion information related to food ingredients that are close to their expiration date.

[1339] Step 3:

[1340] The device notifies the user of promotional information. Specifically, the smartphone app notifies the user that "XX is on sale."

[1341] Data provision and supply chain optimization

[1342] Step 1:

[1343] The server periodically collects and analyzes data on ingredients in the refrigerator.

[1344] Step 2:

[1345] The server generates data on consumer behavior based on region and time of day.

[1346] Step 3:

[1347] The server provides the generated consumer behavior data to partner companies, who then use this data to forecast demand and optimize their supply chains.

[1348] Through the above processing steps, this system effectively manages ingredients, reduces food waste, and optimizes the supply chain.

[1349] Example 1

[1350] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1351] In modern society, managing food expiration dates is an important issue. However, many households do not adequately manage expiration dates, resulting in a large amount of food being wasted. In addition, appropriate promotional information is not provided for food that is approaching its expiration date, resulting in economic losses. Furthermore, data on consumer behavior is not being fully utilized, resulting in supply chain optimization.

[1352] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1353] In this invention, the server includes means for acquiring ingredient information using a detector installed in the food preservation device, means for transmitting the ingredient information to a computing device via a communication device, means for the computing device to monitor the expiration dates of ingredients and notify the user when it determines that the expiration date is approaching, means for the computing device to acquire sales promotion information from affiliated retailers and notify the user of sales promotion information related to ingredients that are approaching their expiration date, and means for the computing device to analyze the ingredient data in the food preservation device, generate data on consumer behavior based on region and time period, and provide the data to other companies. This makes it possible to reduce food waste and optimize consumer behavior.

[1354] A "food preservation device" is a device used to preserve food, such as a refrigerator.

[1355] A "detector" is a device such as a sensor or barcode scanner that obtains information about ingredients.

[1356] "Ingredient information" refers to information such as the name of the ingredient, expiration date, and quantity.

[1357] "Communication device" refers to a network device for transmitting data from a terminal to a server.

[1358] "Computing device" refers to a server or computer used to receive, store, and analyze ingredient information.

[1359] "Notification" refers to a means by which the server notifies users of food ingredients that are nearing their expiration date or promotional information.

[1360] "Affiliated retailer" refers to a store that sells ingredients or products that have a cooperative relationship with the system.

[1361] "Sales promotion information" refers to sales information and campaign information for sales promotion.

[1362] "Memory device" refers to data storage for saving acquired ingredient information and analysis results.

[1363] "Means of visually displaying information" refers to means of showing information to users via smartphone apps, displays, etc.

[1364] "Consumption behavior data" refers to food consumption patterns based on region and time of day.

[1365] "Means of providing data to other companies" refers to means of providing the consumer behavior data analyzed by the server to partner companies via APIs or other means.

[1366] The following describes in detail a specific embodiment of the present invention. The system is comprised of a detector installed in the food preservation device, a communication terminal connected to each household's food preservation device, and a computing device for data processing. This reduces food waste and optimizes consumption behavior.

[1367] Acquiring ingredient information

[1368] When a user places new ingredients in the food storage device, they use a dedicated barcode scanner to read the barcode of the ingredient. This operation obtains information about the ingredient, such as its name, expiration date, and quantity. The device receives this information via a detector in the food storage device.

[1369] Registering and saving food information

[1370] When the terminal receives the barcode information, it automatically retrieves the corresponding ingredient information from the database. The retrieved information is stored in the internal memory of the food storage device and then transmitted to the computing device via the Internet. The computing device then stores the received ingredient information in the database.

[1371] Expiration date monitoring

[1372] The computing device periodically monitors expiration dates based on the food ingredient information sent from the food storage device. When an ingredient approaches its expiration date, the computing device generates a notification message. This message is sent from the computing device to a terminal (smartphone app).

[1373] Expiration date notification

[1374] The notification data generated by the computing device is immediately sent to the terminal, which then notifies the user of a list of ingredients that are approaching their expiration dates, allowing the user to consume ingredients appropriately without missing the expiration dates.

[1375] Providing promotional information

[1376] The computing device periodically receives and stores sales information from affiliated retailers. It searches for sales information related to food items with approaching expiration dates and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and provides specific information.

[1377] Data analysis and provision

[1378] The computing device periodically collects and analyzes food data stored in the food storage device to generate consumption patterns based on region and time of day. This data is provided to partner companies, allowing them to optimize their supply chains based on demand forecasts.

[1379] Specific examples

[1380] 1. Example of food ingredient registration

[1381] When a user purchases milk and places it in the food storage device, they scan the barcode.

[1382] The terminal obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the computing device.

[1383] 2. Example of expiration date notification

[1384] A computing device monitors the expiration date of the milk and generates a notification on the seventh day saying, "Expires in two days."

[1385] The device displays a notification to the user through a smartphone app.

[1386] 3. Examples of sales promotion information provided

[1387] The computing device obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[1388] The device sends the user a sale notification such as "20% off milk."

[1389] 4. Data provision example

[1390] The computing device analyzes regional milk consumption data and identifies when consumption will increase in certain areas.

[1391] The computing device provides the data to partner companies, which then adjust their supply plans.

[1392] Prompt Sentence Examples

[1393] Please tell me in detail the steps by which the sensors in the refrigerator obtain food information.

[1394] What is the algorithm used by the server to generate expiration notifications?

[1395] Please tell us specifically how you will implement a system to obtain sale information and notify users.

[1396] By inputting these prompts into a generative AI model, we can obtain answers such as detailed explanations of each function and analysis of the algorithms.

[1397] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1398] Step 1:

[1399] When a user places a new ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode, which then acquires barcode information (input). The barcode information contains the ingredient's identification code.

[1400] Specific action: The user presses the button on the barcode scanner and reads the barcode on the milk.

[1401] Step 2:

[1402] The terminal receives the barcode information and retrieves detailed information such as the ingredient name, expiration date, and quantity from the database based on the ingredient's identification code (data processing). The retrieved ingredient information (output) is stored in the internal memory and simultaneously sent to the server.

[1403] Specific operation: The terminal receives the barcode "1234567890123", retrieves the information "Milk, expiration date: 10 days later, quantity: 1 liter" from the database, stores it in its internal memory, and then sends it to the server.

[1404] Step 3:

[1405] The server receives the food ingredient information sent from the terminal and stores it in a database (input). It periodically scans the stored data and monitors the expiration date (output).

[1406] What happens: The server stores the information "Milk, expiration date: 10 days later, quantity: 1 liter" in a database and runs a database query at 2 AM every day to scan for the expiration date.

[1407] Step 4:

[1408] The server detects ingredients whose expiration date is approaching (data calculation). If the expiration date is within a certain period of time (e.g., three days before) from the set date, it generates data to notify the user (output).

[1409] Specific operation: The server executes a database query to extract ingredients with expiration dates within three days. For example, it generates notification data such as "Milk expires in two days."

[1410] Step 5:

[1411] The server immediately sends the generated notification data to the device (smartphone app) (input). The device receives the notification data and displays it to the user (output).

[1412] Specific operation: The server sends notification data to the device's push notification API, stating that the milk's expiration date is in two days. The device displays the data on the smartphone app.

[1413] Step 6:

[1414] The server obtains sales information through the API of partner retailers and periodically saves it (input). It searches for sales information related to food items that are close to their expiration date and sends that information to the terminal (output).

[1415] Specific operation: The server retrieves the "20% off milk" sale information from the API of the partner retailer, associates it with information about milk that is nearing its expiration date, and notifies the user.

[1416] Step 7:

[1417] The server periodically collects consumption data on food items stored in the refrigerator and generates consumption behavior patterns based on region and time of day (data analysis: input). The generated consumption behavior data is provided to partner companies (output).

[1418] Specific operation: The server performs monthly data analysis and sends the results of "analysis of peak milk consumption in specific regions" to partner companies via API.

[1419] This allows the system to manage ingredients, notify expiration dates, provide economical purchasing timing, and contribute to supply chain optimization.

[1420] (Application example 1)

[1421] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1422] In managing food in refrigerators, there is a need to reduce the time and errors involved in manually entering food information, efficiently monitor expiration dates, and reduce food waste. It is also expected that the economic burden will be reduced by providing appropriate promotional information for food items approaching their expiration dates in a timely manner. Furthermore, there is a demand for providing companies with data on consumer behavior based on region and time of day to optimize supply plans.

[1423] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1424] In this invention, the server includes a means for acquiring information about ingredients using sensors installed in the refrigerator, a means for transmitting the ingredient information to the server, a means for the server to monitor the expiration dates of ingredients and notify the user when the expiration date is approaching, and a means for the user to scan barcodes using the camera on their smart device when storing ingredients in the refrigerator and automatically register the ingredient information. This eliminates the need to manually enter ingredient information, streamlines expiration date management, and reduces food waste. Furthermore, by providing promotional information at the appropriate time, it is possible to reduce financial burdens, and by providing companies with data on consumer behavior based on region and time of day, it is possible to optimize supply plans.

[1425] A "sensor installed inside a refrigerator" is a device that monitors the environment inside the refrigerator and obtains information about ingredients (e.g., location, temperature, weight, etc.).

[1426] "Ingredient information" is detailed data about ingredients, such as the name, quantity, expiration date, and barcode information of the ingredients.

[1427] A "server" is a computer system that communicates with external devices via a network and stores, processes, and analyzes data.

[1428] A "use-by date" is the date by which food is considered safe to eat, and quality and safety may not be guaranteed after that date.

[1429] A "notification" is a message or alert that the system uses to inform the user.

[1430] A "smart device" is an electronic device that is connected to the Internet and has advanced functions that allow it to run a variety of applications.

[1431] A "camera" is a device used to take images and videos, and is also used to read barcodes.

[1432] A "barcode" is a method of expressing information by arranging numbers and letters in a specific pattern that can be read by a machine.

[1433] "Automatic registration" is a process in which a machine or system automatically enters and stores information without human intervention.

[1434] The following describes in detail specific embodiments of the present invention. The system comprises a sensor installed in a refrigerator, a terminal (smart device) that connects to the refrigerator in each home, and a server.

[1435] Food Registration

[1436] When a user places new ingredients in the refrigerator, they use the camera on their smart device to read the barcode of the ingredient. This barcode information is sent to the terminal, and information such as the ingredient's name, expiration date, and quantity is automatically registered. The terminal stores this information in the refrigerator's internal memory and simultaneously sends it to the server. This eliminates the need for manual input and prevents input errors.

[1437] Monitoring expiration dates of ingredients

[1438] The server periodically receives and stores information about ingredients sent from the refrigerator. It also monitors the expiration dates of ingredients and generates data to notify the user when the expiration date falls within a certain period of time (e.g., three days) from the set date.

[1439] Expiration date notice

[1440] The notification data generated by the server is immediately sent to the device. The device displays a list of ingredients that are approaching their expiration date and provides audio and visual notifications to the user. This allows the user to avoid overlooking expiration dates and consume ingredients appropriately without wasting them.

[1441] Providing promotional information

[1442] The server also periodically receives and stores sales information from affiliated retailers. For food ingredients that are nearing their expiration date, the server searches for related sales information and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and displays detailed information. This allows the user to make purchases at an economically advantageous time.

[1443] Data provision and supply chain optimization

[1444] The server periodically collects and analyzes data on ingredients stored in the refrigerator, generating consumption patterns based on region and time of day. This consumption data is provided to other partner companies, who can use it to forecast demand and optimize the entire supply chain.

[1445] Hardware and software used

[1446] The following hardware and software are used to implement this system:

[1447] Smart Devices: Use your smartphone or smart glasses to scan the barcode with the built-in camera.

[1448] Server: A computer system for receiving, storing, analyzing, and notifying data.

[1449] Python: A scripting language used for data processing and analysis on the server.

[1450] OpenCV: A library for camera operation and barcode scanning.

[1451] Requests: A library for HTTP communication with the server.

[1452] Specific examples

[1453] Example of ingredient registration:

[1454] When a user purchases milk and puts it in the refrigerator, they scan the barcode using the camera on their smart device.

[1455] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[1456] Example of expiration date notification:

[1457] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[1458] The terminal displays a notification to the user on the smart device.

[1459] Examples of promotional information provided:

[1460] The server obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[1461] The device sends the user a sale notification such as "20% off milk."

[1462] Data provided example:

[1463] The server analyzes regional milk consumption data and identifies periods when consumption increases in specific regions.

[1464] The server provides the data to partner companies, which then adjust their supply plans.

[1465] Prompt Sentence Examples

[1466] "Please use the camera to read the barcode inside the refrigerator and register the new food information."

[1467] "Show me notifications about ingredients approaching their expiration date"

[1468] "Get relevant sales information and notifications on your smart device."

[1469] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1470] Step 1:

[1471] When a user places new ingredients in the refrigerator, they scan the barcode using the smart device's camera. At this time, the smart device receives the barcode image as input and extracts the barcode information. Specifically, the camera captures the image and uses the OpenCV library to detect and read the barcode. Once the barcode information is obtained, it is processed into ingredient information including the ingredient's name, expiration date, quantity, etc. and saved on the device.

[1472] Step 2:

[1473] The terminal sends the ingredient information acquired in step 1 to the server. The data sent includes the ingredient's barcode information, name, expiration date, quantity, etc. This information is sent to the server as an HTTP request, and the terminal processes to receive the response (success / failure status code) from the server.

[1474] Step 3:

[1475] The server stores the received ingredient information in a database. This database manages detailed information about ingredients stored in each household's refrigerator. The input ingredient information is added to the database, and the storage result is sent to the terminal as a response.

[1476] Step 4:

[1477] The server periodically monitors the food ingredient information in the database and checks the expiration date of each ingredient. When a specified period (e.g., three days before the expiration date) has passed, notification data for that ingredient is generated. The input food ingredient information is used as the source data, and an algorithm determines whether the expiration date is approaching and outputs the notification data.

[1478] Step 5:

[1479] The notification data generated by the server is immediately sent to the device. The device then displays this notification data to the user. For example, a message such as "The expiration date is in two days" can be displayed on the smart device's display to clearly communicate the information to the user. The notification message can be displayed as a voice notification or a visual pop-up.

[1480] Step 6:

[1481] The server periodically receives sales information from affiliated retailers and stores it in a database. Here, the input is the retailer's sales information (e.g., 20% off milk), and the server processes the sales information to add and update it in the database.

[1482] Step 7:

[1483] The server associates sale information with food ingredient information and searches the database for sale information related to food ingredients that are close to their expiration date. For example, if milk is close to its expiration date, the server searches for related "20% off milk" information and generates it as notification data.

[1484] Step 8:

[1485] The server generates sales promotion information and sends it to the terminal, which then displays it to the user. The user can see the sales promotion information on the smart device display, such as "There are related products on sale," and is encouraged to make purchases at economically advantageous times.

[1486] Step 9:

[1487] The server periodically collects and analyzes food data from the refrigerator and generates consumption behavior patterns based on region and time of day. Here, food information data is used as input and consumption patterns are output using statistical analysis and machine learning models. The generated consumption behavior patterns are stored on the server.

[1488] Step 10:

[1489] The server provides the generated consumer behavior patterns to partner companies and other related organizations. Partner companies can use this data to make demand forecasts and optimize their entire supply chains. Data is provided to companies in the form of automatic transmissions via APIs and regular reports.

[1490] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1491] A specific embodiment of the present invention will be described in detail below. The system comprises a sensor installed in a refrigerator, a terminal linked to each household refrigerator, a server, and an emotion engine that recognizes the user's emotions.

[1492] Food Registration

[1493] First, we will explain the operation when a user stores new ingredients in the refrigerator. When a user places an ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode. This barcode information is sent to the terminal (a sensor system inside the refrigerator), and information such as the ingredient's name, expiration date, and quantity is automatically registered. The terminal saves this information in the refrigerator's internal memory and sends the registration data to a server.

[1494] Monitoring expiration dates of ingredients

[1495] The server periodically receives and stores information about ingredients sent from the refrigerator. It also monitors the expiration dates of ingredients and generates data to notify the user when the expiration date falls within a certain period of time (e.g., three days) from the set date.

[1496] Expiration date notice

[1497] The notification data generated by the server is immediately sent to the device (smartphone app). The device then notifies the user of a list of ingredients that are approaching their expiration date. This allows the user to avoid overlooking ingredients that are approaching their expiration date and consume them appropriately. As a result, food waste can be reduced.

[1498] Providing promotional information

[1499] The server also periodically receives and stores sales information from affiliated retailers. For food items nearing their expiration date, the server searches for related sales information and sends that information to the terminal. The terminal then notifies the user that there are related products on sale and provides specific information, allowing the user to make purchases at an economically advantageous time.

[1500] Emotion engine integration

[1501] The emotion engine collects user emotion data and transmits it to a server in real time. The emotion engine is installed in devices such as smartphones and smart speakers and recognizes emotions from the user's tone of voice and facial expressions.

[1502] Optimizing notifications based on emotions

[1503] The server analyzes the emotion data received from the emotion engine and optimizes the content and timing of notifications based on the user's current emotional state. For example, if the user is stressed, a reminder notification will be sent in a gentle tone, while if the user is relaxed, a notification prompting a specific action will be sent.

[1504] Customize promotional information based on emotions

[1505] The server detects the user's emotional data and personalizes relevant promotional information. For example, if the user is happy, it can provide healthy recipes and discount coupons related to ingredients that are close to their expiration date, encouraging the user to make a purchase.

[1506] Sentiment Data Analysis and Supply Chain Optimization

[1507] The server periodically collects and analyzes data on ingredients and emotions in the refrigerator, and reflects this data in consumer behavior patterns based on region and time of day. This data is provided to partner companies, who can use this information to forecast demand and optimize the entire supply chain.

[1508] Specific examples

[1509] 1. Example of food ingredient registration

[1510] When a user buys milk and puts it in the refrigerator, they scan the barcode.

[1511] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[1512] 2. Example of expiration date notification

[1513] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[1514] The device displays a notification to the user through a smartphone app.

[1515] 3. Example of Emotion-Based Notification

[1516] The emotion engine recognizes that the user is feeling stressed from their tone of voice.

[1517] Instead of the usual reminder notification, the server notifies you with a soft tone message saying "Your milk is about to expire."

[1518] 4. Examples of sales promotion information provided

[1519] The server obtains milk sale information from affiliated supermarkets and associates it with milk that is approaching its expiration date.

[1520] The device sends the user a sale notification such as "20% off milk."

[1521] 5. Data provision example

[1522] The server analyzes regional milk consumption data and identifies periods when consumption increases in specific regions.

[1523] The server provides the data to partner companies, which then adjust their supply plans.

[1524] Through the above-described embodiments, the system contributes to reducing food waste, optimizing consumer behavior, and optimizing the supply chain. Furthermore, by integrating an emotion engine, the system can provide a more personalized user experience and support efficient food ingredient management.

[1525] The processing flow will be explained below.

[1526] Food Registration

[1527] Step 1:

[1528] When a user places purchased ingredients in the refrigerator, they scan the barcode of the ingredients with a dedicated barcode scanner.

[1529] Step 2:

[1530] The terminal (a sensor system inside the refrigerator) receives the barcode and stores the food ingredient information (name, expiration date, quantity, etc.) in a database.

[1531] Step 3:

[1532] The food ingredient information acquired by the terminal is transmitted to the server.

[1533] Monitoring expiration dates of ingredients

[1534] Step 1:

[1535] The server periodically (e.g., once a day) receives the latest ingredient information from each refrigerator.

[1536] Step 2:

[1537] The server monitors the expiration date of the ingredients received, and sets a "warning flag" if the expiration date falls within a certain period of time (e.g., three days) from the set date.

[1538] Expiration date notice

[1539] Step 1:

[1540] The server generates a list of ingredients whose expiration date is approaching and searches for ingredients that satisfy the notification conditions.

[1541] Step 2:

[1542] The server generates notification data and sends it to the device (smartphone app).

[1543] Step 3:

[1544] The device sends a notification to the user's smartphone and displays a list of ingredients that are approaching their expiration date.

[1545] Providing promotional information

[1546] Step 1:

[1547] The server periodically obtains sales information from affiliated stores and stores it in a database.

[1548] Step 2:

[1549] The server searches for sales promotion information related to food ingredients that are close to their expiration date.

[1550] Step 3:

[1551] The device notifies the user of promotional information. Specifically, the smartphone app notifies the user that "XX is on sale."

[1552] Emotion engine integration

[1553] Step 1:

[1554] When a user is in front of the refrigerator or in the kitchen, the emotion engine collects the user's emotional data through the microphone and camera on their smartphone or smart speaker.

[1555] Step 2:

[1556] The emotion engine recognizes emotions from the user's tone of voice and facial expressions, and sends the data to the server in real time.

[1557] Optimizing notifications based on emotions

[1558] Step 1:

[1559] The server analyzes the emotion data received from the emotion engine.

[1560] Step 2:

[1561] The server adjusts the content and timing of notifications based on the user's current emotional state.

[1562] Step 3:

[1563] The device will notify the user with a message tailored to their emotional state, such as "The milk's expiration date is approaching."

[1564] Customize promotional information based on emotions

[1565] Step 1:

[1566] The server personalizes relevant promotional information based on the user's emotional data.

[1567] Step 2:

[1568] The server generates promotional information that matches the emotional state of the user.

[1569] Step 3:

[1570] The device sends notifications appropriate to the user's emotional state, such as "There are related products on sale."

[1571] Data provision and supply chain optimization

[1572] Step 1:

[1573] The server periodically collects and analyzes food data and emotional data from the refrigerator.

[1574] Step 2:

[1575] The server generates consumption behavior data based on region and time period.

[1576] Step 3:

[1577] The server provides the generated consumer behavior data to partner companies, who use this data to forecast demand and optimize their supply chains.

[1578] Through the above processing steps, this system effectively manages ingredients, reduces food waste, and optimizes the supply chain. In addition, the integration of an emotion engine provides a more personalized user experience and supports efficient ingredient management.

[1579] Example 2

[1580] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1581] Currently, managing food expiration dates is a hassle for many households, leading to food waste. Furthermore, notifications sent without considering the user's emotional state can increase their stress. Furthermore, there is a lack of effective ways to utilize consumer behavior data, preventing demand forecasting and supply chain optimization.

[1582] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for acquiring food information using a detection device installed in the refrigerator, a means for transmitting the food information to the server, a means for the server to monitor the food's expiration date and notify the user when it determines that the expiration date is approaching, an emotion recognition device for collecting the user's emotional data and transmitting it to the server, and a means for the server to optimize the content and timing of notifications based on the user's emotional data. This enables users to efficiently manage food expiration dates and reduce food waste. Furthermore, personalized notifications based on the user's emotional state reduce stress and provide a more comfortable user experience. Furthermore, analyzing and providing consumer behavior data enables demand forecasting and supply chain optimization.

[1583] A "detection device" is a device such as a sensor that is installed inside a refrigerator to obtain information about food.

[1584] "Food information" refers to data about food, such as the name, expiration date, and quantity of the food.

[1585] "Server" means a central system that receives, stores, analyzes food information and other related data, and provides notifications and information.

[1586] An "emotion recognition device" is a device that collects emotional data from the user's tone of voice, facial expressions, etc.

[1587] "Emotion data" is information that reflects the user's emotional state and is used to optimize the content and timing of notifications.

[1588] "Notification means" refers to the means used by the server to provide information to the user, and includes, for example, notification via a smartphone app.

[1589] "Sales promotion information" is information that provides economic benefits to users, such as sale information and coupons provided by affiliated sales organizations.

[1590] "Consumption behavior data" is data that analyzes food consumption patterns based on region and time of day, and is used for demand forecasting and supply chain optimization.

[1591] The present invention is a system for managing food in a refrigerator and personalizing the user experience, and consists of the following components:

[1592] System Configuration

[1593] Detection device

[1594] The detector installed in the refrigerator includes a sensor for reading the barcodes of food items. This sensor acquires the barcode information of the food items and transmits it to a terminal for storage in the refrigerator's internal memory. For example, when the barcode of milk is read, the information is transmitted to the terminal and data such as the name of the food, expiration date, and quantity are recorded.

[1595] Terminal

[1596] The terminal is a device connected to the refrigerator, which processes the food information received from the detector, stores it in its internal memory, and transmits the information to the server.

[1597] server

[1598] The server receives food information from the device and stores it in a database. Based on this information, the server monitors food expiration dates and generates notifications when the expiration date approaches. The server also obtains sales promotion information from affiliated sales organizations and notifies users of information related to foods that are approaching their expiration date.

[1599] emotion recognition device

[1600] An emotion recognition device is a device that analyzes a user's tone of voice and facial expressions to collect emotional data. This device is embedded in smartphones, smart speakers, etc. and transmits the emotional data to a server in real time.

[1601] Server Features

[1602] Expiration date monitoring and notification

[1603] The server monitors the food information in the refrigerator and generates a notification a certain period of time (e.g., three days) before the expiration date. For example, when the expiration date of milk approaches, the server creates a notification stating, "Milk expires in two days," and sends it to the user via the terminal.

[1604] Providing promotional information

[1605] The server periodically receives and stores promotional information from affiliated distributors, searches for promotional information related to food products that are close to their expiration date, and sends notifications such as "20% off milk" to the terminal.

[1606] Optimizing notifications based on emotions

[1607] The server analyzes the emotion data received from the emotion recognition device and optimizes the content and timing of notifications based on the user's current emotional state. For example, if the user is feeling stressed, the server may send a reminder with a soft-toned message.

[1608] Analysis and provision of consumer behavior data

[1609] The server periodically analyzes the food and sentiment data stored in the refrigerator to generate patterns of consumer behavior by region and time of day, which is then provided to partner companies for the purposes of demand forecasting and supply chain optimization.

[1610] Specific examples

[1611] 1. Example of food ingredient registration

[1612] When a user buys milk and puts it in the refrigerator, they scan the barcode.

[1613] The device obtains milk information (name: milk, expiration date: 10 days later, quantity: 1 liter) and sends it to the server.

[1614] 2. Example of expiration date notification

[1615] The server monitors the expiration date of milk and generates a notification on the 7th day saying "Expires in 2 days."

[1616] The device displays a notification to the user through a smartphone app.

[1617] 3. Example of Emotion-Based Notification

[1618] The emotion recognition device recognizes that the user is feeling stressed from the tone of their voice.

[1619] Instead of the usual reminder notification, the server notifies you with a soft tone message saying "Your milk is about to expire."

[1620] 4. Examples of sales promotion information provided

[1621] The server acquires milk sale information from affiliated sales organizations and associates it with milk whose expiration date is approaching.

[1622] The device sends the user a sale notification such as "20% off milk."

[1623] This system allows users to efficiently manage food expiration dates and reduce food waste. It also provides notifications based on the user's emotional state, reducing stress and providing a better user experience. Furthermore, by analyzing and providing consumer behavior data, companies can forecast demand and optimize their supply chains.

[1624] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1625] Step 1:

[1626] When a user puts a new ingredient in the refrigerator, they use a dedicated barcode scanner to read the ingredient's barcode. The input is the ingredient's barcode, and the output is the barcode information. A detection device reads this barcode information and sends it to a terminal. For example, when a user puts milk in the refrigerator, the scanner reads the milk's barcode and sends the information to the terminal.

[1627] Step 2:

[1628] Based on the barcode information received by the terminal from the detector, detailed information about the ingredients (such as name, expiration date, and quantity) is obtained from the food database. The input is the barcode information, and the output is detailed information about the ingredients. This detailed information is stored in the terminal's internal memory and sent to the server. For example, when the terminal receives barcode information about milk, it obtains the data "Milk, expiration date: 10 days later, quantity: 1 liter" and sends it to the server.

[1629] Step 3:

[1630] The server receives the ingredient information sent from the terminal and saves it in its own database. The input is the ingredient information sent from the terminal, and the output is the saved data. The server uses this data to monitor the expiration date in detail. For example, the server receives data such as "Milk, expiration date: 10 days later, quantity: 1 liter" and saves it in its own database.

[1631] Step 4:

[1632] The server monitors food expiration dates based on stored data. The input is the stored food ingredient information, and the output is notification data generated based on the expiration date. A notification is generated a certain period of time (e.g., three days) before the expiration date is set, and sent to the terminal. For example, the server monitors the expiration date of milk, and when seven days have passed, it generates a notification stating, "The expiration date is two days later."

[1633] Step 5:

[1634] The device receives the notification data sent from the server and displays the notification to the user through the smartphone app. The input is the notification data from the server, and the output is the notification displayed to the user. For example, the device displays a notification on the smartphone app saying, "The expiration date for the milk is in two days."

[1635] Step 6:

[1636] The server periodically receives and stores promotional information from affiliated sales organizations. The input is the promotional information from the sales organization, and the output is the stored promotional information. For example, the server receives and stores sale information about "20% off milk" from an affiliated supermarket.

[1637] Step 7:

[1638] The server searches for promotional information related to food ingredients that are close to their expiration date and sends that information to the terminal. The input is information about food ingredients that are close to their expiration date and stored promotional information, and the output is notification data for the related promotional information. For example, the server searches for sale information about "20% off milk" related to milk that is close to its expiration date and sends that information to the terminal.

[1639] Step 8:

[1640] The emotion recognition device collects the user's tone of voice and facial expression data and sends it to the server in real time. The input is the user's emotional data, and the output is the emotional data sent to the server. For example, the emotion recognition device recognizes that the user is feeling stressed from the tone of their voice and sends that data to the server.

[1641] Step 9:

[1642] The server analyzes the received emotion data and optimizes the content and timing of notifications. The input is emotion data, and the output is optimized notification data. For example, if the user is feeling stressed, the server generates a soft-toned message saying, "Your milk is about to expire."

[1643] Step 10:

[1644] The server periodically analyzes food ingredient data and emotion data stored in the refrigerator to generate consumption behavior patterns by region and time period. The input is food ingredient data and emotion data, and the output is consumption behavior pattern data. This data is provided to partner companies and used for demand forecasting and supply chain optimization. For example, the server analyzes milk consumption data by region and provides that data to companies to adjust their supply plans.

[1645] (Application example 2)

[1646] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1647] The main functions of conventional food management systems are to notify users of expiration dates and provide promotional information, but they are unable to optimize notifications that take into account the user's emotional state or make meal suggestions based on their emotions, which means that they are unable to reduce users' stress or personalize their experience.In particular, in food delivery services, the lack of suggestions based on the user's emotional state and ingredients that are close to their expiration date has led to issues that result in lower user satisfaction and food waste.

[1648] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring ingredient information using a sensor installed in the refrigerator, means for transmitting the ingredient information to the server, means for monitoring the expiration dates of ingredients and notifying the user when it is determined that the expiration date is approaching, means for acquiring emotion data using an emotion engine that recognizes the user's emotional state, and means for optimizing the content and timing of the notification based on the emotion data. This enables expiration date notifications and meal suggestions that take the user's emotional state into consideration, making it possible to reduce ingredient waste and provide a more personalized food delivery experience.

[1649] A "sensor installed inside a refrigerator" is a device that is attached inside a refrigerator and detects information about food ingredients (such as name, expiration date, quantity, etc.).

[1650] A "server" is a computer system that receives, stores, and processes ingredient information and emotion data.

[1651] "Ingredient information" refers to data such as the name, expiration date, and quantity of each ingredient stored in the refrigerator.

[1652] The "means for monitoring expiration dates" is a function that continuously checks the expiration dates of ingredients and generates an alert when the expiration date approaches.

[1653] "Means for notifying users" refers to a function that sends information about the expiration date of ingredients or promotional information about specific ingredients to the user's device (e.g., smartphone).

[1654] The "emotion engine" is a system that analyzes the user's voice and facial expression data and recognizes the user's emotional state in real time.

[1655] "Emotion data" is data obtained by the emotion engine that represents the user's emotional state (e.g., joy, stress, relaxation, etc.).

[1656] "Means for optimizing notification content and timing" refers to a function that adjusts the content and timing of expiration date notifications and meal suggestions based on emotional data, providing information in the most optimal form for the user.

[1657] "Sales promotion information" refers to various types of marketing information provided to consumers, such as sale information and discount information provided by affiliated retailers.

[1658] The following describes in detail specific embodiments of the present invention. The present invention comprises a sensor installed in a refrigerator, a server that manages ingredient information, an emotion engine, and a user terminal.

[1659] First, a sensor installed in the refrigerator acquires information such as the name, expiration date, and quantity of each ingredient. Specifically, when a user places new ingredients in the refrigerator, a barcode scanner is used to read the ingredient's barcode. This information is sent to the sensor system inside the refrigerator, and the ingredient information is automatically registered. The ingredient information is then sent to the server and stored in the server's database.

[1660] Next, the server periodically monitors the ingredient information. The server continuously checks the expiration date of the ingredients, and when the expiration date approaches, it generates notification data and sends it to the user's device (e.g., smartphone). This allows the user to consume ingredients appropriately without missing any ingredients that are approaching their expiration date.

[1661] The emotion engine analyzes the user's voice and facial expressions to obtain emotional data. For example, it recognizes the user's emotional state, such as whether they are stressed or relaxed, from their voice tone and facial expression. This emotional data is sent to the server in real time.

[1662] The server optimizes the content and timing of notifications based on the acquired emotional data. For example, if the user is stressed, the notification will be delivered in a softer tone, while if the user is relaxed, the notification will prompt specific actions.

[1663] In addition, the server obtains sales promotion information from affiliated retailers. For food items nearing their expiration date, the server searches for related sales promotion information and sends that information to the user's device. This allows the user to purchase food items at an economically advantageous time.

[1664] Consider the following scenario as a concrete example. When a user scans the barcode of milk when putting it in the refrigerator, a sensor inside the refrigerator acquires the milk's information and sends it to a server. The server monitors the expiration date and, when the expiration date approaches, notifies the user's smartphone that "The milk's expiration date is approaching." If the user is feeling stressed, the notification will be made in a softer tone. The server also obtains milk sale information from affiliated stores and notifies the user.

[1665] An example of how a generative AI model can be used is the following prompt:

[1666] Example prompt:

[1667] Generate meal suggestions suitable for a stressed user based on the expiration dates of ingredients in the refrigerator and the user's emotional data. For example, suggest a relaxing dessert or a stress-relieving menu.

[1668] As a result, this system improves the efficiency of food management in the refrigerator and provides optimal information and experience for users.

[1669] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1670] Step 1:

[1671] When a user puts an ingredient in the refrigerator, they scan the barcode. The input is the ingredient's barcode data. The barcode scanner is used to process the data, obtaining information such as the ingredient's name, expiration date, and quantity. The output is the obtained ingredient information.

[1672] Step 2:

[1673] The sensor system inside the refrigerator sends the acquired ingredient information to a server. The input is the ingredient information obtained from the sensor system. The sensor system performs data calculations to transfer this data to a designated server. The output is the ingredient information sent to the server.

[1674] Step 3:

[1675] The server stores the received ingredient information in a database. The input is the ingredient information sent from the sensor system. The server processes the data by adding new ingredient information to the database. The output is the stored ingredient information.

[1676] Step 4:

[1677] The server periodically monitors the ingredient information in the database. The input is the ingredient information in the database. The server performs data calculations to detect the expiration date of each ingredient and generate notification data when the expiration date approaches. The output is the generated notification data.

[1678] Step 5:

[1679] The server sends the generated notification data to the user's device (e.g., smartphone). The input is the notification data. The server performs data calculations to transfer the notification data from the database. The output is the notification sent to the smartphone.

[1680] Step 6:

[1681] The emotion engine analyzes the user's voice and facial expression data to generate emotion data. The input is the user's voice data and facial expression data. The emotion engine performs data analysis to recognize the user's emotional state. The output is emotion data.

[1682] Step 7:

[1683] Emotion data is sent to the server. The input is emotion data provided by the emotion engine. The server stores this in a database and performs data calculations to optimize the content and timing of notifications. The output is optimized notification data.

[1684] Step 8:

[1685] The server obtains promotional information from affiliated retailers and searches for promotional information related to food ingredients with an approaching expiration date. The input is the promotional information from the retailer and the food ingredient database information. The server performs data operations to search for and integrate related promotional information. The output is notification data including the promotional information.

[1686] Step 9:

[1687] The optimized notification data and promotional information are sent to the user's terminal. The input is the notification data and promotional information sent from the server. The user's terminal processes the data and displays it as a notification. The output is the user's receipt of the notification.

[1688] The above are the specific processing steps of the system program that realizes this application example. The specific operations performed in each step smoothly manage food ingredient information, notify expiration dates, and optimize emotional state recognition and notification.

[1689] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1690] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1691] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1692] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1693] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1694] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1695] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1696] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1697] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1698] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1699] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1700] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1701] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1702] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1703] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1704] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1705] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1706] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1707] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1708] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1709] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1710] The following is further disclosed regarding the above embodiment.

[1711] (Claim 1)

[1712] A means of acquiring information about ingredients using sensors installed inside the refrigerator;

[1713] means for transmitting the ingredient information to a server;

[1714] A means for the server to monitor the expiration date of ingredients and notify the user when it determines that the expiration date is approaching;

[1715] A system including:

[1716] (Claim 2)

[1717] 2. The system according to claim 1, further comprising means for acquiring sales promotion information from affiliated retailers, and notifying the user of sales promotion information relating to foodstuffs whose expiration date is approaching.

[1718] (Claim 3)

[1719] The system according to claim 1, further comprising means for the server to analyze the food ingredient data in the refrigerator, generate data on consumption behavior based on region and time period, and provide the data to other companies.

[1720] "Example 1"

[1721] (Claim 1)

[1722] A means for acquiring information about ingredients using a detector installed in the food preservation device;

[1723] means for transmitting the ingredient information to a computing device via a communication device;

[1724] a computing device for monitoring expiration dates of ingredients and notifying a user when it determines that the expiration date is approaching;

[1725] a means for acquiring sales promotion information from affiliated retailers and notifying a user of the sales promotion information relating to foodstuffs whose expiration date is approaching;

[1726] A computing device analyzes the food ingredient data in the food storage device, generates data on consumption behavior based on region and time period, and provides the data to other companies;

[1727] A system including:

[1728] (Claim 2)

[1729] 10. The system of claim 1, further comprising means for the computing device to scan ingredient barcodes within the food storage device and store the information in the storage device.

[1730] (Claim 3)

[1731] 10. The system of claim 1, further comprising means for transmitting notification data generated by the computing device based on the expiration date to a communication terminal of a user and visually displaying the information to the user.

[1732] "Application Example 1"

[1733] (Claim 1)

[1734] A means of acquiring information about ingredients using sensors installed inside the refrigerator;

[1735] means for transmitting ingredient information to a server;

[1736] A server monitors the expiration date of ingredients and notifies the user when the expiration date is approaching;

[1737] A means for automatically registering food information by scanning barcodes using a camera on a smart device when a user stores food in a refrigerator;

[1738] A system including:

[1739] (Claim 2)

[1740] 2. The system according to claim 1, further comprising means for acquiring sales promotion information from affiliated retailers, and notifying the user of sales promotion information relating to foodstuffs whose expiration date is approaching.

[1741] (Claim 3)

[1742] The system according to claim 1, further comprising means for the server to analyze the food ingredient data in the refrigerator, generate data on consumption behavior based on region and time period, and provide the data to other companies.

[1743] "Example 2: Combining Emotion Engines"

[1744] (Claim 1)

[1745] A means for acquiring information about food using a detection device installed in the refrigerator;

[1746] means for transmitting the food information to a server;

[1747] A means for the server to monitor the expiration date of food and notify the user when it determines that the expiration date is approaching;

[1748] an emotion recognition device that collects emotion data of a user and transmits it to a server;

[1749] A means for the server to optimize the content and timing of notifications based on the user's emotional data;

[1750] A system including:

[1751] (Claim 2)

[1752] 2. The system of claim 1, further comprising means for the server to acquire promotional information from affiliated sales organizations and notify the user of promotional information related to food products that are approaching their expiration date.

[1753] (Claim 3)

[1754] 10. The system of claim 1, further comprising means for the server to analyze the food data in the refrigerator, generate data on consumption behavior based on region and time period, and provide the data to other organizations.

[1755] "Application example 2 when combining emotion engines"

[1756] (Claim 1)

[1757] A means of acquiring information about ingredients using sensors installed inside the refrigerator;

[1758] means for transmitting the ingredient information to a server;

[1759] A means for the server to monitor the expiration date of ingredients and notify the user when it determines that the expiration date is approaching;

[1760] means for acquiring emotion data using an emotion engine that recognizes the user's emotional state;

[1761] means for optimizing notification content and timing based on emotion data;

[1762] A system including:

[1763] (Claim 2)

[1764] 2. The system according to claim 1, further comprising means for acquiring sales promotion information from affiliated retailers, and notifying the user of sales promotion information relating to foodstuffs whose expiration date is approaching.

[1765] (Claim 3)

[1766] The system according to claim 1, further comprising means for the server to analyze the food ingredient data in the refrigerator, generate data on consumption behavior based on region and time period, and provide the data to other companies. [Explanation of symbols]

[1767] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of acquiring information about ingredients using sensors installed inside the refrigerator; means for transmitting the ingredient information to a server; A means for the server to monitor the expiration date of ingredients and notify the user when it determines that the expiration date is approaching; A system including:

2. The system according to claim 1, further comprising means for acquiring sales promotion information from affiliated retailers, and notifying the user of sales promotion information relating to foodstuffs whose expiration date is approaching.

3. The system according to claim 1, further comprising means for the server to analyze the food ingredient data in the refrigerator, generate data on consumption behavior based on region and time period, and provide the data to other companies.

Images