System

Abstract

A system is provided.SOLUTION: A system includes a means for receiving purchased commodity information from an electronic settlement application, a means for determining whether or not the purchased commodity needs to be stored in a refrigerator by an AI, a means for registering information on the purchased commodity determined to need to be stored in the refrigerator in a refrigerator sub-application, and a means for notifying user terminals of registration information on the purchased commodity.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Today, many consumers lead busy lives that make it difficult to manage purchased ingredients and plan meals appropriately. They also often waste food because they don't know how much food is being consumed in their refrigerators. While there is a need for a system that automatically manages the ingredients in a refrigerator and provides recipe suggestions based on that information, no effective system exists. Furthermore, refrigerator manufacturers lack a consistent way to incorporate this functionality into all refrigerators they market. [Means for solving the problem]

[0005] This invention provides a system that links an electronic payment app with a refrigerator sub-app. Specifically, an AI model analyzes purchased product information received from the electronic payment app and determines whether the product can be stored in the refrigerator. Product information that requires storage in the refrigerator is registered in the refrigerator sub-app and notified to the user's device. In addition, information about ingredients in the refrigerator is updated based on data from the refrigerator sub-app, and recipes are generated based on the ingredients the user has. This allows users to automate the management of purchased items and create efficient meal plans. Furthermore, a common API is provided to refrigerator manufacturers, which links data from interior cameras and sensors to automatically update information about ingredients in the refrigerator. A system is also provided for managing API usage and license revenue.

[0006] An "electronic payment app" is an application that allows users to make online payments for the purchase of goods and services using smartphones or other digital devices.

[0007] "Purchased product information" refers to data such as the name, category, quantity, and price of products purchased through an electronic payment app.

[0008] An "AI model" is an algorithm that uses artificial intelligence technology to analyze data and make judgments.

[0009] "Possibility of storing in refrigerator" is an index that indicates whether or not a product needs to be stored in a refrigerator based on the characteristics of the product.

[0010] The "refrigerator sub-app" is an application for managing and tracking food items in the refrigerator.

[0011] A "user terminal" is a device used by a user, such as a smartphone, tablet, or PC.

[0012] "Purchased product registration information" is data registered in the sub-app so that purchased products can be stored in the refrigerator.

[0013] A "refrigerator manufacturer" is a company that manufactures and sells refrigerators.

[0014] A "common API" is a common interface that enables data exchange between different systems and applications.

[0015] "Sensor data" is data obtained from sensors that measure the temperature, humidity, position, etc. inside the refrigerator.

[0016] "Camera data" is data obtained from videos and images taken by a camera placed inside the refrigerator.

[0017] A "recipe" is information that shows how to make a dish using specific ingredients and cooking procedures.

[0018] "License revenue" refers to revenue earned through licensing fees for using APIs or specific technologies.

[0019] A "system" is a collection of multiple components and applications that work together to achieve a specific function. [Brief explanation of the drawings]

[0020] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6]FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0021] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0022] First, the terms used in the following description will be explained.

[0023] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0024] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0025] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0026] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0027] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0028] [First embodiment]

[0029] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0030] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0031] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0032] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0033] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0034] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0035] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0036] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0037] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0038] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0039] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0040] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0041] 1. System Overview

[0042] The system of this invention receives information about products purchased by users through an electronic payment app, and based on that information, AI determines whether the product needs to be stored in the refrigerator and automatically registers that information in the refrigerator sub-app. It also includes a function to manage information about ingredients in the refrigerator and suggest recipes to users based on that information. This system works in conjunction with refrigerator manufacturers, acquiring sensor and camera data through a common API, allowing it to update information about ingredients in the refrigerator in real time.

[0043] 2. System Configuration and Operation

[0044] Receiving purchase information

[0045] User:

[0046] When a user purchases ingredients at a supermarket or online shop and pays using an electronic payment app, purchased product information is generated.

[0047] Device:

[0048] Purchase information (product name, category, quantity, etc.) is sent from the electronic payment app to the server.

[0049] server:

[0050] The server receives the purchased product information and uses an AI model to determine whether it needs to be stored in a refrigerator.

[0051] Determining whether it can be stored in a refrigerator

[0052] server:

[0053] The AI ​​model analyzes the purchased product information and determines whether it needs to be stored in a refrigerator. For example, it determines that milk and butter are products that need to be stored in a refrigerator.

[0054] Device:

[0055] The information about products that need to be stored in the refrigerator is registered in the refrigerator sub-application, which receives the information from the server. The registered information is also sent to the user's device.

[0056] Refrigerator management and recipe suggestions

[0057] Terminal (refrigerator):

[0058] The storage status of ingredients is detected using cameras and sensors inside the refrigerator, and that information is sent to a server.

[0059] server:

[0060] The system analyzes sensor and camera data received from the refrigerator and updates the information about ingredients in the refrigerator. It then generates recipes based on the updated information.

[0061] Terminal (user terminal):

[0062] The server receives the recipe suggestions and notifies the user. The user checks the recipe, takes out the necessary ingredients from the refrigerator, and starts cooking.

[0063] Collaboration with refrigerator manufacturers

[0064] server:

[0065] We provide refrigerator manufacturers with a common API to collect sensor and camera data from refrigerators, enabling accurate and real-time updates of food ingredients stored in refrigerators. We also monitor API usage and manage license revenue.

[0066] Terminal (refrigerator):

[0067] Information from inside the refrigerator is sent to the server via the API, and management information from the server is displayed inside the refrigerator.

[0068] Specific examples

[0069] Example of operation at time of purchase

[0070] 1. A user selects items (milk, butter, bread) at a supermarket and completes payment using an electronic payment app.

[0071] 2. The device sends the purchased product information to the server, which uses AI to determine whether the milk and butter need to be stored in the refrigerator.

[0072] 3. The registration information is sent to the refrigerator sub-app, and the user stores the purchased milk and butter in the refrigerator.

[0073] Example of refrigerator management and recipe suggestions

[0074] 1. The terminal (refrigerator) detects newly stored ingredients and sends the information to the server.

[0075] 2. The server updates the information about ingredients in the refrigerator and generates a recipe based on the ingredients you have.

[0076] 3. The generated recipe is sent to the user's device, and the user checks the recipe and cooks the ingredients.

[0077] This system allows users to automate the management of their purchases, reducing wasteful consumption and enabling effective meal planning. Furthermore, by collaborating with refrigerator manufacturers, it is possible to update information about ingredients in the refrigerator in real time.

[0078] The processing flow will be explained below.

[0079] Program processing steps

[0080] Receiving and determining purchase information

[0081] Step 1:

[0082] A user selects a product at a supermarket or online shop and completes the purchase process using an electronic payment app.

[0083] Step 2:

[0084] The device (smartphone, etc.) captures information about the purchased product (product name, category, quantity, etc.) and sends it to the server.

[0085] Step 3:

[0086] The server receives the purchased product information sent from the electronic payment application.

[0087] Step 4:

[0088] The server uses an AI model to analyze the received purchase information and determine whether it needs to be stored in a refrigerator.

[0089] For example, it determines that "milk" and "butter" need to be stored in the refrigerator, but "bread" does not.

[0090] Step 5:

[0091] The server generates a list of products that need to be stored in the refrigerator and issues an instruction to register the list in the refrigerator sub-app.

[0092] Step 6:

[0093] The terminal (smartphone, etc.) receives instructions from the server and registers the purchased items in the refrigerator sub-app.

[0094] After the terminal has completed registration of the purchased product information, it notifies the user.

[0095] Refrigerator management and recipe suggestions

[0096] Step 7:

[0097] The food purchased by the user is stored in the refrigerator.

[0098] Step 8:

[0099] The terminal (refrigerator) detects newly stored ingredients using its internal camera and sensors and sends this information to the server.

[0100] Step 9:

[0101] The server receives the data sent from the refrigerator sub-application and updates the information about ingredients in the refrigerator.

[0102] Step 10:

[0103] The server generates recipes based on the ingredients in the refrigerator.

[0104] For example, generate a recipe for "Omelette with butter and eggs."

[0105] Step 11:

[0106] The server transmits the generated recipe information to the user's terminal.

[0107] Step 12:

[0108] The terminal (smartphone, etc.) notifies the user of the recipe suggestions sent from the server.

[0109] Step 13:

[0110] The user checks the recipe, takes the necessary ingredients from the refrigerator, and starts cooking.

[0111] Collaboration with refrigerator manufacturers

[0112] Step 14:

[0113] The server provides a common API to refrigerator manufacturers.

[0114] Step 15:

[0115] The terminal (refrigerator) sends information from the interior camera and sensors to the server via API.

[0116] Step 16:

[0117] The server analyzes the data sent from the refrigerator and automatically updates the information about ingredients in the refrigerator.

[0118] Step 17:

[0119] The server monitors API usage and manages license revenue from refrigerator manufacturers.

[0120] This allows users to automate the management of their purchases and plan meals efficiently, while refrigerator manufacturers can also earn revenue from the use and integration of the system.

[0121] Example 1

[0122] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0123] In modern life, proper storage of purchased ingredients and their efficient use are important issues. In particular, there is a demand for a system that can automatically manage ingredient purchasing information, accurately update the information on ingredients in the refrigerator, and suggest recipes. However, with conventional technologies, it is difficult to efficiently link the management of purchased items with the updating of the information on ingredients in the refrigerator, which places a heavy burden on users.

[0124] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0125] In this invention, the server includes means for receiving purchased item information from the electronic payment app, means for determining using AI whether or not the purchased items need to be stored in the refrigerator, means for registering information on the purchased items determined to need to be stored in the refrigerator in the refrigerator sub-app, means for notifying the user terminal of the registered purchased item information, means for detecting food ingredient information using sensors and cameras inside the refrigerator and sending that information to the server, and means for updating the food ingredient information in the refrigerator based on the food ingredient information received by the server. This automates the management of purchased items and accurately updates the food ingredient information in the refrigerator in real time, enabling users to use ingredients efficiently and plan meals.

[0126] An "electronic payment app" is software that allows users to manage purchase information and make payments online or in physical stores.

[0127] "Purchased product information" is data including detailed information about products purchased by a user, such as product names, categories, and quantities.

[0128] An "AI model" is an artificial intelligence model that uses machine learning algorithms to analyze data and make predictions or decisions for specific tasks.

[0129] The "refrigerator sub-app" is an auxiliary application that manages information about ingredients in the refrigerator and notifies the user's device of this information.

[0130] A "user terminal" is an electronic device that can be directly operated by a user, such as a smartphone or tablet.

[0131] A "sensor" is a device that detects the state inside a refrigerator and transmits that data to the outside.

[0132] The "camera" is a photographic device for capturing image information of the inside of the refrigerator.

[0133] A "server" is a central system that provides data processing and storage functions and integrates and manages data from various devices.

[0134] A "recipe" is an instruction manual that lists the steps and ingredients needed for cooking based on the ingredients in the refrigerator.

[0135] A "database" is a system for efficiently storing and managing large amounts of data.

[0136] A "common API" is a common program interface that enables data exchange between different systems.

[0137] "License revenue" is revenue earned by providing third parties with the right to use specific technology or software.

[0138] The present invention provides a system that efficiently manages ingredients purchased by users, updates ingredient information in the refrigerator in real time, and suggests suitable recipes, allowing users to use purchased food without waste and plan meal preparations accordingly.

[0139] The main components of the system include:

[0140] 1. Electronic payment app: This is software that manages purchase information when a user purchases ingredients. When a user purchases ingredients at a supermarket or online shop and completes payment using an electronic payment app, purchase information is generated. Examples include SmartPay and PayPal.

[0141] 1. Server: Receives purchase information sent from the electronic payment app and uses an AI model (using, for example, TensorFlow or PyTorch) to determine whether the purchased items need to be stored in the refrigerator. The server also manages information about ingredients in the refrigerator and processes the data to provide users with appropriate recipes.

[0142] 1. Refrigerator sub-application: This application manages information about ingredients in the refrigerator and notifies the user. The sub-application receives information about products that need to be stored in the refrigerator from the server and notifies the user's device.

[0143] 1. Sensors and cameras: These are devices used to monitor the state inside the refrigerator and detect the presence of ingredients. Data from these devices is sent to the server and used to update the ingredient information. Examples include built-in image recognition cameras and temperature sensors.

[0144] 1. User terminal: An electronic device that can be directly operated by the user, such as a smartphone or tablet. This receives recipe suggestions sent from the server and notifies the user.

[0145] To illustrate, consider the following scenario:

[0146] 1. Receiving purchase information:

[0147] A user purchases food items such as milk, butter, and bread at a supermarket and completes the payment using an electronic payment app. The device then sends the purchased item information from the electronic payment app to the server.

[0148] 1. Refrigerator storage criteria:

[0149] The server uses an AI model to determine whether or not a product needs to be refrigerated based on the product information. For example, it may determine that milk and butter need to be refrigerated, while bread is recommended to be stored at room temperature.

[0150] 1. Register for the refrigerator sub-app:

[0151] The server sends the result of the judgment to the refrigerator sub-application, which notifies the user's device. The user then stores the milk and butter they bought in the refrigerator.

[0152] 1. Refrigerator management and recipe suggestions:

[0153] The device (refrigerator) uses a camera to recognize newly stored ingredients and sends the information to the server. The server updates the information about ingredients in the refrigerator and generates recipes based on the ingredients it has. For example, it generates a recipe for "cream stew" based on the latest list of ingredients in the refrigerator (milk, butter, eggs, cheese) and sends it to the user's device. The user's device notifies the user of the recipe, and the user begins cooking based on that information.

[0154] An example of a prompt to be input to the generative AI model is, "Please explain the processing flow of a system in which a user purchases milk, butter, and bread at the supermarket, determines the need to store them in the refrigerator based on that information, registers them in the refrigerator sub-app, and then manages the information about ingredients in the refrigerator and suggests recipes."

[0155] As a result, the present invention provides a system that is highly convenient for users, reduces food waste, and assists in efficient meal planning.

[0156] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0157] Processing Steps

[0158] Step 1: Obtain purchase information

[0159] User: Purchases ingredients at a supermarket or online shop and completes payment using an electronic payment app.

[0160] Input: Purchased product information (product name, category, quantity, etc.)

[0161] How it works: A user uses the SuperPay app to buy milk, butter, and bread.

[0162] Output: Purchase product information is generated from the electronic payment app and sent to the terminal.

[0163] Step 2: Submit your purchase information

[0164] Terminal: Sends purchased product information to the server.

[0165] Input: Purchase information obtained from the electronic payment app

[0166] Operation: The terminal sends purchased product information to the server.

[0167] Output: The server receives the purchased product information.

[0168] Step 3: Determine if refrigeration is necessary

[0169] Server: Uses AI models to determine whether purchased items need to be stored in the refrigerator.

[0170] Input: Purchase product information

[0171] How it works: The server uses TensorFlow to determine that milk and butter need to be stored in the refrigerator, but bread can be stored at room temperature.

[0172] Output: A list of products that need to be stored in the refrigerator and those that can be stored at room temperature

[0173] Step 4: Registering the refrigerator sub-app

[0174] Server: Sends information about products that need to be stored in the refrigerator to the refrigerator sub-app.

[0175] Input: List of products that need to be kept in the refrigerator

[0176] Operation: The server sends milk and butter information to the refrigerator sub-app, which then notifies the user's device.

[0177] Output: Product information registered in the refrigerator sub-app and notification to the user

[0178] Step 5: Get information about ingredients in the refrigerator

[0179] Terminal (refrigerator): Detects food ingredient information using sensors and cameras inside the refrigerator and sends that information to the server.

[0180] Input: Sensor data and camera data from inside the refrigerator

[0181] How it works: The refrigerator's camera recognizes newly placed ingredients (milk and butter), and the sensor captures temperature information.

[0182] Output: Sensor data and camera data are sent to the server.

[0183] Step 6: Update the ingredients in your refrigerator

[0184] Server: Updates the information about ingredients in the refrigerator based on the received sensor data and camera data.

[0185] Input: Sensor data and camera data

[0186] Operation: The server updates the ingredient information in the database to keep the ingredient list up to date.

[0187] Output: Updated ingredient information

[0188] Step 7: Recipe suggestions

[0189] Server: Generates recipes using AI based on updated ingredient information.

[0190] Input: Updated ingredient information

[0191] Operation: Generates a recipe for "cream stew" based on the ingredient information held by the server.

[0192] Output: The generated recipe

[0193] Step 8: Notify the user

[0194] Terminal (user terminal): Receives recipe suggestions sent from the server and notifies the user.

[0195] Input: Generated recipe

[0196] Operation: The user terminal notifies the user of the recipe.

[0197] Output: The user receives a recipe notification.

[0198] These steps allow users to automate their purchase management, update their refrigerator with ingredients, and suggest recipes, reducing food waste and enabling efficient meal planning.

[0199] (Application example 1)

[0200] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0201] Conventional refrigerator management systems require users to manually enter information about purchased products, which takes time and effort. They also lack a means to automatically determine whether purchased products need to be stored in the refrigerator and manage them appropriately. Furthermore, recipe suggestions based on refrigerator inventory information are not provided in real time, making it difficult to make effective use of ingredients.

[0202] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0203] In this invention, the server includes means for receiving purchased product information from the electronic payment app, means for determining using AI whether the purchased product needs to be stored in the refrigerator, means for registering information on the purchased product determined to need to be stored in the refrigerator in a refrigerator sub-app, means for notifying the user terminal of the registered purchased product information, and means for automatically reading the purchased product information at the physical store and means for updating inventory information in the refrigerator based on that information, managing ingredients, and suggesting recipes. This allows the user to automatically register information on purchased products in the refrigerator, allowing them to grasp inventory information in the refrigerator in real time and use ingredients efficiently.

[0204] An "electronic payment app" is an application that allows users to pay for goods electronically.

[0205] "Purchased product information" refers to information such as the name, category, and quantity of the product purchased by the user.

[0206] "AI" stands for artificial intelligence, a system that learns and reasons to solve specific problems.

[0207] The "refrigerator sub-app" is an application that manages refrigerator inventory information and provides the user with information on ingredients and recipe suggestions.

[0208] A "user terminal" is an information processing device operated by a user, such as a smartphone, tablet, or personal computer.

[0209] A "physical store" is a store where users can physically purchase products.

[0210] "Means for automatically reading information about purchased items" refers to technology that automatically obtains information about purchased items using sensors, cameras, etc.

[0211] "Inventory information" is information that indicates the current status of ingredients and products in the refrigerator.

[0212] "Recipe suggestion" refers to suggesting cooking methods to the user based on information about ingredients in the refrigerator.

[0213] "Sensor data" is digital data obtained from sensors regarding temperature, humidity, the presence of objects, etc.

[0214] "Camera data" refers to image and video data captured by a camera.

[0215] A "common API" is an application program interface that can be used commonly across multiple systems.

[0216] A "smart device" is an electronic device that can connect to the Internet and allows users to input and display information.

[0217] "Licensing revenue" is revenue earned by permitting the use of a certain technology or product.

[0218] This invention is a system that automatically acquires information about products purchased by a user, updates refrigerator inventory based on that information, and manages ingredients and suggests recipes. This system is configured using the following hardware and software.

[0219] Hardware

[0220] Smartphone: A device that allows users to purchase products and use electronic payment apps.

[0221] Smart glasses: devices that display real-time information while users are shopping in a physical store.

[0222] Refrigerator: A home appliance equipped with sensors and cameras inside that can obtain information about the food inside the refrigerator.

[0223] Server: A central processing unit that processes purchased product information and data from the refrigerator.

[0224] software

[0225] Electronic payment app: An application that allows users to purchase products and make payments.

[0226] Refrigerator sub-app: An application that manages the inventory information in the refrigerator and notifies the user.

[0227] Common API: An interface for receiving and analyzing sensor data and camera data from refrigerator manufacturers.

[0228] AI model: An artificial intelligence model that analyzes purchased product information and determines whether or not it needs to be stored in the refrigerator.

[0229] In this system, users first purchase products at a physical store and pay through an electronic payment app. Purchase information is automatically sent to a smartphone at the time of payment. This information is then sent to a server, where an AI model analyzes it and determines whether the product needs to be stored in a refrigerator.

[0230] Next, product information that needs to be stored in the refrigerator is registered in the refrigerator sub-app and simultaneously notified to the user's device. The user receives this notification and stores the necessary ingredients in the refrigerator.

[0231] The refrigerator uses internal sensors and cameras to detect newly stored ingredients. The detected data is sent to the server via a common API. The server uses this data to update the refrigerator's inventory and generate optimal recipes.

[0232] The generated recipe is sent to the user's device, and the user can start cooking based on the recipe. This allows the user to automatically register purchased product information in the refrigerator, keep track of inventory in the refrigerator in real time, and use ingredients efficiently.

[0233] Specific examples

[0234] For example, suppose a user purchases milk, butter, and bread at a physical store and completes the payment using an electronic payment app. At this time, the purchase information sent from the payment app is sent to a server, where an AI model analyzes it and determines that the milk and butter need to be stored in the refrigerator. This information is registered in the refrigerator sub-app, and the user checks the information on their smartphone. Next, sensors and cameras inside the refrigerator detect the newly placed ingredients and send the information to the server. The server analyzes the data and generates a recipe (e.g., salad, pasta, smoothie) based on the ingredients on hand, and notifies the user.

[0235] Prompt Sentence Examples

[0236] When a user purchases food at a physical store and completes the payment through a payment app on their smartphone, please explain the process by which the purchase information is registered in the refrigerator sub-app. Also, please explain how recipes are suggested based on the information about ingredients in the refrigerator.

[0237] In this way, the present invention greatly improves the user's daily life, allowing for efficient food management and cooking.

[0238] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0239] Step 1:

[0240] A user purchases an item in a physical store and pays using an electronic payment app.

[0241] Input: Information about the product purchased by the user (product name, category, quantity, etc.)

[0242] What happens: A user uses their smartphone to scan an item into a payment app and complete the payment.

[0243] Output: Purchase product information is generated and sent to your smartphone.

[0244] Step 2:

[0245] The smartphone sends the purchased product information to the server.

[0246] Input: Purchase information obtained from your smartphone

[0247] Specific operation: The smartphone obtains the purchased product information from the electronic payment app and sends it to the server as an HTTP request.

[0248] Output: Purchase information is sent to the server.

[0249] Step 3:

[0250] The server uses an AI model to determine whether refrigeration is necessary.

[0251] Input: Purchase information sent to the server

[0252] How it works: The server inputs purchased product information into the AI ​​model and determines whether or not the product needs to be stored in the refrigerator. The AI ​​model makes this determination based on the product name and category.

[0253] Output: A list of items that need to be kept in a refrigerator is generated.

[0254] Step 4:

[0255] The server registers product information that needs to be stored in the refrigerator in the refrigerator sub-app.

[0256] Input: List of products that require refrigeration as determined by the server

[0257] Specific operation: The server sends the registration information to the refrigerator sub-app via API.

[0258] Output: Product information that needs to be stored is registered in the refrigerator sub-app.

[0259] Step 5:

[0260] The refrigerator sub-app notifies the user device of the registration information

[0261] Input: Product information registered in the refrigerator sub-app

[0262] Specific operation: The refrigerator sub-app notifies the user device of the information received from the server. The information is sent to the user using push notifications, etc.

[0263] Output: The registration information for the purchased product is displayed on the user's device.

[0264] Step 6:

[0265] The user places the product in the refrigerator

[0266] Input: Registration information displayed on the user's device

[0267] Specific operation: The user places the purchased items in the refrigerator. The product placement is performed while confirming the registration information.

[0268] Output: The product is stored in the refrigerator.

[0269] Step 7:

[0270] The refrigerator detects the food stored inside using internal sensors and cameras.

[0271] Input: New item placed in refrigerator

[0272] How it works: The refrigerator uses internal sensors and cameras to detect newly added ingredients.

[0273] Output: The detected ingredients information is generated.

[0274] Step 8:

[0275] The refrigerator sends the food information it detects to the server.

[0276] Input: Detected ingredient information

[0277] Specific operation: The refrigerator sends ingredient information to the server using a common API.

[0278] Output: Ingredient information is sent to the server.

[0279] Step 9:

[0280] The server updates the ingredient information and generates the optimal recipe.

[0281] Input: Ingredient information sent to the server

[0282] Specific operation: The server updates the inventory data in the refrigerator based on the received ingredient information and generates the optimal recipe using a generative AI model.

[0283] Output: The optimal recipe information is generated.

[0284] Step 10:

[0285] The server sends the generated recipe to the user's device.

[0286] Input: Generated recipe information

[0287] Specific operation: The server sends the generated recipe information to the user's device. The recipe information is sent to the user using push notifications, etc.

[0288] Output: Recipe information is displayed on the user's device.

[0289] Through the above processing steps, the user can automatically register information about purchased items in the refrigerator, grasp inventory information in the refrigerator in real time, and use ingredients efficiently.

[0290] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0291] 1. System Overview

[0292] The system of this invention receives information about products purchased by users through an electronic payment app, and based on that information, AI determines whether the product needs to be stored in the refrigerator and automatically registers that information in the refrigerator sub-app. It also includes a function to manage information about ingredients in the refrigerator and suggest recipes to users based on that information. This system works in conjunction with refrigerator manufacturers, acquiring sensor and camera data through a common API to update information about ingredients in the refrigerator in real time. Furthermore, by combining it with an emotion engine, recipes are personalized based on the user's emotions, improving the user experience.

[0293] 2. System Configuration and Operation

[0294] Receiving purchase information

[0295] User:

[0296] When a user purchases ingredients at a supermarket or online shop and pays using an electronic payment app, purchased product information is generated.

[0297] Device:

[0298] Purchase information (product name, category, quantity, etc.) is sent from the electronic payment app to the server.

[0299] server:

[0300] The server receives the purchased product information and uses an AI model to determine whether it needs to be stored in a refrigerator.

[0301] Determining whether it can be stored in a refrigerator

[0302] server:

[0303] The AI ​​model analyzes the purchased product information and determines whether it needs to be stored in a refrigerator. For example, milk and butter are determined to be products that need to be stored in a refrigerator.

[0304] Device:

[0305] The information about products that need to be stored in the refrigerator is registered in the refrigerator sub-application, which receives the information from the server. The registered information is also sent to the user's device.

[0306] Refrigerator management and recipe suggestions

[0307] Terminal (refrigerator):

[0308] The storage status of ingredients is detected using cameras and sensors inside the refrigerator, and that information is sent to a server.

[0309] server:

[0310] The system analyzes sensor and camera data received from the refrigerator and updates the information about ingredients in the refrigerator. It then generates recipes based on the updated information.

[0311] Terminal (user terminal):

[0312] The server receives the recipe suggestions and notifies the user. The user checks the recipe, takes out the necessary ingredients from the refrigerator, and starts cooking.

[0313] Collaboration with refrigerator manufacturers

[0314] server:

[0315] We provide refrigerator manufacturers with a common API to collect sensor and camera data from refrigerators, enabling accurate and real-time updates of food ingredients stored in refrigerators. We also monitor API usage and manage license revenue.

[0316] Terminal (refrigerator):

[0317] Information from inside the refrigerator is sent to the server via the API, and management information from the server is displayed inside the refrigerator.

[0318] Incorporating an emotion engine

[0319] User Emotion Recognition and Recipe Personalization

[0320] Device (smartphone, etc.):

[0321] It incorporates an emotion engine that analyzes emotions through facial recognition and voice data of the user. For example, it uses a camera and microphone to analyze facial expressions and voices while the user is operating the device.

[0322] server:

[0323] It receives emotional data sent from the emotion engine and personalizes recipes based on that data, for example, suggesting easy-to-make recipes if the user is tired.

[0324] Terminal (user terminal):

[0325] It notifies users of personalized recipes and makes meal suggestions based on the user's emotions.

[0326] Emotion-based food consumption and management

[0327] server:

[0328] The system continuously monitors the user's emotional data and suggests food consumption and management based on their emotions. For example, if the user is feeling stressed, it will suggest recipes with ingredients that are effective in reducing stress.

[0329] Device (smartphone, etc.):

[0330] It notifies users in a timely manner and provides alerts about food expiration dates and conditions based on emotional data.

[0331] Specific examples

[0332] Example of operation at time of purchase

[0333] 1. A user selects items (milk, butter, bread) at a supermarket and completes the purchase using an electronic payment app.

[0334] 2. The device sends the purchased product information to the server, which uses AI to determine whether the milk and butter need to be stored in the refrigerator.

[0335] 3. The registration information is sent to the refrigerator sub-app, and the user stores the purchased milk and butter in the refrigerator.

[0336] Example of refrigerator management and recipe suggestions

[0337] 1. The terminal (refrigerator) detects newly stored ingredients and sends the information to the server.

[0338] 2. The server updates the information about ingredients in the refrigerator and generates a recipe based on the ingredients you have.

[0339] 3. The generated recipe is sent to the user's device, and the user checks the recipe and cooks the ingredients.

[0340] Example of Emotion Engine in Action

[0341] 1. The device (smartphone, etc.) collects the user's facial recognition and voice data and analyzes it using an emotion engine.

[0342] 2. Based on the emotional data, the server suggests recipes using ingredients that have a relaxing effect, for example, if the user feels like relaxing.

[0343] 3. The user checks the appropriate recipe and uses it to cook.

[0344] This system allows users to automate their purchase management, reducing wasteful consumption and creating effective meal plans. Furthermore, the incorporation of an emotion engine enables personalized meal suggestions that are in line with the user's emotions, which is expected to improve the user experience.

[0345] The processing flow will be explained below.

[0346] Program processing steps

[0347] Receiving and determining purchase information

[0348] Step 1:

[0349] A user selects a product at a supermarket or online shop and completes the purchase process using an electronic payment app.

[0350] Step 2:

[0351] The device (smartphone, etc.) captures information about the purchased product (product name, category, quantity, etc.) and sends it to the server.

[0352] Step 3:

[0353] The server receives the purchased product information sent from the electronic payment application.

[0354] Step 4:

[0355] The server uses an AI model to analyze the received purchase information and determine whether it needs to be stored in a refrigerator.

[0356] For example, it determines that "milk" and "butter" need to be stored in the refrigerator, but "bread" does not.

[0357] Step 5:

[0358] The server generates a list of products that need to be stored in the refrigerator and issues an instruction to register the list in the refrigerator sub-app.

[0359] Step 6:

[0360] The terminal (smartphone, etc.) receives instructions from the server and registers the purchased items in the refrigerator sub-app.

[0361] After the terminal has completed registration of the purchased product information, it notifies the user.

[0362] Refrigerator management and recipe suggestions

[0363] Step 7:

[0364] The food purchased by the user is stored in the refrigerator.

[0365] Step 8:

[0366] The terminal (refrigerator) detects newly stored ingredients using its internal camera and sensors and sends this information to the server.

[0367] Step 9:

[0368] The server receives the data sent from the refrigerator sub-application and updates the information about ingredients in the refrigerator.

[0369] Step 10:

[0370] The server generates recipes based on the ingredients in the refrigerator.

[0371] For example, generate a recipe for "Omelette with butter and eggs."

[0372] Step 11:

[0373] The server transmits the generated recipe information to the user's terminal.

[0374] Step 12:

[0375] The terminal (smartphone, etc.) notifies the user of the recipe suggestions sent from the server.

[0376] Step 13:

[0377] The user checks the recipe, takes the necessary ingredients from the refrigerator, and starts cooking.

[0378] Collaboration with refrigerator manufacturers

[0379] Step 14:

[0380] The server provides a common API to refrigerator manufacturers.

[0381] Step 15:

[0382] The terminal (refrigerator) sends information from the interior camera and sensors to the server via API.

[0383] Step 16:

[0384] The server analyzes the data sent from the refrigerator and automatically updates the information about ingredients in the refrigerator.

[0385] Step 17:

[0386] The server monitors API usage and manages license revenue from refrigerator manufacturers.

[0387] Incorporating an emotion engine

[0388] Step 18:

[0389] The device (smartphone, etc.) collects the user's facial recognition and voice data and analyzes it using an emotion engine.

[0390] Step 19:

[0391] The server uses emotional data to suggest easy-to-make recipes, for example, if the user feels tired.

[0392] Step 20:

[0393] The device (smartphone, etc.) notifies the user of the personalized recipe.

[0394] Step 21:

[0395] The user checks the recipe based on their emotions and uses the recipe to cook.

[0396] Step 22:

[0397] The server continuously monitors emotional data and suggests recipes using ingredients that are effective in reducing stress.

[0398] Step 23:

[0399] The device (smartphone, etc.) notifies the user in a timely manner and provides alerts regarding the expiration date and condition of ingredients based on emotional data.

[0400] This allows users to automate the management of their purchases and create efficient meal plans, and the incorporation of an emotion engine makes it possible to make personalized meal suggestions that are in line with the user's emotions, which is expected to improve the user experience.

[0401] Example 2

[0402] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0403] Currently, there are many grocery management systems and recipe recommendation systems on the market, but these systems have limitations in terms of centralized management and personalized recommendations. In particular, in terms of refrigerator ingredient management and recipe recommendations based on the user's mood, existing systems lack flexibility and do not sufficiently improve the user experience. Furthermore, they lack automated mechanisms for reducing food waste and efficiently planning meals. This invention aims to solve these problems and improve the quality of life of users.

[0404] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0405] In this invention, the server includes: means for receiving purchased product information from an electronic payment app; means for determining whether a purchased product requires refrigeration using AI; and means for registering information about purchased products determined to require refrigeration in a refrigeration appliance app. This allows for efficient management of purchased product information and enables appropriate storage based on the information. The server also includes means for updating food product information stored in the refrigeration appliance based on data from the refrigeration appliance app; means for generating recipes based on the food product information stored in the refrigeration appliance; and means for transmitting the generated recipes to a user terminal. This allows for real-time updates of food product information stored in the refrigeration appliance, and for suggesting recipes suitable for the user based on the updated information. The server also includes means for providing a common API to refrigeration appliance manufacturers; means for receiving sensor data and image data from the refrigeration appliance; means for analyzing the data and updating the food product information stored in the refrigeration appliance; and means for monitoring the usage of the common API and managing license revenue. This allows for efficient collection and management of refrigeration appliance data, enabling appropriate food product management and revenue generation. The server also includes means for recognizing user emotions; means for personalizing recipes based on the emotion data; and means for notifying the user terminal of the personalized recipes. This allows for personalized meal suggestions based on the user's emotions, further improving the user experience.

[0406] 1. An "electronic payment app" is a software application that a user uses to purchase goods online or in-store, and that digitizes the payment process.

[0407] 2. "Purchased Product Information" refers to detailed data of the product purchased by the user, including product name, category, quantity, price, etc.

[0408] 3. "AI" stands for artificial intelligence, which refers to algorithms and models that analyze data and perform specific tasks automatically.

[0409] 4. "Refrigeration equipment app" is software that supports the operation of refrigeration equipment, managing food ingredient information and monitoring storage conditions.

[0410] 5. "User terminal" refers to a device used by a user, such as a smartphone, tablet, PC, or other electronic device.

[0411] 6. "Sensor Data" means information detected by sensors, such as temperature, humidity, and location within refrigeration equipment.

[0412] 7. "Image data" refers to image files taken by a camera placed inside the refrigeration equipment, which are used to visually record the condition and placement of ingredients.

[0413] 8. "Common API" means a common application program interface that can be used by multiple refrigeration equipment manufacturers, enabling data transmission and reception and the sharing of functions.

[0414] 9. "License Revenue" means revenue received from refrigeration equipment manufacturers for use of the provided common API, including API usage fees and service fees.

[0415] 10. "Emotional Data" means information about a user's emotional state obtained through facial recognition or voice analysis, and is data indicating joy, sadness, fatigue, stress, etc.

[0416] 11. "Personalized Cooking Recipes" are cooking recipes that are specifically tailored to a user's individual emotional state and preferences and that are provided as part of meal suggestions to the user.

[0417] 1. System Overview

[0418] The system of this invention receives information about products purchased by users through an electronic payment app, and based on that information, AI determines the need for refrigeration and automatically registers that information in the refrigeration appliance app. It also includes a function to manage food information stored in the refrigeration appliance and suggest recipes to users based on that information. This system works in conjunction with refrigeration appliance manufacturers, acquiring sensor data and image data through a common API to update the food information stored in the refrigeration appliance in real time. Furthermore, by incorporating an emotion engine, recipes are personalized based on the user's emotions, improving the user experience.

[0419] 2. System Configuration and Operation

[0420] Receiving purchase information

[0421] When a user purchases groceries at a supermarket or online shop and pays using an electronic payment app, purchased product information (product name, category, quantity, etc.) is generated within the electronic payment app.

[0422] The terminal (electronic payment app) sends the purchased product information to the server. For example, it is sent to the cloud server in the form of an electronic receipt.

[0423] The server receives the purchased product information and records it in a database.

[0424] Determining whether refrigerated storage is possible

[0425] The server uses AI models to determine the need for refrigeration based on the purchased product information received, for example, milk and perishables.

[0426] The device registers the information about the products that need to be stored sent from the server in the refrigeration equipment app and notifies the user's device, allowing the user to store their purchased items appropriately.

[0427] Refrigeration equipment management and recipe suggestions

[0428] The terminal (refrigeration equipment) uses internal sensors and cameras to monitor the condition of the food and sends the data to a server.

[0429] The server analyzes the received sensor data and image data and updates the information in the refrigeration equipment, for example, identifying the type and quantity of ingredients.

[0430] The server generates usable recipes based on updated ingredient information, taking into account nutritional value and user preferences.

[0431] The terminal (user terminal) receives the recipe suggestions sent from the server and notifies the user, who can then check the recipes and prepare the suggested meals.

[0432] Collaboration with refrigeration equipment manufacturers

[0433] The server provides a common API to refrigeration equipment manufacturers and acquires sensor data and image data, for example, to manage fresh vegetables and dairy products.

[0434] The terminal (refrigeration equipment) sends data to the server via API, which updates the information in the refrigeration equipment in real time, allowing users to manage their ingredients appropriately.

[0435] Specific examples

[0436] Example of operation at time of purchase

[0437] 1. A user selects milk, butter, and bread at a supermarket and completes the purchase using an electronic payment app.

[0438] 2. The terminal (electronic payment app) sends the purchased product information to the server.

[0439] 3. The server uses AI to determine that the milk and butter need to be refrigerated.

[0440] 4. The registration information is sent to the refrigerator app, and the user places the milk and butter in the refrigerator.

[0441] Example of refrigeration equipment management and recipe suggestions

[0442] 1. The terminal (refrigeration equipment) detects newly stored food items and sends the information to the server.

[0443] 2. The server updates the food information in the refrigerator and generates a recipe based on the ingredients it has.

[0444] 3. The generated recipe is sent to the user's device, and the user checks the recipe and cooks the ingredients.

[0445] Example of Emotion Engine in Action

[0446] 1. The device (smartphone, etc.) collects the user's facial recognition and voice data and analyzes it using an emotion engine.

[0447] 2. Based on the emotional data, the server suggests recipes using ingredients that have a relaxing effect if the user feels like relaxing.

[0448] 3. The user checks the appropriate recipe and uses it to cook.

[0449] Example prompts for generative AI models

[0450] Receiving purchase information

[0451] Prompt: Receive information about the milk, butter, and bread purchased by a user at the supermarket from an electronic payment app, and use AI to determine whether they need to be stored in the refrigerator.

[0452] Refrigeration equipment management and recipe suggestions

[0453] Prompt: Send the newly stored ingredients to the server, update the ingredients in the refrigerator, and suggest a recipe.

[0454] Emotion Engine Operation

[0455] Prompt: Collect facial and voice data from the user and suggest recipes suitable for when the user wants to relax.

[0456] By using this system, users can automate the management of their purchases, reduce wasteful consumption, and create effective meal plans. In addition, the incorporation of an emotion engine enables personalized meal suggestions that are in line with the user's emotions, which is expected to improve the user experience.

[0457] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0458] Step 1:

[0459] A user purchases groceries at a supermarket or online shop and pays using an electronic payment app. At this time, product information (product name, category, quantity, etc.) is generated within the electronic payment app. The user inputs their intention to purchase a product, and as a result, product information is generated.

[0460] Step 2:

[0461] The terminal (electronic payment app) sends purchased product information to the server. For example, the purchase information is sent to the cloud server in the form of an electronic receipt. The terminal receives purchased product information as input and sends it to the server as output.

[0462] Step 3:

[0463] The server receives the purchased product information and records it in the database. The server analyzes the purchased product information received as input and saves it in the database. This records the purchased product information.

[0464] Step 4:

[0465] The server uses an AI model to determine the need for refrigerated storage based on the received purchase information. Purchase information is provided to the AI ​​model as input, and the AI ​​analyzes the data to determine the need for refrigerated storage. For example, it may determine that milk or fresh food needs to be stored in a refrigerator.

[0466] Step 5:

[0467] The terminal registers the information about the products that require storage sent from the server in the refrigeration equipment app and notifies the user terminal.The terminal receives information about the products that require refrigeration storage from the server as input, processes it as output to be registered in the refrigeration equipment app, and notifies the user terminal.

[0468] Step 6:

[0469] The terminal (refrigeration equipment) monitors the condition of food using internal sensors and cameras and sends the data to a server. The refrigeration equipment's sensors and cameras acquire food condition data as input and send it to the server as output. For example, sensor data on temperature and humidity, and image data of food.

[0470] Step 7:

[0471] The server analyzes the received sensor data and image data and updates the food ingredient information in the refrigerator. The server receives sensor data and image data as input, analyzes them to identify the type and amount of food ingredients, and updates the food ingredient information in the refrigerator as output.

[0472] Step 8:

[0473] The server generates usable recipes based on updated ingredient information, taking into account nutritional value and user preferences. Using the latest ingredient information as input, the AI ​​model analyzes the data and generates suitable recipes as output.

[0474] Step 9:

[0475] The terminal (user terminal) receives the recipe suggestions sent from the server and notifies the user. It receives the recipe information provided by the server as input and suggests it to the user using the notification function as output. The user checks this recipe and prepares the suggested meal.

[0476] Step 10:

[0477] The device (such as a smartphone) collects the user's facial recognition and voice data and analyzes it using an emotion engine. The user's facial recognition and voice data are taken as input, and the output is analyzed by the emotion engine. For example, data collection using a camera or microphone.

[0478] Step 11:

[0479] The server analyzes the emotional data and suggests recipes suitable for when the user wants to relax or when they are tired. The server receives the emotional data as input, analyzes it with an AI model, and generates suitable recipes as output.

[0480] Step 12:

[0481] The user checks the personalized recipe notified to the device and uses that recipe to cook. The notified recipe information is received as input, and cooking is performed based on that recipe as output. The user's cooking action is the final execution stage.

[0482] (Application example 2)

[0483] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0484] In conventional refrigerator and purchasing information management systems, storing and managing purchased products is often done manually, which is cumbersome for users and makes it difficult to effectively manage ingredients. Furthermore, there is a lack of personalized recipe suggestions based on the user's emotions and condition, which hinders the improvement of the user experience. There is a need for an efficient and user-friendly system that can solve these problems.

[0485] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0486] In this invention, the server includes means for receiving purchased product information from the electronic payment app, means for determining using AI whether the purchased product needs to be stored in the refrigerator, means for registering information on the purchased product determined to need to be stored in the refrigerator in a refrigerator sub-app, means for notifying the user terminal of the registered purchased product information, means for automatically acquiring purchase information from a physical store and linking with the refrigerator, means for personalizing recipes based on the user's emotions using an emotion engine, and means for analyzing the user's emotion data and suggesting appropriate ingredient consumption and recipes. This automates the storage and management of purchased products and makes it possible to realize personalized ingredient consumption and recipe suggestions based on the user's emotions.

[0487] An "electronic payment app" is a software application that receives purchase information electronically and completes payments.

[0488] "Purchased product information" is data including detailed information such as the name, category, and quantity of the product purchased by the user.

[0489] "AI" refers to artificial intelligence systems that automate specific tasks, analyze data, and make decisions.

[0490] The "refrigerator sub-app" is an auxiliary application for managing and updating information about ingredients in the refrigerator.

[0491] A "user terminal" is an electronic device operated by a user, such as a smartphone or tablet.

[0492] A "physical store" is a physical sales location where products are actually displayed and sold.

[0493] An "emotion engine" is software that analyzes a user's emotions and makes appropriate suggestions based on their state.

[0494] A "recipe" is a document that describes the steps and ingredients for making a particular dish.

[0495] A "common API" is a standardized interface that allows different systems to communicate with each other and exchange data.

[0496] "Sensor data" is information obtained from sensors that detect the temperature, humidity, and presence of objects inside the refrigerator.

[0497] "Camera data" refers to data acquired as video or images of the inside of a refrigerator.

[0498] A "generative AI model" is an artificial intelligence model designed to generate specific outputs based on user input.

[0499] A "prompt sentence" is an input sentence that a generative AI model uses to generate appropriate output.

[0500] 1. System Overview

[0501] This invention is a system that receives information about products purchased by users through an electronic payment app, and then uses AI to determine whether or not to store the products in the refrigerator, automatically registering them in a refrigerator sub-app. It also includes a function to manage information about ingredients in the refrigerator and suggest recipes based on that information, allowing recipes to be personalized based on the user's emotions.

[0502] 2. Program Generation

[0503] The program to realize this system is as follows, but the details of the code are not included. The specific main functions are as follows:

[0504] Receiving purchase information: The user sends product information from the electronic payment app.

[0505] Refrigerated storage decision: AI analyzes product information and determines the need for refrigerated storage.

[0506] Information registration: Register the necessary product information in the refrigerator sub-app.

[0507] User notification: Registration information is sent to the user's device.

[0508] Sentiment Analysis: The emotion engine analyzes user emotions and personalizes recipes.

[0509] Recipe suggestion: Generates recipes based on emotion data and notifies the user.

[0510] 3. Processing Description

[0511] Receiving purchase information

[0512] When a user purchases a product through an electronic payment app, purchased product information (product name, category, quantity, etc.) is generated. The server receives and analyzes this information.

[0513] Refrigerator storage judgment

[0514] The server uses an AI model to analyze the received purchase information and determine which products need to be stored in the refrigerator. The necessary product information is then registered in the refrigerator sub-app.

[0515] Physical store integration and continuous processing

[0516] Information about products purchased at physical stores is automatically sent to a server via the electronic payment app, which links purchase information with information in the refrigerator, reducing the burden on users.

[0517] Refrigerator management

[0518] Sensor and camera data from inside the refrigerator is sent to the server via a common API, which then analyzes the data and updates the information about ingredients in the refrigerator in real time.

[0519] Emotional engine and recipe suggestions

[0520] The emotion engine analyzes the user's facial expressions and voice to generate emotion data. Based on this, the server suggests recipes appropriate for the user's condition. For example, if the user is tired, it will suggest easy-to-make recipes.

[0521] 4. Usage example

[0522] Recipe suggestions based on refrigerator status data and user emotion data

[0523] Current refrigerator data:

[0524] Milk: 1 bottle

[0525] Bread: 2 pieces

[0526] User sentiment data:

[0527] Facial Expression: Tired

[0528] Audio Tone: Low

[0529] Prompt Sentence Examples

[0530] The system suggests suitable recipes based on the current state of the refrigerator and the user's emotional data.

[0531] For example, generate a recipe with the following conditions:

[0532] Refrigerator Data:

[0533] Milk: 1 bottle

[0534] Bread: 2 pieces

[0535] User sentiment data:

[0536] Facial Expression: Tired

[0537] Audio Tone: Low

[0538] Generated recipe example:

[0539] 1. Easy pasta recipes

[0540] 2. Relaxing Salad Recipes

[0541] As described above, the system receives information about purchased items and provides consistent support, from managing items in the refrigerator to suggesting recipes based on the user's emotions. This reduces the burden on the user and provides a personalized experience.

[0542] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0543] Step 1:

[0544] When a user purchases a product through an electronic payment app, purchased product information (product name, category, quantity, etc.) is generated. The user purchases ingredients at a supermarket or online shop and completes the payment through the electronic payment app. At this time, purchased product information is automatically generated.

[0545] Input: Purchased product information (product name, category, quantity, etc.)

[0546] Output: Purchase product information data

[0547] Step 2:

[0548] The terminal sends the purchased product information to the server. The purchased product information is sent from the electronic payment app to the server.

[0549] Input: Purchase product information data

[0550] Output: Purchased product information sent to the server

[0551] Step 3:

[0552] The server uses an AI model to analyze the received purchase information and determine whether or not the product needs to be stored in a refrigerator. For example, it identifies products that need to be refrigerated, such as milk and butter.

[0553] Input: Purchase information sent to the server

[0554] Processing: AI model determines whether refrigeration is necessary

[0555] Output: List of items that need to be stored

[0556] Step 4:

[0557] The server registers product information that needs to be stored in the refrigerator sub-app, which then automatically records these products.

[0558] Input: List of items that need to be stored

[0559] Output: Product information registered in the refrigerator sub-app

[0560] Step 5:

[0561] The server notifies the user of the registration information on their smartphone or other device, allowing them to check the products they have purchased.

[0562] Input: Product information registered in the refrigerator sub-app

[0563] Output: Notification to user terminal

[0564] Step 6:

[0565] The emotion engine is used to analyze the user's emotions. Facial expressions and voice data are acquired from the user's smartphone or tablet, and the emotion engine analyzes them.

[0566] Input: User facial expressions and voice data

[0567] Processing: Sentiment analysis using the emotion engine

[0568] Output: User emotion data

[0569] Step 7:

[0570] The server personalizes recipes based on the user's emotional data: if the user is tired, it suggests simple recipes, and if the user is relaxed, it suggests more elaborate recipes.

[0571] Input: User emotion data, information about ingredients in the refrigerator

[0572] Processing: Recipe generation based on sentiment data

[0573] Output: personalized recipe

[0574] Step 8:

[0575] The server sends personalized recipes to the user's device, which then notifies the user of the recipes on their smartphone or other device, allowing the user to check the appropriate recipes.

[0576] Input: Personalized Recipe

[0577] Output: Recipe notification to user's device

[0578] Examples of specific examples and prompts

[0579] Usage example

[0580] The system suggests suitable recipes based on the current state of the refrigerator and the user's emotional data.

[0581] For example, generate a recipe with the following conditions:

[0582] Refrigerator Data:

[0583] Milk: 1 bottle

[0584] Bread: 2 pieces

[0585] User sentiment data:

[0586] Facial Expression: Tired

[0587] Audio Tone: Low

[0588] Generated recipe example:

[0589] 1. Easy pasta recipes

[0590] 2. Relaxing Salad Recipes

[0591] The above is a specific embodiment of the invention, which allows users to store and manage their purchases automatically and provides a personalized experience based on emotions.

[0592] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0593] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0594] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0595] [Second embodiment]

[0596] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0597] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0598] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0599] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0600] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0601] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0602] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0603] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0604] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0605] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0606] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0607] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0608] 1. System Overview

[0609] The system of this invention receives information about products purchased by users through an electronic payment app, and based on that information, AI determines whether the product needs to be stored in the refrigerator and automatically registers that information in the refrigerator sub-app. It also includes a function to manage information about ingredients in the refrigerator and suggest recipes to users based on that information. This system works in conjunction with refrigerator manufacturers, acquiring sensor and camera data through a common API, allowing it to update information about ingredients in the refrigerator in real time.

[0610] 2. System Configuration and Operation

[0611] Receiving purchase information

[0612] User:

[0613] When a user purchases ingredients at a supermarket or online shop and pays using an electronic payment app, purchased product information is generated.

[0614] Device:

[0615] Purchase information (product name, category, quantity, etc.) is sent from the electronic payment app to the server.

[0616] server:

[0617] The server receives the purchased product information and uses an AI model to determine whether it needs to be stored in a refrigerator.

[0618] Determining whether it can be stored in a refrigerator

[0619] server:

[0620] The AI ​​model analyzes the purchased product information and determines whether it needs to be stored in a refrigerator. For example, it determines that milk and butter are products that need to be stored in a refrigerator.

[0621] Device:

[0622] The information about products that need to be stored in the refrigerator is registered in the refrigerator sub-application, which receives the information from the server. The registered information is also sent to the user's device.

[0623] Refrigerator management and recipe suggestions

[0624] Terminal (refrigerator):

[0625] The storage status of ingredients is detected using cameras and sensors inside the refrigerator, and that information is sent to a server.

[0626] server:

[0627] The system analyzes sensor and camera data received from the refrigerator and updates the information about ingredients in the refrigerator. It then generates recipes based on the updated information.

[0628] Terminal (user terminal):

[0629] The server receives the recipe suggestions and notifies the user. The user checks the recipe, takes out the necessary ingredients from the refrigerator, and starts cooking.

[0630] Collaboration with refrigerator manufacturers

[0631] server:

[0632] We provide refrigerator manufacturers with a common API to collect sensor and camera data from refrigerators, enabling accurate and real-time updates of food ingredients stored in refrigerators. We also monitor API usage and manage license revenue.

[0633] Terminal (refrigerator):

[0634] Information from inside the refrigerator is sent to the server via the API, and management information from the server is displayed inside the refrigerator.

[0635] Specific examples

[0636] Example of operation at time of purchase

[0637] 1. A user selects items (milk, butter, bread) at a supermarket and completes payment using an electronic payment app.

[0638] 2. The device sends the purchased product information to the server, which uses AI to determine whether the milk and butter need to be stored in the refrigerator.

[0639] 3. The registration information is sent to the refrigerator sub-app, and the user stores the purchased milk and butter in the refrigerator.

[0640] Example of refrigerator management and recipe suggestions

[0641] 1. The terminal (refrigerator) detects newly stored ingredients and sends the information to the server.

[0642] 2. The server updates the information about ingredients in the refrigerator and generates a recipe based on the ingredients you have.

[0643] 3. The generated recipe is sent to the user's device, and the user checks the recipe and cooks the ingredients.

[0644] This system allows users to automate the management of their purchases, reducing wasteful consumption and enabling effective meal planning. Furthermore, by collaborating with refrigerator manufacturers, it is possible to update information about ingredients in the refrigerator in real time.

[0645] The processing flow will be explained below.

[0646] Program processing steps

[0647] Receiving and determining purchase information

[0648] Step 1:

[0649] A user selects a product at a supermarket or online shop and completes the purchase process using an electronic payment app.

[0650] Step 2:

[0651] The device (smartphone, etc.) captures information about the purchased product (product name, category, quantity, etc.) and sends it to the server.

[0652] Step 3:

[0653] The server receives the purchased product information sent from the electronic payment application.

[0654] Step 4:

[0655] The server uses an AI model to analyze the received purchase information and determine whether it needs to be stored in a refrigerator.

[0656] For example, it determines that "milk" and "butter" need to be stored in the refrigerator, but "bread" does not.

[0657] Step 5:

[0658] The server generates a list of products that need to be stored in the refrigerator and issues an instruction to register the list in the refrigerator sub-app.

[0659] Step 6:

[0660] The terminal (smartphone, etc.) receives instructions from the server and registers the purchased items in the refrigerator sub-app.

[0661] After the terminal has completed registration of the purchased product information, it notifies the user.

[0662] Refrigerator management and recipe suggestions

[0663] Step 7:

[0664] The food purchased by the user is stored in the refrigerator.

[0665] Step 8:

[0666] The terminal (refrigerator) detects newly stored ingredients using its internal camera and sensors and sends this information to the server.

[0667] Step 9:

[0668] The server receives the data sent from the refrigerator sub-application and updates the information about ingredients in the refrigerator.

[0669] Step 10:

[0670] The server generates recipes based on the ingredients in the refrigerator.

[0671] For example, generate a recipe for "Omelette with butter and eggs."

[0672] Step 11:

[0673] The server transmits the generated recipe information to the user's terminal.

[0674] Step 12:

[0675] The terminal (smartphone, etc.) notifies the user of the recipe suggestions sent from the server.

[0676] Step 13:

[0677] The user checks the recipe, takes the necessary ingredients from the refrigerator, and starts cooking.

[0678] Collaboration with refrigerator manufacturers

[0679] Step 14:

[0680] The server provides a common API to refrigerator manufacturers.

[0681] Step 15:

[0682] The terminal (refrigerator) sends information from the interior camera and sensors to the server via API.

[0683] Step 16:

[0684] The server analyzes the data sent from the refrigerator and automatically updates the information about ingredients in the refrigerator.

[0685] Step 17:

[0686] The server monitors API usage and manages license revenue from refrigerator manufacturers.

[0687] This allows users to automate the management of their purchases and plan meals efficiently, while refrigerator manufacturers can also earn revenue from the use and integration of the system.

[0688] Example 1

[0689] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0690] In modern life, proper storage of purchased ingredients and their efficient use are important issues. In particular, there is a demand for a system that can automatically manage ingredient purchasing information, accurately update the information on ingredients in the refrigerator, and suggest recipes. However, with conventional technologies, it is difficult to efficiently link the management of purchased items with the updating of the information on ingredients in the refrigerator, which places a heavy burden on users.

[0691] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0692] In this invention, the server includes means for receiving purchased item information from the electronic payment app, means for determining using AI whether or not the purchased items need to be stored in the refrigerator, means for registering information on the purchased items determined to need to be stored in the refrigerator in the refrigerator sub-app, means for notifying the user terminal of the registered purchased item information, means for detecting food ingredient information using sensors and cameras inside the refrigerator and sending that information to the server, and means for updating the food ingredient information in the refrigerator based on the food ingredient information received by the server. This automates the management of purchased items and accurately updates the food ingredient information in the refrigerator in real time, enabling users to use ingredients efficiently and plan meals.

[0693] An "electronic payment app" is software that allows users to manage purchase information and make payments online or in physical stores.

[0694] "Purchased product information" is data including detailed information about products purchased by a user, such as product names, categories, and quantities.

[0695] An "AI model" is an artificial intelligence model that uses machine learning algorithms to analyze data and make predictions or decisions for specific tasks.

[0696] The "refrigerator sub-app" is an auxiliary application that manages information about ingredients in the refrigerator and notifies the user's device of this information.

[0697] A "user terminal" is an electronic device that can be directly operated by a user, such as a smartphone or tablet.

[0698] A "sensor" is a device that detects the state inside a refrigerator and transmits that data to the outside.

[0699] The "camera" is a photographic device for capturing image information of the inside of the refrigerator.

[0700] A "server" is a central system that provides data processing and storage functions and integrates and manages data from various devices.

[0701] A "recipe" is an instruction manual that lists the steps and ingredients needed for cooking based on the ingredients in the refrigerator.

[0702] A "database" is a system for efficiently storing and managing large amounts of data.

[0703] A "common API" is a common program interface that enables data exchange between different systems.

[0704] "License revenue" is revenue earned by providing third parties with the right to use specific technology or software.

[0705] The present invention provides a system that efficiently manages ingredients purchased by users, updates ingredient information in the refrigerator in real time, and suggests suitable recipes, allowing users to use purchased food without waste and plan meal preparations accordingly.

[0706] The main components of the system include:

[0707] 1. Electronic payment app: This is software that manages purchase information when a user purchases ingredients. When a user purchases ingredients at a supermarket or online shop and completes payment using an electronic payment app, purchase information is generated. Examples include SmartPay and PayPal.

[0708] 1. Server: Receives purchase information sent from the electronic payment app and uses an AI model (using, for example, TensorFlow or PyTorch) to determine whether the purchased items need to be stored in the refrigerator. The server also manages information about ingredients in the refrigerator and processes the data to provide users with appropriate recipes.

[0709] 1. Refrigerator sub-application: This application manages information about ingredients in the refrigerator and notifies the user. The sub-application receives information about products that need to be stored in the refrigerator from the server and notifies the user's device.

[0710] 1. Sensors and cameras: These are devices used to monitor the state inside the refrigerator and detect the presence of ingredients. Data from these devices is sent to the server and used to update the ingredient information. Examples include built-in image recognition cameras and temperature sensors.

[0711] 1. User terminal: An electronic device that can be directly operated by the user, such as a smartphone or tablet. This receives recipe suggestions sent from the server and notifies the user.

[0712] To illustrate, consider the following scenario:

[0713] 1. Receiving purchase information:

[0714] A user purchases food items such as milk, butter, and bread at a supermarket and completes the payment using an electronic payment app. The device then sends the purchased item information from the electronic payment app to the server.

[0715] 1. Refrigerator storage criteria:

[0716] The server uses an AI model to determine whether or not a product needs to be refrigerated based on the product information. For example, it may determine that milk and butter need to be refrigerated, while bread is recommended to be stored at room temperature.

[0717] 1. Register for the refrigerator sub-app:

[0718] The server sends the result of the judgment to the refrigerator sub-application, which notifies the user's device. The user then stores the milk and butter they bought in the refrigerator.

[0719] 1. Refrigerator management and recipe suggestions:

[0720] The device (refrigerator) uses a camera to recognize newly stored ingredients and sends the information to the server. The server updates the information about ingredients in the refrigerator and generates recipes based on the ingredients it has. For example, it generates a recipe for "cream stew" based on the latest list of ingredients in the refrigerator (milk, butter, eggs, cheese) and sends it to the user's device. The user's device notifies the user of the recipe, and the user begins cooking based on that information.

[0721] An example of a prompt to be input to the generative AI model is, "Please explain the processing flow of a system in which a user purchases milk, butter, and bread at the supermarket, determines the need to store them in the refrigerator based on that information, registers them in the refrigerator sub-app, and then manages the information about ingredients in the refrigerator and suggests recipes."

[0722] As a result, the present invention provides a system that is highly convenient for users, reduces food waste, and assists in efficient meal planning.

[0723] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0724] Processing Steps

[0725] Step 1: Obtain purchase information

[0726] User: Purchases ingredients at a supermarket or online shop and completes payment using an electronic payment app.

[0727] Input: Purchased product information (product name, category, quantity, etc.)

[0728] How it works: A user uses the SuperPay app to buy milk, butter, and bread.

[0729] Output: Purchase product information is generated from the electronic payment app and sent to the terminal.

[0730] Step 2: Submit your purchase information

[0731] Terminal: Sends purchased product information to the server.

[0732] Input: Purchase information obtained from the electronic payment app

[0733] Operation: The terminal sends purchased product information to the server.

[0734] Output: The server receives the purchased product information.

[0735] Step 3: Determine if refrigeration is necessary

[0736] Server: Uses AI models to determine whether purchased items need to be stored in the refrigerator.

[0737] Input: Purchase product information

[0738] How it works: The server uses TensorFlow to determine that milk and butter need to be stored in the refrigerator, but bread can be stored at room temperature.

[0739] Output: A list of products that need to be stored in the refrigerator and those that can be stored at room temperature

[0740] Step 4: Registering the refrigerator sub-app

[0741] Server: Sends information about products that need to be stored in the refrigerator to the refrigerator sub-app.

[0742] Input: List of products that need to be kept in the refrigerator

[0743] Operation: The server sends milk and butter information to the refrigerator sub-app, which then notifies the user's device.

[0744] Output: Product information registered in the refrigerator sub-app and notification to the user

[0745] Step 5: Get information about ingredients in the refrigerator

[0746] Terminal (refrigerator): Detects food ingredient information using sensors and cameras inside the refrigerator and sends that information to the server.

[0747] Input: Sensor data and camera data from inside the refrigerator

[0748] How it works: The refrigerator's camera recognizes newly placed ingredients (milk and butter), and the sensor captures temperature information.

[0749] Output: Sensor data and camera data are sent to the server.

[0750] Step 6: Update the ingredients in your refrigerator

[0751] Server: Updates the information about ingredients in the refrigerator based on the received sensor data and camera data.

[0752] Input: Sensor data and camera data

[0753] Operation: The server updates the ingredient information in the database to keep the ingredient list up to date.

[0754] Output: Updated ingredient information

[0755] Step 7: Recipe suggestions

[0756] Server: Generates recipes using AI based on updated ingredient information.

[0757] Input: Updated ingredient information

[0758] Operation: Generates a recipe for "cream stew" based on the ingredient information held by the server.

[0759] Output: The generated recipe

[0760] Step 8: Notify the user

[0761] Terminal (user terminal): Receives recipe suggestions sent from the server and notifies the user.

[0762] Input: Generated recipe

[0763] Operation: The user terminal notifies the user of the recipe.

[0764] Output: The user receives a recipe notification.

[0765] These steps allow users to automate their purchase management, update their refrigerator with ingredients, and suggest recipes, reducing food waste and enabling efficient meal planning.

[0766] (Application example 1)

[0767] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0768] Conventional refrigerator management systems require users to manually enter information about purchased products, which takes time and effort. They also lack a means to automatically determine whether purchased products need to be stored in the refrigerator and manage them appropriately. Furthermore, recipe suggestions based on refrigerator inventory information are not provided in real time, making it difficult to make effective use of ingredients.

[0769] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0770] In this invention, the server includes means for receiving purchased product information from the electronic payment app, means for determining using AI whether the purchased product needs to be stored in the refrigerator, means for registering information on the purchased product determined to need to be stored in the refrigerator in a refrigerator sub-app, means for notifying the user terminal of the registered purchased product information, and means for automatically reading the purchased product information at the physical store and means for updating inventory information in the refrigerator based on that information, managing ingredients, and suggesting recipes. This allows the user to automatically register information on purchased products in the refrigerator, allowing them to grasp inventory information in the refrigerator in real time and use ingredients efficiently.

[0771] An "electronic payment app" is an application that allows users to pay for goods electronically.

[0772] "Purchased product information" refers to information such as the name, category, and quantity of the product purchased by the user.

[0773] "AI" stands for artificial intelligence, a system that learns and reasons to solve specific problems.

[0774] The "refrigerator sub-app" is an application that manages refrigerator inventory information and provides the user with information on ingredients and recipe suggestions.

[0775] A "user terminal" is an information processing device operated by a user, such as a smartphone, tablet, or personal computer.

[0776] A "physical store" is a store where users can physically purchase products.

[0777] "Means for automatically reading information about purchased items" refers to technology that automatically obtains information about purchased items using sensors, cameras, etc.

[0778] "Inventory information" is information that indicates the current status of ingredients and products in the refrigerator.

[0779] "Recipe suggestion" refers to suggesting cooking methods to the user based on information about ingredients in the refrigerator.

[0780] "Sensor data" is digital data obtained from sensors regarding temperature, humidity, the presence of objects, etc.

[0781] "Camera data" refers to image and video data captured by a camera.

[0782] A "common API" is an application program interface that can be used commonly across multiple systems.

[0783] A "smart device" is an electronic device that can connect to the Internet and allows users to input and display information.

[0784] "Licensing revenue" is revenue earned by permitting the use of a certain technology or product.

[0785] This invention is a system that automatically acquires information about products purchased by a user, updates refrigerator inventory based on that information, and manages ingredients and suggests recipes. This system is configured using the following hardware and software.

[0786] Hardware

[0787] Smartphone: A device that allows users to purchase products and use electronic payment apps.

[0788] Smart glasses: devices that display real-time information while users are shopping in a physical store.

[0789] Refrigerator: A home appliance equipped with sensors and cameras inside that can obtain information about the food inside the refrigerator.

[0790] Server: A central processing unit that processes purchased product information and data from the refrigerator.

[0791] software

[0792] Electronic payment app: An application that allows users to purchase products and make payments.

[0793] Refrigerator sub-app: An application that manages the inventory information in the refrigerator and notifies the user.

[0794] Common API: An interface for receiving and analyzing sensor data and camera data from refrigerator manufacturers.

[0795] AI model: An artificial intelligence model that analyzes purchased product information and determines whether or not it needs to be stored in the refrigerator.

[0796] In this system, users first purchase products at a physical store and pay through an electronic payment app. Purchase information is automatically sent to a smartphone at the time of payment. This information is then sent to a server, where an AI model analyzes it and determines whether the product needs to be stored in a refrigerator.

[0797] Next, product information that needs to be stored in the refrigerator is registered in the refrigerator sub-app and simultaneously notified to the user's device. The user receives this notification and stores the necessary ingredients in the refrigerator.

[0798] The refrigerator uses internal sensors and cameras to detect newly stored ingredients. The detected data is sent to the server via a common API. The server uses this data to update the refrigerator's inventory and generate optimal recipes.

[0799] The generated recipe is sent to the user's device, and the user can start cooking based on the recipe. This allows the user to automatically register purchased product information in the refrigerator, keep track of inventory in the refrigerator in real time, and use ingredients efficiently.

[0800] Specific examples

[0801] For example, suppose a user purchases milk, butter, and bread at a physical store and completes the payment using an electronic payment app. At this time, the purchase information sent from the payment app is sent to a server, where an AI model analyzes it and determines that the milk and butter need to be stored in the refrigerator. This information is registered in the refrigerator sub-app, and the user checks the information on their smartphone. Next, sensors and cameras inside the refrigerator detect the newly placed ingredients and send the information to the server. The server analyzes the data and generates a recipe (e.g., salad, pasta, smoothie) based on the ingredients on hand, and notifies the user.

[0802] Prompt Sentence Examples

[0803] When a user purchases food at a physical store and completes the payment through a payment app on their smartphone, please explain the process by which the purchase information is registered in the refrigerator sub-app. Also, please explain how recipes are suggested based on the information about ingredients in the refrigerator.

[0804] In this way, the present invention greatly improves the user's daily life, allowing for efficient food management and cooking.

[0805] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0806] Step 1:

[0807] A user purchases an item in a physical store and pays using an electronic payment app.

[0808] Input: Information about the product purchased by the user (product name, category, quantity, etc.)

[0809] What happens: A user uses their smartphone to scan an item into a payment app and complete the payment.

[0810] Output: Purchase product information is generated and sent to your smartphone.

[0811] Step 2:

[0812] The smartphone sends the purchased product information to the server.

[0813] Input: Purchase information obtained from your smartphone

[0814] Specific operation: The smartphone obtains the purchased product information from the electronic payment app and sends it to the server as an HTTP request.

[0815] Output: Purchase information is sent to the server.

[0816] Step 3:

[0817] The server uses an AI model to determine whether refrigeration is necessary.

[0818] Input: Purchase information sent to the server

[0819] How it works: The server inputs purchased product information into the AI ​​model and determines whether or not the product needs to be stored in the refrigerator. The AI ​​model makes this determination based on the product name and category.

[0820] Output: A list of items that need to be kept in a refrigerator is generated.

[0821] Step 4:

[0822] The server registers product information that needs to be stored in the refrigerator in the refrigerator sub-app.

[0823] Input: List of products that require refrigeration as determined by the server

[0824] Specific operation: The server sends the registration information to the refrigerator sub-app via API.

[0825] Output: Product information that needs to be stored is registered in the refrigerator sub-app.

[0826] Step 5:

[0827] The refrigerator sub-app notifies the user device of the registration information

[0828] Input: Product information registered in the refrigerator sub-app

[0829] Specific operation: The refrigerator sub-app notifies the user device of the information received from the server. The information is sent to the user using push notifications, etc.

[0830] Output: The registration information for the purchased product is displayed on the user's device.

[0831] Step 6:

[0832] The user places the product in the refrigerator

[0833] Input: Registration information displayed on the user's device

[0834] Specific operation: The user places the purchased items in the refrigerator. The product placement is performed while confirming the registration information.

[0835] Output: The product is stored in the refrigerator.

[0836] Step 7:

[0837] The refrigerator detects the food stored inside using internal sensors and cameras.

[0838] Input: New item placed in refrigerator

[0839] How it works: The refrigerator uses internal sensors and cameras to detect newly added ingredients.

[0840] Output: The detected ingredients information is generated.

[0841] Step 8:

[0842] The refrigerator sends the food information it detects to the server.

[0843] Input: Detected ingredient information

[0844] Specific operation: The refrigerator sends ingredient information to the server using a common API.

[0845] Output: Ingredient information is sent to the server.

[0846] Step 9:

[0847] The server updates the ingredient information and generates the optimal recipe.

[0848] Input: Ingredient information sent to the server

[0849] Specific operation: The server updates the inventory data in the refrigerator based on the received ingredient information and generates the optimal recipe using a generative AI model.

[0850] Output: The optimal recipe information is generated.

[0851] Step 10:

[0852] The server sends the generated recipe to the user's device.

[0853] Input: Generated recipe information

[0854] Specific operation: The server sends the generated recipe information to the user's device. The recipe information is sent to the user using push notifications, etc.

[0855] Output: Recipe information is displayed on the user's device.

[0856] Through the above processing steps, the user can automatically register information about purchased items in the refrigerator, grasp inventory information in the refrigerator in real time, and use ingredients efficiently.

[0857] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0858] 1. System Overview

[0859] The system of this invention receives information about products purchased by users through an electronic payment app, and based on that information, AI determines whether the product needs to be stored in the refrigerator and automatically registers that information in the refrigerator sub-app. It also includes a function to manage information about ingredients in the refrigerator and suggest recipes to users based on that information. This system works in conjunction with refrigerator manufacturers, acquiring sensor and camera data through a common API to update information about ingredients in the refrigerator in real time. Furthermore, by combining it with an emotion engine, recipes are personalized based on the user's emotions, improving the user experience.

[0860] 2. System Configuration and Operation

[0861] Receiving purchase information

[0862] User:

[0863] When a user purchases ingredients at a supermarket or online shop and pays using an electronic payment app, purchased product information is generated.

[0864] Device:

[0865] Purchase information (product name, category, quantity, etc.) is sent from the electronic payment app to the server.

[0866] server:

[0867] The server receives the purchased product information and uses an AI model to determine whether it needs to be stored in a refrigerator.

[0868] Determining whether it can be stored in a refrigerator

[0869] server:

[0870] The AI ​​model analyzes the purchased product information and determines whether it needs to be stored in a refrigerator. For example, milk and butter are determined to be products that need to be stored in a refrigerator.

[0871] Device:

[0872] The information about products that need to be stored in the refrigerator is registered in the refrigerator sub-application, which receives the information from the server. The registered information is also sent to the user's device.

[0873] Refrigerator management and recipe suggestions

[0874] Terminal (refrigerator):

[0875] The storage status of ingredients is detected using cameras and sensors inside the refrigerator, and that information is sent to a server.

[0876] server:

[0877] The system analyzes sensor and camera data received from the refrigerator and updates the information about ingredients in the refrigerator. It then generates recipes based on the updated information.

[0878] Terminal (user terminal):

[0879] The server receives the recipe suggestions and notifies the user. The user checks the recipe, takes out the necessary ingredients from the refrigerator, and starts cooking.

[0880] Collaboration with refrigerator manufacturers

[0881] server:

[0882] We provide refrigerator manufacturers with a common API to collect sensor and camera data from refrigerators, enabling accurate and real-time updates of food ingredients stored in refrigerators. We also monitor API usage and manage license revenue.

[0883] Terminal (refrigerator):

[0884] Information from inside the refrigerator is sent to the server via the API, and management information from the server is displayed inside the refrigerator.

[0885] Incorporating an emotion engine

[0886] User Emotion Recognition and Recipe Personalization

[0887] Device (smartphone, etc.):

[0888] It incorporates an emotion engine that analyzes emotions through facial recognition and voice data of the user. For example, it uses a camera and microphone to analyze facial expressions and voices while the user is operating the device.

[0889] server:

[0890] It receives emotional data sent from the emotion engine and personalizes recipes based on that data, for example, suggesting easy-to-make recipes if the user is tired.

[0891] Terminal (user terminal):

[0892] It notifies users of personalized recipes and makes meal suggestions based on the user's emotions.

[0893] Emotion-based food consumption and management

[0894] server:

[0895] The system continuously monitors the user's emotional data and suggests food consumption and management based on their emotions. For example, if the user is feeling stressed, it will suggest recipes with ingredients that are effective in reducing stress.

[0896] Device (smartphone, etc.):

[0897] It notifies users in a timely manner and provides alerts about food expiration dates and conditions based on emotional data.

[0898] Specific examples

[0899] Example of operation at time of purchase

[0900] 1. A user selects items (milk, butter, bread) at a supermarket and completes the purchase using an electronic payment app.

[0901] 2. The device sends the purchased product information to the server, which uses AI to determine whether the milk and butter need to be stored in the refrigerator.

[0902] 3. The registration information is sent to the refrigerator sub-app, and the user stores the purchased milk and butter in the refrigerator.

[0903] Example of refrigerator management and recipe suggestions

[0904] 1. The terminal (refrigerator) detects newly stored ingredients and sends the information to the server.

[0905] 2. The server updates the information about ingredients in the refrigerator and generates a recipe based on the ingredients you have.

[0906] 3. The generated recipe is sent to the user's device, and the user checks the recipe and cooks the ingredients.

[0907] Example of Emotion Engine in Action

[0908] 1. The device (smartphone, etc.) collects the user's facial recognition and voice data and analyzes it using an emotion engine.

[0909] 2. Based on the emotional data, the server suggests recipes using ingredients that have a relaxing effect, for example, if the user feels like relaxing.

[0910] 3. The user checks the appropriate recipe and uses it to cook.

[0911] This system allows users to automate their purchase management, reducing wasteful consumption and creating effective meal plans. Furthermore, the incorporation of an emotion engine enables personalized meal suggestions that are in line with the user's emotions, which is expected to improve the user experience.

[0912] The processing flow will be explained below.

[0913] Program processing steps

[0914] Receiving and determining purchase information

[0915] Step 1:

[0916] A user selects a product at a supermarket or online shop and completes the purchase process using an electronic payment app.

[0917] Step 2:

[0918] The device (smartphone, etc.) captures information about the purchased product (product name, category, quantity, etc.) and sends it to the server.

[0919] Step 3:

[0920] The server receives the purchased product information sent from the electronic payment application.

[0921] Step 4:

[0922] The server uses an AI model to analyze the received purchase information and determine whether it needs to be stored in a refrigerator.

[0923] For example, it determines that "milk" and "butter" need to be stored in the refrigerator, but "bread" does not.

[0924] Step 5:

[0925] The server generates a list of products that need to be stored in the refrigerator and issues an instruction to register the list in the refrigerator sub-app.

[0926] Step 6:

[0927] The terminal (smartphone, etc.) receives instructions from the server and registers the purchased items in the refrigerator sub-app.

[0928] After the terminal has completed registration of the purchased product information, it notifies the user.

[0929] Refrigerator management and recipe suggestions

[0930] Step 7:

[0931] The food purchased by the user is stored in the refrigerator.

[0932] Step 8:

[0933] The terminal (refrigerator) detects newly stored ingredients using its internal camera and sensors and sends this information to the server.

[0934] Step 9:

[0935] The server receives the data sent from the refrigerator sub-application and updates the information about ingredients in the refrigerator.

[0936] Step 10:

[0937] The server generates recipes based on the ingredients in the refrigerator.

[0938] For example, generate a recipe for "Omelette with butter and eggs."

[0939] Step 11:

[0940] The server transmits the generated recipe information to the user's terminal.

[0941] Step 12:

[0942] The terminal (smartphone, etc.) notifies the user of the recipe suggestions sent from the server.

[0943] Step 13:

[0944] The user checks the recipe, takes the necessary ingredients from the refrigerator, and starts cooking.

[0945] Collaboration with refrigerator manufacturers

[0946] Step 14:

[0947] The server provides a common API to refrigerator manufacturers.

[0948] Step 15:

[0949] The terminal (refrigerator) sends information from the interior camera and sensors to the server via API.

[0950] Step 16:

[0951] The server analyzes the data sent from the refrigerator and automatically updates the information about ingredients in the refrigerator.

[0952] Step 17:

[0953] The server monitors API usage and manages license revenue from refrigerator manufacturers.

[0954] Incorporating an emotion engine

[0955] Step 18:

[0956] The device (smartphone, etc.) collects the user's facial recognition and voice data and analyzes it using an emotion engine.

[0957] Step 19:

[0958] The server uses emotional data to suggest easy-to-make recipes, for example, if the user feels tired.

[0959] Step 20:

[0960] The device (smartphone, etc.) notifies the user of the personalized recipe.

[0961] Step 21:

[0962] The user checks the recipe based on their emotions and uses the recipe to cook.

[0963] Step 22:

[0964] The server continuously monitors emotional data and suggests recipes using ingredients that are effective in reducing stress.

[0965] Step 23:

[0966] The device (smartphone, etc.) notifies the user in a timely manner and provides alerts regarding the expiration date and condition of ingredients based on emotional data.

[0967] This allows users to automate the management of their purchases and create efficient meal plans, and the incorporation of an emotion engine makes it possible to make personalized meal suggestions that are in line with the user's emotions, which is expected to improve the user experience.

[0968] Example 2

[0969] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0970] Currently, there are many grocery management systems and recipe recommendation systems on the market, but these systems have limitations in terms of centralized management and personalized recommendations. In particular, in terms of refrigerator ingredient management and recipe recommendations based on the user's mood, existing systems lack flexibility and do not sufficiently improve the user experience. Furthermore, they lack automated mechanisms for reducing food waste and efficiently planning meals. This invention aims to solve these problems and improve the quality of life of users.

[0971] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0972] In this invention, the server includes: means for receiving purchased product information from an electronic payment app; means for determining whether a purchased product requires refrigeration using AI; and means for registering information about purchased products determined to require refrigeration in a refrigeration appliance app. This allows for efficient management of purchased product information and enables appropriate storage based on the information. The server also includes means for updating food product information stored in the refrigeration appliance based on data from the refrigeration appliance app; means for generating recipes based on the food product information stored in the refrigeration appliance; and means for transmitting the generated recipes to a user terminal. This allows for real-time updates of food product information stored in the refrigeration appliance, and for suggesting recipes suitable for the user based on the updated information. The server also includes means for providing a common API to refrigeration appliance manufacturers; means for receiving sensor data and image data from the refrigeration appliance; means for analyzing the data and updating the food product information stored in the refrigeration appliance; and means for monitoring the usage of the common API and managing license revenue. This allows for efficient collection and management of refrigeration appliance data, enabling appropriate food product management and revenue generation. The server also includes means for recognizing user emotions; means for personalizing recipes based on the emotion data; and means for notifying the user terminal of the personalized recipes. This allows for personalized meal suggestions based on the user's emotions, further improving the user experience.

[0973] 1. An "electronic payment app" is a software application that a user uses to purchase goods online or in-store, and that digitizes the payment process.

[0974] 2. "Purchased Product Information" refers to detailed data of the product purchased by the user, including product name, category, quantity, price, etc.

[0975] 3. "AI" stands for artificial intelligence, which refers to algorithms and models that analyze data and perform specific tasks automatically.

[0976] 4. "Refrigeration equipment app" is software that supports the operation of refrigeration equipment, managing food ingredient information and monitoring storage conditions.

[0977] 5. "User terminal" refers to a device used by a user, such as a smartphone, tablet, PC, or other electronic device.

[0978] 6. "Sensor Data" means information detected by sensors, such as temperature, humidity, and location within refrigeration equipment.

[0979] 7. "Image data" refers to image files taken by a camera placed inside the refrigeration equipment, which are used to visually record the condition and placement of ingredients.

[0980] 8. "Common API" means a common application program interface that can be used by multiple refrigeration equipment manufacturers, enabling data transmission and reception and the sharing of functions.

[0981] 9. "License Revenue" means revenue received from refrigeration equipment manufacturers for use of the provided common API, including API usage fees and service fees.

[0982] 10. "Emotional Data" means information about a user's emotional state obtained through facial recognition or voice analysis, and is data indicating joy, sadness, fatigue, stress, etc.

[0983] 11. "Personalized Cooking Recipes" are cooking recipes that are specifically tailored to a user's individual emotional state and preferences and that are provided as part of meal suggestions to the user.

[0984] 1. System Overview

[0985] The system of this invention receives information about products purchased by users through an electronic payment app, and based on that information, AI determines the need for refrigeration and automatically registers that information in the refrigeration appliance app. It also includes a function to manage food information stored in the refrigeration appliance and suggest recipes to users based on that information. This system works in conjunction with refrigeration appliance manufacturers, acquiring sensor data and image data through a common API to update the food information stored in the refrigeration appliance in real time. Furthermore, by incorporating an emotion engine, recipes are personalized based on the user's emotions, improving the user experience.

[0986] 2. System Configuration and Operation

[0987] Receiving purchase information

[0988] When a user purchases groceries at a supermarket or online shop and pays using an electronic payment app, purchased product information (product name, category, quantity, etc.) is generated within the electronic payment app.

[0989] The terminal (electronic payment app) sends the purchased product information to the server. For example, it is sent to the cloud server in the form of an electronic receipt.

[0990] The server receives the purchased product information and records it in a database.

[0991] Determining whether refrigerated storage is possible

[0992] The server uses AI models to determine the need for refrigeration based on the purchased product information received, for example, milk and perishables.

[0993] The device registers the information about the products that need to be stored sent from the server in the refrigeration equipment app and notifies the user's device, allowing the user to store their purchased items appropriately.

[0994] Refrigeration equipment management and recipe suggestions

[0995] The terminal (refrigeration equipment) uses internal sensors and cameras to monitor the condition of the food and sends the data to a server.

[0996] The server analyzes the received sensor data and image data and updates the information in the refrigeration equipment, for example, identifying the type and quantity of ingredients.

[0997] The server generates usable recipes based on updated ingredient information, taking into account nutritional value and user preferences.

[0998] The terminal (user terminal) receives the recipe suggestions sent from the server and notifies the user, who can then check the recipes and prepare the suggested meals.

[0999] Collaboration with refrigeration equipment manufacturers

[1000] The server provides a common API to refrigeration equipment manufacturers and acquires sensor data and image data, for example, to manage fresh vegetables and dairy products.

[1001] The terminal (refrigeration equipment) sends data to the server via API, which updates the information in the refrigeration equipment in real time, allowing users to manage their ingredients appropriately.

[1002] Specific examples

[1003] Example of operation at time of purchase

[1004] 1. A user selects milk, butter, and bread at a supermarket and completes the purchase using an electronic payment app.

[1005] 2. The terminal (electronic payment app) sends the purchased product information to the server.

[1006] 3. The server uses AI to determine that the milk and butter need to be refrigerated.

[1007] 4. The registration information is sent to the refrigerator app, and the user places the milk and butter in the refrigerator.

[1008] Example of refrigeration equipment management and recipe suggestions

[1009] 1. The terminal (refrigeration equipment) detects newly stored food items and sends the information to the server.

[1010] 2. The server updates the food information in the refrigerator and generates a recipe based on the ingredients it has.

[1011] 3. The generated recipe is sent to the user's device, and the user checks the recipe and cooks the ingredients.

[1012] Example of Emotion Engine in Action

[1013] 1. The device (smartphone, etc.) collects the user's facial recognition and voice data and analyzes it using an emotion engine.

[1014] 2. Based on the emotional data, the server suggests recipes using ingredients that have a relaxing effect if the user feels like relaxing.

[1015] 3. The user checks the appropriate recipe and uses it to cook.

[1016] Example prompts for generative AI models

[1017] Receiving purchase information

[1018] Prompt: Receive information about the milk, butter, and bread purchased by a user at the supermarket from an electronic payment app, and use AI to determine whether they need to be stored in the refrigerator.

[1019] Refrigeration equipment management and recipe suggestions

[1020] Prompt: Send the newly stored ingredients to the server, update the ingredients in the refrigerator, and suggest a recipe.

[1021] Emotion Engine Operation

[1022] Prompt: Collect facial and voice data from the user and suggest recipes suitable for when the user wants to relax.

[1023] By using this system, users can automate the management of their purchases, reduce wasteful consumption, and create effective meal plans. In addition, the incorporation of an emotion engine enables personalized meal suggestions that are in line with the user's emotions, which is expected to improve the user experience.

[1024] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1025] Step 1:

[1026] A user purchases groceries at a supermarket or online shop and pays using an electronic payment app. At this time, product information (product name, category, quantity, etc.) is generated within the electronic payment app. The user inputs their intention to purchase a product, and as a result, product information is generated.

[1027] Step 2:

[1028] The terminal (electronic payment app) sends purchased product information to the server. For example, the purchase information is sent to the cloud server in the form of an electronic receipt. The terminal receives purchased product information as input and sends it to the server as output.

[1029] Step 3:

[1030] The server receives the purchased product information and records it in the database. The server analyzes the purchased product information received as input and saves it in the database. This records the purchased product information.

[1031] Step 4:

[1032] The server uses an AI model to determine the need for refrigerated storage based on the received purchase information. Purchase information is provided to the AI ​​model as input, and the AI ​​analyzes the data to determine the need for refrigerated storage. For example, it may determine that milk or fresh food needs to be stored in a refrigerator.

[1033] Step 5:

[1034] The terminal registers the information about the products that require storage sent from the server in the refrigeration equipment app and notifies the user terminal.The terminal receives information about the products that require refrigeration storage from the server as input, processes it as output to be registered in the refrigeration equipment app, and notifies the user terminal.

[1035] Step 6:

[1036] The terminal (refrigeration equipment) monitors the condition of food using internal sensors and cameras and sends the data to a server. The refrigeration equipment's sensors and cameras acquire food condition data as input and send it to the server as output. For example, sensor data on temperature and humidity, and image data of food.

[1037] Step 7:

[1038] The server analyzes the received sensor data and image data and updates the food ingredient information in the refrigerator. The server receives sensor data and image data as input, analyzes them to identify the type and amount of food ingredients, and updates the food ingredient information in the refrigerator as output.

[1039] Step 8:

[1040] The server generates usable recipes based on updated ingredient information, taking into account nutritional value and user preferences. Using the latest ingredient information as input, the AI ​​model analyzes the data and generates suitable recipes as output.

[1041] Step 9:

[1042] The terminal (user terminal) receives the recipe suggestions sent from the server and notifies the user. It receives the recipe information provided by the server as input and suggests it to the user using the notification function as output. The user checks this recipe and prepares the suggested meal.

[1043] Step 10:

[1044] The device (such as a smartphone) collects the user's facial recognition and voice data and analyzes it using an emotion engine. The user's facial recognition and voice data are taken as input, and the output is analyzed by the emotion engine. For example, data collection using a camera or microphone.

[1045] Step 11:

[1046] The server analyzes the emotional data and suggests recipes suitable for when the user wants to relax or when they are tired. The server receives the emotional data as input, analyzes it with an AI model, and generates suitable recipes as output.

[1047] Step 12:

[1048] The user checks the personalized recipe notified to the device and uses that recipe to cook. The notified recipe information is received as input, and cooking is performed based on that recipe as output. The user's cooking action is the final execution stage.

[1049] (Application example 2)

[1050] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1051] In conventional refrigerator and purchasing information management systems, storing and managing purchased products is often done manually, which is cumbersome for users and makes it difficult to effectively manage ingredients. Furthermore, there is a lack of personalized recipe suggestions based on the user's emotions and condition, which hinders the improvement of the user experience. There is a need for an efficient and user-friendly system that can solve these problems.

[1052] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1053] In this invention, the server includes means for receiving purchased product information from the electronic payment app, means for determining using AI whether the purchased product needs to be stored in the refrigerator, means for registering information on the purchased product determined to need to be stored in the refrigerator in a refrigerator sub-app, means for notifying the user terminal of the registered purchased product information, means for automatically acquiring purchase information from a physical store and linking with the refrigerator, means for personalizing recipes based on the user's emotions using an emotion engine, and means for analyzing the user's emotion data and suggesting appropriate ingredient consumption and recipes. This automates the storage and management of purchased products and makes it possible to realize personalized ingredient consumption and recipe suggestions based on the user's emotions.

[1054] An "electronic payment app" is a software application that receives purchase information electronically and completes payments.

[1055] "Purchased product information" is data including detailed information such as the name, category, and quantity of the product purchased by the user.

[1056] "AI" refers to artificial intelligence systems that automate specific tasks, analyze data, and make decisions.

[1057] The "refrigerator sub-app" is an auxiliary application for managing and updating information about ingredients in the refrigerator.

[1058] A "user terminal" is an electronic device operated by a user, such as a smartphone or tablet.

[1059] A "physical store" is a physical sales location where products are actually displayed and sold.

[1060] An "emotion engine" is software that analyzes a user's emotions and makes appropriate suggestions based on their state.

[1061] A "recipe" is a document that describes the steps and ingredients for making a particular dish.

[1062] A "common API" is a standardized interface that allows different systems to communicate with each other and exchange data.

[1063] "Sensor data" is information obtained from sensors that detect the temperature, humidity, and presence of objects inside the refrigerator.

[1064] "Camera data" refers to data acquired as video or images of the inside of a refrigerator.

[1065] A "generative AI model" is an artificial intelligence model designed to generate specific outputs based on user input.

[1066] A "prompt sentence" is an input sentence that a generative AI model uses to generate appropriate output.

[1067] 1. System Overview

[1068] This invention is a system that receives information about products purchased by users through an electronic payment app, and then uses AI to determine whether or not to store the products in the refrigerator, automatically registering them in a refrigerator sub-app. It also includes a function to manage information about ingredients in the refrigerator and suggest recipes based on that information, allowing recipes to be personalized based on the user's emotions.

[1069] 2. Program Generation

[1070] The program to realize this system is as follows, but the details of the code are not included. The specific main functions are as follows:

[1071] Receiving purchase information: The user sends product information from the electronic payment app.

[1072] Refrigerated storage decision: AI analyzes product information and determines the need for refrigerated storage.

[1073] Information registration: Register the necessary product information in the refrigerator sub-app.

[1074] User notification: Registration information is sent to the user's device.

[1075] Sentiment Analysis: The emotion engine analyzes user emotions and personalizes recipes.

[1076] Recipe suggestion: Generates recipes based on emotion data and notifies the user.

[1077] 3. Processing Description

[1078] Receiving purchase information

[1079] When a user purchases a product through an electronic payment app, purchased product information (product name, category, quantity, etc.) is generated. The server receives and analyzes this information.

[1080] Refrigerator storage judgment

[1081] The server uses an AI model to analyze the received purchase information and determine which products need to be stored in the refrigerator. The necessary product information is then registered in the refrigerator sub-app.

[1082] Physical store integration and continuous processing

[1083] Information about products purchased at physical stores is automatically sent to a server via the electronic payment app, which links purchase information with information in the refrigerator, reducing the burden on users.

[1084] Refrigerator management

[1085] Sensor and camera data from inside the refrigerator is sent to the server via a common API, which then analyzes the data and updates the information about ingredients in the refrigerator in real time.

[1086] Emotional engine and recipe suggestions

[1087] The emotion engine analyzes the user's facial expressions and voice to generate emotion data. Based on this, the server suggests recipes appropriate for the user's condition. For example, if the user is tired, it will suggest easy-to-make recipes.

[1088] 4. Usage example

[1089] Recipe suggestions based on refrigerator status data and user emotion data

[1090] Current refrigerator data:

[1091] Milk: 1 bottle

[1092] Bread: 2 pieces

[1093] User sentiment data:

[1094] Facial Expression: Tired

[1095] Audio Tone: Low

[1096] Prompt Sentence Examples

[1097] The system suggests suitable recipes based on the current state of the refrigerator and the user's emotional data.

[1098] For example, generate a recipe with the following conditions:

[1099] Refrigerator Data:

[1100] Milk: 1 bottle

[1101] Bread: 2 pieces

[1102] User sentiment data:

[1103] Facial Expression: Tired

[1104] Audio Tone: Low

[1105] Generated recipe example:

[1106] 1. Easy pasta recipes

[1107] 2. Relaxing Salad Recipes

[1108] As described above, the system receives information about purchased items and provides consistent support, from managing items in the refrigerator to suggesting recipes based on the user's emotions. This reduces the burden on the user and provides a personalized experience.

[1109] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1110] Step 1:

[1111] When a user purchases a product through an electronic payment app, purchased product information (product name, category, quantity, etc.) is generated. The user purchases ingredients at a supermarket or online shop and completes the payment through the electronic payment app. At this time, purchased product information is automatically generated.

[1112] Input: Purchased product information (product name, category, quantity, etc.)

[1113] Output: Purchase product information data

[1114] Step 2:

[1115] The terminal sends the purchased product information to the server. The purchased product information is sent from the electronic payment app to the server.

[1116] Input: Purchase product information data

[1117] Output: Purchased product information sent to the server

[1118] Step 3:

[1119] The server uses an AI model to analyze the received purchase information and determine whether or not the product needs to be stored in a refrigerator. For example, it identifies products that need to be refrigerated, such as milk and butter.

[1120] Input: Purchase information sent to the server

[1121] Processing: AI model determines whether refrigeration is necessary

[1122] Output: List of items that need to be stored

[1123] Step 4:

[1124] The server registers product information that needs to be stored in the refrigerator sub-app, which then automatically records these products.

[1125] Input: List of items that need to be stored

[1126] Output: Product information registered in the refrigerator sub-app

[1127] Step 5:

[1128] The server notifies the user of the registration information on their smartphone or other device, allowing them to check the products they have purchased.

[1129] Input: Product information registered in the refrigerator sub-app

[1130] Output: Notification to user terminal

[1131] Step 6:

[1132] The emotion engine is used to analyze the user's emotions. Facial expressions and voice data are acquired from the user's smartphone or tablet, and the emotion engine analyzes them.

[1133] Input: User facial expressions and voice data

[1134] Processing: Sentiment analysis using the emotion engine

[1135] Output: User emotion data

[1136] Step 7:

[1137] The server personalizes recipes based on the user's emotional data: if the user is tired, it suggests simple recipes, and if the user is relaxed, it suggests more elaborate recipes.

[1138] Input: User emotion data, information about ingredients in the refrigerator

[1139] Processing: Recipe generation based on sentiment data

[1140] Output: personalized recipe

[1141] Step 8:

[1142] The server sends personalized recipes to the user's device, which then notifies the user of the recipes on their smartphone or other device, allowing the user to check the appropriate recipes.

[1143] Input: Personalized Recipe

[1144] Output: Recipe notification to user's device

[1145] Examples of specific examples and prompts

[1146] Usage example

[1147] The system suggests suitable recipes based on the current state of the refrigerator and the user's emotional data.

[1148] For example, generate a recipe with the following conditions:

[1149] Refrigerator Data:

[1150] Milk: 1 bottle

[1151] Bread: 2 pieces

[1152] User sentiment data:

[1153] Facial Expression: Tired

[1154] Audio Tone: Low

[1155] Generated recipe example:

[1156] 1. Easy pasta recipes

[1157] 2. Relaxing Salad Recipes

[1158] The above is a specific embodiment of the invention, which allows users to store and manage their purchases automatically and provides a personalized experience based on emotions.

[1159] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1160] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1161] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1162] [Third embodiment]

[1163] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1164] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1165] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1166] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1167] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1168] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1169] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1170] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1171] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1172] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1173] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1174] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1175] 1. System Overview

[1176] The system of this invention receives information about products purchased by users through an electronic payment app, and based on that information, AI determines whether the product needs to be stored in the refrigerator and automatically registers that information in the refrigerator sub-app. It also includes a function to manage information about ingredients in the refrigerator and suggest recipes to users based on that information. This system works in conjunction with refrigerator manufacturers, acquiring sensor and camera data through a common API, allowing it to update information about ingredients in the refrigerator in real time.

[1177] 2. System Configuration and Operation

[1178] Receiving purchase information

[1179] User:

[1180] When a user purchases ingredients at a supermarket or online shop and pays using an electronic payment app, purchased product information is generated.

[1181] Device:

[1182] Purchase information (product name, category, quantity, etc.) is sent from the electronic payment app to the server.

[1183] server:

[1184] The server receives the purchased product information and uses an AI model to determine whether it needs to be stored in a refrigerator.

[1185] Determining whether it can be stored in a refrigerator

[1186] server:

[1187] The AI ​​model analyzes the purchased product information and determines whether it needs to be stored in a refrigerator. For example, it determines that milk and butter are products that need to be stored in a refrigerator.

[1188] Device:

[1189] The information about products that need to be stored in the refrigerator is registered in the refrigerator sub-application, which receives the information from the server. The registered information is also sent to the user's device.

[1190] Refrigerator management and recipe suggestions

[1191] Terminal (refrigerator):

[1192] The storage status of ingredients is detected using cameras and sensors inside the refrigerator, and that information is sent to a server.

[1193] server:

[1194] The system analyzes sensor and camera data received from the refrigerator and updates the information about ingredients in the refrigerator. It then generates recipes based on the updated information.

[1195] Terminal (user terminal):

[1196] The server receives the recipe suggestions and notifies the user. The user checks the recipe, takes out the necessary ingredients from the refrigerator, and starts cooking.

[1197] Collaboration with refrigerator manufacturers

[1198] server:

[1199] We provide refrigerator manufacturers with a common API to collect sensor and camera data from refrigerators, enabling accurate and real-time updates of food ingredients stored in refrigerators. We also monitor API usage and manage license revenue.

[1200] Terminal (refrigerator):

[1201] Information from inside the refrigerator is sent to the server via the API, and management information from the server is displayed inside the refrigerator.

[1202] Specific examples

[1203] Example of operation at time of purchase

[1204] 1. A user selects items (milk, butter, bread) at a supermarket and completes payment using an electronic payment app.

[1205] 2. The device sends the purchased product information to the server, which uses AI to determine whether the milk and butter need to be stored in the refrigerator.

[1206] 3. The registration information is sent to the refrigerator sub-app, and the user stores the purchased milk and butter in the refrigerator.

[1207] Example of refrigerator management and recipe suggestions

[1208] 1. The terminal (refrigerator) detects newly stored ingredients and sends the information to the server.

[1209] 2. The server updates the information about ingredients in the refrigerator and generates a recipe based on the ingredients you have.

[1210] 3. The generated recipe is sent to the user's device, and the user checks the recipe and cooks the ingredients.

[1211] This system allows users to automate the management of their purchases, reducing wasteful consumption and enabling effective meal planning. Furthermore, by collaborating with refrigerator manufacturers, it is possible to update information about ingredients in the refrigerator in real time.

[1212] The processing flow will be explained below.

[1213] Program processing steps

[1214] Receiving and determining purchase information

[1215] Step 1:

[1216] A user selects a product at a supermarket or online shop and completes the purchase process using an electronic payment app.

[1217] Step 2:

[1218] The device (smartphone, etc.) captures information about the purchased product (product name, category, quantity, etc.) and sends it to the server.

[1219] Step 3:

[1220] The server receives the purchased product information sent from the electronic payment application.

[1221] Step 4:

[1222] The server uses an AI model to analyze the received purchase information and determine whether it needs to be stored in a refrigerator.

[1223] For example, it determines that "milk" and "butter" need to be stored in the refrigerator, but "bread" does not.

[1224] Step 5:

[1225] The server generates a list of products that need to be stored in the refrigerator and issues an instruction to register the list in the refrigerator sub-app.

[1226] Step 6:

[1227] The terminal (smartphone, etc.) receives instructions from the server and registers the purchased items in the refrigerator sub-app.

[1228] After the terminal has completed registration of the purchased product information, it notifies the user.

[1229] Refrigerator management and recipe suggestions

[1230] Step 7:

[1231] The food purchased by the user is stored in the refrigerator.

[1232] Step 8:

[1233] The terminal (refrigerator) detects newly stored ingredients using its internal camera and sensors and sends this information to the server.

[1234] Step 9:

[1235] The server receives the data sent from the refrigerator sub-application and updates the information about ingredients in the refrigerator.

[1236] Step 10:

[1237] The server generates recipes based on the ingredients in the refrigerator.

[1238] For example, generate a recipe for "Omelette with butter and eggs."

[1239] Step 11:

[1240] The server transmits the generated recipe information to the user's terminal.

[1241] Step 12:

[1242] The terminal (smartphone, etc.) notifies the user of the recipe suggestions sent from the server.

[1243] Step 13:

[1244] The user checks the recipe, takes the necessary ingredients from the refrigerator, and starts cooking.

[1245] Collaboration with refrigerator manufacturers

[1246] Step 14:

[1247] The server provides a common API to refrigerator manufacturers.

[1248] Step 15:

[1249] The terminal (refrigerator) sends information from the interior camera and sensors to the server via API.

[1250] Step 16:

[1251] The server analyzes the data sent from the refrigerator and automatically updates the information about ingredients in the refrigerator.

[1252] Step 17:

[1253] The server monitors API usage and manages license revenue from refrigerator manufacturers.

[1254] This allows users to automate the management of their purchases and plan meals efficiently, while refrigerator manufacturers can also earn revenue from the use and integration of the system.

[1255] Example 1

[1256] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1257] In modern life, proper storage of purchased ingredients and their efficient use are important issues. In particular, there is a demand for a system that can automatically manage ingredient purchasing information, accurately update the information on ingredients in the refrigerator, and suggest recipes. However, with conventional technologies, it is difficult to efficiently link the management of purchased items with the updating of the information on ingredients in the refrigerator, which places a heavy burden on users.

[1258] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1259] In this invention, the server includes means for receiving purchased item information from the electronic payment app, means for determining using AI whether or not the purchased items need to be stored in the refrigerator, means for registering information on the purchased items determined to need to be stored in the refrigerator in the refrigerator sub-app, means for notifying the user terminal of the registered purchased item information, means for detecting food ingredient information using sensors and cameras inside the refrigerator and sending that information to the server, and means for updating the food ingredient information in the refrigerator based on the food ingredient information received by the server. This automates the management of purchased items and accurately updates the food ingredient information in the refrigerator in real time, enabling users to use ingredients efficiently and plan meals.

[1260] An "electronic payment app" is software that allows users to manage purchase information and make payments online or in physical stores.

[1261] "Purchased product information" is data including detailed information about products purchased by a user, such as product names, categories, and quantities.

[1262] An "AI model" is an artificial intelligence model that uses machine learning algorithms to analyze data and make predictions or decisions for specific tasks.

[1263] The "refrigerator sub-app" is an auxiliary application that manages information about ingredients in the refrigerator and notifies the user's device of this information.

[1264] A "user terminal" is an electronic device that can be directly operated by a user, such as a smartphone or tablet.

[1265] A "sensor" is a device that detects the state inside a refrigerator and transmits that data to the outside.

[1266] The "camera" is a photographic device for capturing image information of the inside of the refrigerator.

[1267] A "server" is a central system that provides data processing and storage functions and integrates and manages data from various devices.

[1268] A "recipe" is an instruction manual that lists the steps and ingredients needed for cooking based on the ingredients in the refrigerator.

[1269] A "database" is a system for efficiently storing and managing large amounts of data.

[1270] A "common API" is a common program interface that enables data exchange between different systems.

[1271] "License revenue" is revenue earned by providing third parties with the right to use specific technology or software.

[1272] The present invention provides a system that efficiently manages ingredients purchased by users, updates ingredient information in the refrigerator in real time, and suggests suitable recipes, allowing users to use purchased food without waste and plan meal preparations accordingly.

[1273] The main components of the system include:

[1274] 1. Electronic payment app: This is software that manages purchase information when a user purchases ingredients. When a user purchases ingredients at a supermarket or online shop and completes payment using an electronic payment app, purchase information is generated. Examples include SmartPay and PayPal.

[1275] 1. Server: Receives purchase information sent from the electronic payment app and uses an AI model (using, for example, TensorFlow or PyTorch) to determine whether the purchased items need to be stored in the refrigerator. The server also manages information about ingredients in the refrigerator and processes the data to provide users with appropriate recipes.

[1276] 1. Refrigerator sub-application: This application manages information about ingredients in the refrigerator and notifies the user. The sub-application receives information about products that need to be stored in the refrigerator from the server and notifies the user's device.

[1277] 1. Sensors and cameras: These are devices used to monitor the state inside the refrigerator and detect the presence of ingredients. Data from these devices is sent to the server and used to update the ingredient information. Examples include built-in image recognition cameras and temperature sensors.

[1278] 1. User terminal: An electronic device that can be directly operated by the user, such as a smartphone or tablet. This receives recipe suggestions sent from the server and notifies the user.

[1279] To illustrate, consider the following scenario:

[1280] 1. Receiving purchase information:

[1281] A user purchases food items such as milk, butter, and bread at a supermarket and completes the payment using an electronic payment app. The device then sends the purchased item information from the electronic payment app to the server.

[1282] 1. Refrigerator storage criteria:

[1283] The server uses an AI model to determine whether or not a product needs to be refrigerated based on the product information. For example, it may determine that milk and butter need to be refrigerated, while bread is recommended to be stored at room temperature.

[1284] 1. Register for the refrigerator sub-app:

[1285] The server sends the result of the judgment to the refrigerator sub-application, which notifies the user's device. The user then stores the milk and butter they bought in the refrigerator.

[1286] 1. Refrigerator management and recipe suggestions:

[1287] The device (refrigerator) uses a camera to recognize newly stored ingredients and sends the information to the server. The server updates the information about ingredients in the refrigerator and generates recipes based on the ingredients it has. For example, it generates a recipe for "cream stew" based on the latest list of ingredients in the refrigerator (milk, butter, eggs, cheese) and sends it to the user's device. The user's device notifies the user of the recipe, and the user begins cooking based on that information.

[1288] An example of a prompt to be input to the generative AI model is, "Please explain the processing flow of a system in which a user purchases milk, butter, and bread at the supermarket, determines the need to store them in the refrigerator based on that information, registers them in the refrigerator sub-app, and then manages the information about ingredients in the refrigerator and suggests recipes."

[1289] As a result, the present invention provides a system that is highly convenient for users, reduces food waste, and assists in efficient meal planning.

[1290] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1291] Processing Steps

[1292] Step 1: Obtain purchase information

[1293] User: Purchases ingredients at a supermarket or online shop and completes payment using an electronic payment app.

[1294] Input: Purchased product information (product name, category, quantity, etc.)

[1295] How it works: A user uses the SuperPay app to buy milk, butter, and bread.

[1296] Output: Purchase product information is generated from the electronic payment app and sent to the terminal.

[1297] Step 2: Submit your purchase information

[1298] Terminal: Sends purchased product information to the server.

[1299] Input: Purchase information obtained from the electronic payment app

[1300] Operation: The terminal sends purchased product information to the server.

[1301] Output: The server receives the purchased product information.

[1302] Step 3: Determine if refrigeration is necessary

[1303] Server: Uses AI models to determine whether purchased items need to be stored in the refrigerator.

[1304] Input: Purchase product information

[1305] How it works: The server uses TensorFlow to determine that milk and butter need to be stored in the refrigerator, but bread can be stored at room temperature.

[1306] Output: A list of products that need to be stored in the refrigerator and those that can be stored at room temperature

[1307] Step 4: Registering the refrigerator sub-app

[1308] Server: Sends information about products that need to be stored in the refrigerator to the refrigerator sub-app.

[1309] Input: List of products that need to be kept in the refrigerator

[1310] Operation: The server sends milk and butter information to the refrigerator sub-app, which then notifies the user's device.

[1311] Output: Product information registered in the refrigerator sub-app and notification to the user

[1312] Step 5: Get information about ingredients in the refrigerator

[1313] Terminal (refrigerator): Detects food ingredient information using sensors and cameras inside the refrigerator and sends that information to the server.

[1314] Input: Sensor data and camera data from inside the refrigerator

[1315] How it works: The refrigerator's camera recognizes newly placed ingredients (milk and butter), and the sensor captures temperature information.

[1316] Output: Sensor data and camera data are sent to the server.

[1317] Step 6: Update the ingredients in your refrigerator

[1318] Server: Updates the information about ingredients in the refrigerator based on the received sensor data and camera data.

[1319] Input: Sensor data and camera data

[1320] Operation: The server updates the ingredient information in the database to keep the ingredient list up to date.

[1321] Output: Updated ingredient information

[1322] Step 7: Recipe suggestions

[1323] Server: Generates recipes using AI based on updated ingredient information.

[1324] Input: Updated ingredient information

[1325] Operation: Generates a recipe for "cream stew" based on the ingredient information held by the server.

[1326] Output: The generated recipe

[1327] Step 8: Notify the user

[1328] Terminal (user terminal): Receives recipe suggestions sent from the server and notifies the user.

[1329] Input: Generated recipe

[1330] Operation: The user terminal notifies the user of the recipe.

[1331] Output: The user receives a recipe notification.

[1332] These steps allow users to automate their purchase management, update their refrigerator with ingredients, and suggest recipes, reducing food waste and enabling efficient meal planning.

[1333] (Application example 1)

[1334] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1335] Conventional refrigerator management systems require users to manually enter information about purchased products, which takes time and effort. They also lack a means to automatically determine whether purchased products need to be stored in the refrigerator and manage them appropriately. Furthermore, recipe suggestions based on refrigerator inventory information are not provided in real time, making it difficult to make effective use of ingredients.

[1336] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1337] In this invention, the server includes means for receiving purchased product information from the electronic payment app, means for determining using AI whether the purchased product needs to be stored in the refrigerator, means for registering information on the purchased product determined to need to be stored in the refrigerator in a refrigerator sub-app, means for notifying the user terminal of the registered purchased product information, and means for automatically reading the purchased product information at the physical store and means for updating inventory information in the refrigerator based on that information, managing ingredients, and suggesting recipes. This allows the user to automatically register information on purchased products in the refrigerator, allowing them to grasp inventory information in the refrigerator in real time and use ingredients efficiently.

[1338] An "electronic payment app" is an application that allows users to pay for goods electronically.

[1339] "Purchased product information" refers to information such as the name, category, and quantity of the product purchased by the user.

[1340] "AI" stands for artificial intelligence, a system that learns and reasons to solve specific problems.

[1341] The "refrigerator sub-app" is an application that manages refrigerator inventory information and provides the user with information on ingredients and recipe suggestions.

[1342] A "user terminal" is an information processing device operated by a user, such as a smartphone, tablet, or personal computer.

[1343] A "physical store" is a store where users can physically purchase products.

[1344] "Means for automatically reading information about purchased items" refers to technology that automatically obtains information about purchased items using sensors, cameras, etc.

[1345] "Inventory information" is information that indicates the current status of ingredients and products in the refrigerator.

[1346] "Recipe suggestion" refers to suggesting cooking methods to the user based on information about ingredients in the refrigerator.

[1347] "Sensor data" is digital data obtained from sensors regarding temperature, humidity, the presence of objects, etc.

[1348] "Camera data" refers to image and video data captured by a camera.

[1349] A "common API" is an application program interface that can be used commonly across multiple systems.

[1350] A "smart device" is an electronic device that can connect to the Internet and allows users to input and display information.

[1351] "Licensing revenue" is revenue earned by permitting the use of a certain technology or product.

[1352] This invention is a system that automatically acquires information about products purchased by a user, updates refrigerator inventory based on that information, and manages ingredients and suggests recipes. This system is configured using the following hardware and software.

[1353] Hardware

[1354] Smartphone: A device that allows users to purchase products and use electronic payment apps.

[1355] Smart glasses: devices that display real-time information while users are shopping in a physical store.

[1356] Refrigerator: A home appliance equipped with sensors and cameras inside that can obtain information about the food inside the refrigerator.

[1357] Server: A central processing unit that processes purchased product information and data from the refrigerator.

[1358] software

[1359] Electronic payment app: An application that allows users to purchase products and make payments.

[1360] Refrigerator sub-app: An application that manages the inventory information in the refrigerator and notifies the user.

[1361] Common API: An interface for receiving and analyzing sensor data and camera data from refrigerator manufacturers.

[1362] AI model: An artificial intelligence model that analyzes purchased product information and determines whether or not it needs to be stored in the refrigerator.

[1363] In this system, users first purchase products at a physical store and pay through an electronic payment app. Purchase information is automatically sent to a smartphone at the time of payment. This information is then sent to a server, where an AI model analyzes it and determines whether the product needs to be stored in a refrigerator.

[1364] Next, product information that needs to be stored in the refrigerator is registered in the refrigerator sub-app and simultaneously notified to the user's device. The user receives this notification and stores the necessary ingredients in the refrigerator.

[1365] The refrigerator uses internal sensors and cameras to detect newly stored ingredients. The detected data is sent to the server via a common API. The server uses this data to update the refrigerator's inventory and generate optimal recipes.

[1366] The generated recipe is sent to the user's device, and the user can start cooking based on the recipe. This allows the user to automatically register purchased product information in the refrigerator, keep track of inventory in the refrigerator in real time, and use ingredients efficiently.

[1367] Specific examples

[1368] For example, suppose a user purchases milk, butter, and bread at a physical store and completes the payment using an electronic payment app. At this time, the purchase information sent from the payment app is sent to a server, where an AI model analyzes it and determines that the milk and butter need to be stored in the refrigerator. This information is registered in the refrigerator sub-app, and the user checks the information on their smartphone. Next, sensors and cameras inside the refrigerator detect the newly placed ingredients and send the information to the server. The server analyzes the data and generates a recipe (e.g., salad, pasta, smoothie) based on the ingredients on hand, and notifies the user.

[1369] Prompt Sentence Examples

[1370] When a user purchases food at a physical store and completes the payment through a payment app on their smartphone, please explain the process by which the purchase information is registered in the refrigerator sub-app. Also, please explain how recipes are suggested based on the information about ingredients in the refrigerator.

[1371] In this way, the present invention greatly improves the user's daily life, allowing for efficient food management and cooking.

[1372] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1373] Step 1:

[1374] A user purchases an item in a physical store and pays using an electronic payment app.

[1375] Input: Information about the product purchased by the user (product name, category, quantity, etc.)

[1376] What happens: A user uses their smartphone to scan an item into a payment app and complete the payment.

[1377] Output: Purchase product information is generated and sent to your smartphone.

[1378] Step 2:

[1379] The smartphone sends the purchased product information to the server.

[1380] Input: Purchase information obtained from your smartphone

[1381] Specific operation: The smartphone obtains the purchased product information from the electronic payment app and sends it to the server as an HTTP request.

[1382] Output: Purchase information is sent to the server.

[1383] Step 3:

[1384] The server uses an AI model to determine whether refrigeration is necessary.

[1385] Input: Purchase information sent to the server

[1386] How it works: The server inputs purchased product information into the AI ​​model and determines whether or not the product needs to be stored in the refrigerator. The AI ​​model makes this determination based on the product name and category.

[1387] Output: A list of items that need to be kept in a refrigerator is generated.

[1388] Step 4:

[1389] The server registers product information that needs to be stored in the refrigerator in the refrigerator sub-app.

[1390] Input: List of products that require refrigeration as determined by the server

[1391] Specific operation: The server sends the registration information to the refrigerator sub-app via API.

[1392] Output: Product information that needs to be stored is registered in the refrigerator sub-app.

[1393] Step 5:

[1394] The refrigerator sub-app notifies the user device of the registration information

[1395] Input: Product information registered in the refrigerator sub-app

[1396] Specific operation: The refrigerator sub-app notifies the user device of the information received from the server. The information is sent to the user using push notifications, etc.

[1397] Output: The registration information for the purchased product is displayed on the user's device.

[1398] Step 6:

[1399] The user places the product in the refrigerator

[1400] Input: Registration information displayed on the user's device

[1401] Specific operation: The user places the purchased items in the refrigerator. The product placement is performed while confirming the registration information.

[1402] Output: The product is stored in the refrigerator.

[1403] Step 7:

[1404] The refrigerator detects the food stored inside using internal sensors and cameras.

[1405] Input: New item placed in refrigerator

[1406] How it works: The refrigerator uses internal sensors and cameras to detect newly added ingredients.

[1407] Output: The detected ingredients information is generated.

[1408] Step 8:

[1409] The refrigerator sends the food information it detects to the server.

[1410] Input: Detected ingredient information

[1411] Specific operation: The refrigerator sends ingredient information to the server using a common API.

[1412] Output: Ingredient information is sent to the server.

[1413] Step 9:

[1414] The server updates the ingredient information and generates the optimal recipe.

[1415] Input: Ingredient information sent to the server

[1416] Specific operation: The server updates the inventory data in the refrigerator based on the received ingredient information and generates the optimal recipe using a generative AI model.

[1417] Output: The optimal recipe information is generated.

[1418] Step 10:

[1419] The server sends the generated recipe to the user's device.

[1420] Input: Generated recipe information

[1421] Specific operation: The server sends the generated recipe information to the user's device. The recipe information is sent to the user using push notifications, etc.

[1422] Output: Recipe information is displayed on the user's device.

[1423] Through the above processing steps, the user can automatically register information about purchased items in the refrigerator, grasp inventory information in the refrigerator in real time, and use ingredients efficiently.

[1424] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1425] 1. System Overview

[1426] The system of this invention receives information about products purchased by users through an electronic payment app, and based on that information, AI determines whether the product needs to be stored in the refrigerator and automatically registers that information in the refrigerator sub-app. It also includes a function to manage information about ingredients in the refrigerator and suggest recipes to users based on that information. This system works in conjunction with refrigerator manufacturers, acquiring sensor and camera data through a common API to update information about ingredients in the refrigerator in real time. Furthermore, by combining it with an emotion engine, recipes are personalized based on the user's emotions, improving the user experience.

[1427] 2. System Configuration and Operation

[1428] Receiving purchase information

[1429] User:

[1430] When a user purchases ingredients at a supermarket or online shop and pays using an electronic payment app, purchased product information is generated.

[1431] Device:

[1432] Purchase information (product name, category, quantity, etc.) is sent from the electronic payment app to the server.

[1433] server:

[1434] The server receives the purchased product information and uses an AI model to determine whether it needs to be stored in a refrigerator.

[1435] Determining whether it can be stored in a refrigerator

[1436] server:

[1437] The AI ​​model analyzes the purchased product information and determines whether it needs to be stored in a refrigerator. For example, milk and butter are determined to be products that need to be stored in a refrigerator.

[1438] Device:

[1439] The information about products that need to be stored in the refrigerator is registered in the refrigerator sub-application, which receives the information from the server. The registered information is also sent to the user's device.

[1440] Refrigerator management and recipe suggestions

[1441] Terminal (refrigerator):

[1442] The storage status of ingredients is detected using cameras and sensors inside the refrigerator, and that information is sent to a server.

[1443] server:

[1444] The system analyzes sensor and camera data received from the refrigerator and updates the information about ingredients in the refrigerator. It then generates recipes based on the updated information.

[1445] Terminal (user terminal):

[1446] The server receives the recipe suggestions and notifies the user. The user checks the recipe, takes out the necessary ingredients from the refrigerator, and starts cooking.

[1447] Collaboration with refrigerator manufacturers

[1448] server:

[1449] We provide refrigerator manufacturers with a common API to collect sensor and camera data from refrigerators, enabling accurate and real-time updates of food ingredients stored in refrigerators. We also monitor API usage and manage license revenue.

[1450] Terminal (refrigerator):

[1451] Information from inside the refrigerator is sent to the server via the API, and management information from the server is displayed inside the refrigerator.

[1452] Incorporating an emotion engine

[1453] User Emotion Recognition and Recipe Personalization

[1454] Device (smartphone, etc.):

[1455] It incorporates an emotion engine that analyzes emotions through facial recognition and voice data of the user. For example, it uses a camera and microphone to analyze facial expressions and voices while the user is operating the device.

[1456] server:

[1457] It receives emotional data sent from the emotion engine and personalizes recipes based on that data, for example, suggesting easy-to-make recipes if the user is tired.

[1458] Terminal (user terminal):

[1459] It notifies users of personalized recipes and makes meal suggestions based on the user's emotions.

[1460] Emotion-based food consumption and management

[1461] server:

[1462] The system continuously monitors the user's emotional data and suggests food consumption and management based on their emotions. For example, if the user is feeling stressed, it will suggest recipes with ingredients that are effective in reducing stress.

[1463] Device (smartphone, etc.):

[1464] It notifies users in a timely manner and provides alerts about food expiration dates and conditions based on emotional data.

[1465] Specific examples

[1466] Example of operation at time of purchase

[1467] 1. A user selects items (milk, butter, bread) at a supermarket and completes the purchase using an electronic payment app.

[1468] 2. The device sends the purchased product information to the server, which uses AI to determine whether the milk and butter need to be stored in the refrigerator.

[1469] 3. The registration information is sent to the refrigerator sub-app, and the user stores the purchased milk and butter in the refrigerator.

[1470] Example of refrigerator management and recipe suggestions

[1471] 1. The terminal (refrigerator) detects newly stored ingredients and sends the information to the server.

[1472] 2. The server updates the information about ingredients in the refrigerator and generates a recipe based on the ingredients you have.

[1473] 3. The generated recipe is sent to the user's device, and the user checks the recipe and cooks the ingredients.

[1474] Example of Emotion Engine in Action

[1475] 1. The device (smartphone, etc.) collects the user's facial recognition and voice data and analyzes it using an emotion engine.

[1476] 2. Based on the emotional data, the server suggests recipes using ingredients that have a relaxing effect, for example, if the user feels like relaxing.

[1477] 3. The user checks the appropriate recipe and uses it to cook.

[1478] This system allows users to automate their purchase management, reducing wasteful consumption and creating effective meal plans. Furthermore, the incorporation of an emotion engine enables personalized meal suggestions that are in line with the user's emotions, which is expected to improve the user experience.

[1479] The processing flow will be explained below.

[1480] Program processing steps

[1481] Receiving and determining purchase information

[1482] Step 1:

[1483] A user selects a product at a supermarket or online shop and completes the purchase process using an electronic payment app.

[1484] Step 2:

[1485] The device (smartphone, etc.) captures information about the purchased product (product name, category, quantity, etc.) and sends it to the server.

[1486] Step 3:

[1487] The server receives the purchased product information sent from the electronic payment application.

[1488] Step 4:

[1489] The server uses an AI model to analyze the received purchase information and determine whether it needs to be stored in a refrigerator.

[1490] For example, it determines that "milk" and "butter" need to be stored in the refrigerator, but "bread" does not.

[1491] Step 5:

[1492] The server generates a list of products that need to be stored in the refrigerator and issues an instruction to register the list in the refrigerator sub-app.

[1493] Step 6:

[1494] The terminal (smartphone, etc.) receives instructions from the server and registers the purchased items in the refrigerator sub-app.

[1495] After the terminal has completed registration of the purchased product information, it notifies the user.

[1496] Refrigerator management and recipe suggestions

[1497] Step 7:

[1498] The food purchased by the user is stored in the refrigerator.

[1499] Step 8:

[1500] The terminal (refrigerator) detects newly stored ingredients using its internal camera and sensors and sends this information to the server.

[1501] Step 9:

[1502] The server receives the data sent from the refrigerator sub-application and updates the information about ingredients in the refrigerator.

[1503] Step 10:

[1504] The server generates recipes based on the ingredients in the refrigerator.

[1505] For example, generate a recipe for "Omelette with butter and eggs."

[1506] Step 11:

[1507] The server transmits the generated recipe information to the user's terminal.

[1508] Step 12:

[1509] The terminal (smartphone, etc.) notifies the user of the recipe suggestions sent from the server.

[1510] Step 13:

[1511] The user checks the recipe, takes the necessary ingredients from the refrigerator, and starts cooking.

[1512] Collaboration with refrigerator manufacturers

[1513] Step 14:

[1514] The server provides a common API to refrigerator manufacturers.

[1515] Step 15:

[1516] The terminal (refrigerator) sends information from the interior camera and sensors to the server via API.

[1517] Step 16:

[1518] The server analyzes the data sent from the refrigerator and automatically updates the information about ingredients in the refrigerator.

[1519] Step 17:

[1520] The server monitors API usage and manages license revenue from refrigerator manufacturers.

[1521] Incorporating an emotion engine

[1522] Step 18:

[1523] The device (smartphone, etc.) collects the user's facial recognition and voice data and analyzes it using an emotion engine.

[1524] Step 19:

[1525] The server uses emotional data to suggest easy-to-make recipes, for example, if the user feels tired.

[1526] Step 20:

[1527] The device (smartphone, etc.) notifies the user of the personalized recipe.

[1528] Step 21:

[1529] The user checks the recipe based on their emotions and uses the recipe to cook.

[1530] Step 22:

[1531] The server continuously monitors emotional data and suggests recipes using ingredients that are effective in reducing stress.

[1532] Step 23:

[1533] The device (smartphone, etc.) notifies the user in a timely manner and provides alerts regarding the expiration date and condition of ingredients based on emotional data.

[1534] This allows users to automate the management of their purchases and create efficient meal plans, and the incorporation of an emotion engine makes it possible to make personalized meal suggestions that are in line with the user's emotions, which is expected to improve the user experience.

[1535] Example 2

[1536] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1537] Currently, there are many grocery management systems and recipe recommendation systems on the market, but these systems have limitations in terms of centralized management and personalized recommendations. In particular, in terms of refrigerator ingredient management and recipe recommendations based on the user's mood, existing systems lack flexibility and do not sufficiently improve the user experience. Furthermore, they lack automated mechanisms for reducing food waste and efficiently planning meals. This invention aims to solve these problems and improve the quality of life of users.

[1538] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1539] In this invention, the server includes: means for receiving purchased product information from an electronic payment app; means for determining whether a purchased product requires refrigeration using AI; and means for registering information about purchased products determined to require refrigeration in a refrigeration appliance app. This allows for efficient management of purchased product information and enables appropriate storage based on the information. The server also includes means for updating food product information stored in the refrigeration appliance based on data from the refrigeration appliance app; means for generating recipes based on the food product information stored in the refrigeration appliance; and means for transmitting the generated recipes to a user terminal. This allows for real-time updates of food product information stored in the refrigeration appliance, and for suggesting recipes suitable for the user based on the updated information. The server also includes means for providing a common API to refrigeration appliance manufacturers; means for receiving sensor data and image data from the refrigeration appliance; means for analyzing the data and updating the food product information stored in the refrigeration appliance; and means for monitoring the usage of the common API and managing license revenue. This allows for efficient collection and management of refrigeration appliance data, enabling appropriate food product management and revenue generation. The server also includes means for recognizing user emotions; means for personalizing recipes based on the emotion data; and means for notifying the user terminal of the personalized recipes. This allows for personalized meal suggestions based on the user's emotions, further improving the user experience.

[1540] 1. An "electronic payment app" is a software application that a user uses to purchase goods online or in-store, and that digitizes the payment process.

[1541] 2. "Purchased Product Information" refers to detailed data of the product purchased by the user, including product name, category, quantity, price, etc.

[1542] 3. "AI" stands for artificial intelligence, which refers to algorithms and models that analyze data and perform specific tasks automatically.

[1543] 4. "Refrigeration equipment app" is software that supports the operation of refrigeration equipment, managing food ingredient information and monitoring storage conditions.

[1544] 5. "User terminal" refers to a device used by a user, such as a smartphone, tablet, PC, or other electronic device.

[1545] 6. "Sensor Data" means information detected by sensors, such as temperature, humidity, and location within refrigeration equipment.

[1546] 7. "Image data" refers to image files taken by a camera placed inside the refrigeration equipment, which are used to visually record the condition and placement of ingredients.

[1547] 8. "Common API" means a common application program interface that can be used by multiple refrigeration equipment manufacturers, enabling data transmission and reception and the sharing of functions.

[1548] 9. "License Revenue" means revenue received from refrigeration equipment manufacturers for use of the provided common API, including API usage fees and service fees.

[1549] 10. "Emotional Data" means information about a user's emotional state obtained through facial recognition or voice analysis, and is data indicating joy, sadness, fatigue, stress, etc.

[1550] 11. "Personalized Cooking Recipes" are cooking recipes that are specifically tailored to a user's individual emotional state and preferences and that are provided as part of meal suggestions to the user.

[1551] 1. System Overview

[1552] The system of this invention receives information about products purchased by users through an electronic payment app, and based on that information, AI determines the need for refrigeration and automatically registers that information in the refrigeration appliance app. It also includes a function to manage food information stored in the refrigeration appliance and suggest recipes to users based on that information. This system works in conjunction with refrigeration appliance manufacturers, acquiring sensor data and image data through a common API to update the food information stored in the refrigeration appliance in real time. Furthermore, by incorporating an emotion engine, recipes are personalized based on the user's emotions, improving the user experience.

[1553] 2. System Configuration and Operation

[1554] Receiving purchase information

[1555] When a user purchases groceries at a supermarket or online shop and pays using an electronic payment app, purchased product information (product name, category, quantity, etc.) is generated within the electronic payment app.

[1556] The terminal (electronic payment app) sends the purchased product information to the server. For example, it is sent to the cloud server in the form of an electronic receipt.

[1557] The server receives the purchased product information and records it in a database.

[1558] Determining whether refrigerated storage is possible

[1559] The server uses AI models to determine the need for refrigeration based on the purchased product information received, for example, milk and perishables.

[1560] The device registers the information about the products that need to be stored sent from the server in the refrigeration equipment app and notifies the user's device, allowing the user to store their purchased items appropriately.

[1561] Refrigeration equipment management and recipe suggestions

[1562] The terminal (refrigeration equipment) uses internal sensors and cameras to monitor the condition of the food and sends the data to a server.

[1563] The server analyzes the received sensor data and image data and updates the information in the refrigeration equipment, for example, identifying the type and quantity of ingredients.

[1564] The server generates usable recipes based on updated ingredient information, taking into account nutritional value and user preferences.

[1565] The terminal (user terminal) receives the recipe suggestions sent from the server and notifies the user, who can then check the recipes and prepare the suggested meals.

[1566] Collaboration with refrigeration equipment manufacturers

[1567] The server provides a common API to refrigeration equipment manufacturers and acquires sensor data and image data, for example, to manage fresh vegetables and dairy products.

[1568] The terminal (refrigeration equipment) sends data to the server via API, which updates the information in the refrigeration equipment in real time, allowing users to manage their ingredients appropriately.

[1569] Specific examples

[1570] Example of operation at time of purchase

[1571] 1. A user selects milk, butter, and bread at a supermarket and completes the purchase using an electronic payment app.

[1572] 2. The terminal (electronic payment app) sends the purchased product information to the server.

[1573] 3. The server uses AI to determine that the milk and butter need to be refrigerated.

[1574] 4. The registration information is sent to the refrigerator app, and the user places the milk and butter in the refrigerator.

[1575] Example of refrigeration equipment management and recipe suggestions

[1576] 1. The terminal (refrigeration equipment) detects newly stored food items and sends the information to the server.

[1577] 2. The server updates the food information in the refrigerator and generates a recipe based on the ingredients it has.

[1578] 3. The generated recipe is sent to the user's device, and the user checks the recipe and cooks the ingredients.

[1579] Example of Emotion Engine in Action

[1580] 1. The device (smartphone, etc.) collects the user's facial recognition and voice data and analyzes it using an emotion engine.

[1581] 2. Based on the emotional data, the server suggests recipes using ingredients that have a relaxing effect if the user feels like relaxing.

[1582] 3. The user checks the appropriate recipe and uses it to cook.

[1583] Example prompts for generative AI models

[1584] Receiving purchase information

[1585] Prompt: Receive information about the milk, butter, and bread purchased by a user at the supermarket from an electronic payment app, and use AI to determine whether they need to be stored in the refrigerator.

[1586] Refrigeration equipment management and recipe suggestions

[1587] Prompt: Send the newly stored ingredients to the server, update the ingredients in the refrigerator, and suggest a recipe.

[1588] Emotion Engine Operation

[1589] Prompt: Collect facial and voice data from the user and suggest recipes suitable for when the user wants to relax.

[1590] By using this system, users can automate the management of their purchases, reduce wasteful consumption, and create effective meal plans. In addition, the incorporation of an emotion engine enables personalized meal suggestions that are in line with the user's emotions, which is expected to improve the user experience.

[1591] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1592] Step 1:

[1593] A user purchases groceries at a supermarket or online shop and pays using an electronic payment app. At this time, product information (product name, category, quantity, etc.) is generated within the electronic payment app. The user inputs their intention to purchase a product, and as a result, product information is generated.

[1594] Step 2:

[1595] The terminal (electronic payment app) sends purchased product information to the server. For example, the purchase information is sent to the cloud server in the form of an electronic receipt. The terminal receives purchased product information as input and sends it to the server as output.

[1596] Step 3:

[1597] The server receives the purchased product information and records it in the database. The server analyzes the purchased product information received as input and saves it in the database. This records the purchased product information.

[1598] Step 4:

[1599] The server uses an AI model to determine the need for refrigerated storage based on the received purchase information. Purchase information is provided to the AI ​​model as input, and the AI ​​analyzes the data to determine the need for refrigerated storage. For example, it may determine that milk or fresh food needs to be stored in a refrigerator.

[1600] Step 5:

[1601] The terminal registers the information about the products that require storage sent from the server in the refrigeration equipment app and notifies the user terminal.The terminal receives information about the products that require refrigeration storage from the server as input, processes it as output to be registered in the refrigeration equipment app, and notifies the user terminal.

[1602] Step 6:

[1603] The terminal (refrigeration equipment) monitors the condition of food using internal sensors and cameras and sends the data to a server. The refrigeration equipment's sensors and cameras acquire food condition data as input and send it to the server as output. For example, sensor data on temperature and humidity, and image data of food.

[1604] Step 7:

[1605] The server analyzes the received sensor data and image data and updates the food ingredient information in the refrigerator. The server receives sensor data and image data as input, analyzes them to identify the type and amount of food ingredients, and updates the food ingredient information in the refrigerator as output.

[1606] Step 8:

[1607] The server generates usable recipes based on updated ingredient information, taking into account nutritional value and user preferences. Using the latest ingredient information as input, the AI ​​model analyzes the data and generates suitable recipes as output.

[1608] Step 9:

[1609] The terminal (user terminal) receives the recipe suggestions sent from the server and notifies the user. It receives the recipe information provided by the server as input and suggests it to the user using the notification function as output. The user checks this recipe and prepares the suggested meal.

[1610] Step 10:

[1611] The device (such as a smartphone) collects the user's facial recognition and voice data and analyzes it using an emotion engine. The user's facial recognition and voice data are taken as input, and the output is analyzed by the emotion engine. For example, data collection using a camera or microphone.

[1612] Step 11:

[1613] The server analyzes the emotional data and suggests recipes suitable for when the user wants to relax or when they are tired. The server receives the emotional data as input, analyzes it with an AI model, and generates suitable recipes as output.

[1614] Step 12:

[1615] The user checks the personalized recipe notified to the device and uses that recipe to cook. The notified recipe information is received as input, and cooking is performed based on that recipe as output. The user's cooking action is the final execution stage.

[1616] (Application example 2)

[1617] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1618] In conventional refrigerator and purchasing information management systems, storing and managing purchased products is often done manually, which is cumbersome for users and makes it difficult to effectively manage ingredients. Furthermore, there is a lack of personalized recipe suggestions based on the user's emotions and condition, which hinders the improvement of the user experience. There is a need for an efficient and user-friendly system that can solve these problems.

[1619] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1620] In this invention, the server includes means for receiving purchased product information from the electronic payment app, means for determining using AI whether the purchased product needs to be stored in the refrigerator, means for registering information on the purchased product determined to need to be stored in the refrigerator in a refrigerator sub-app, means for notifying the user terminal of the registered purchased product information, means for automatically acquiring purchase information from a physical store and linking with the refrigerator, means for personalizing recipes based on the user's emotions using an emotion engine, and means for analyzing the user's emotion data and suggesting appropriate ingredient consumption and recipes. This automates the storage and management of purchased products and makes it possible to realize personalized ingredient consumption and recipe suggestions based on the user's emotions.

[1621] An "electronic payment app" is a software application that receives purchase information electronically and completes payments.

[1622] "Purchased product information" is data including detailed information such as the name, category, and quantity of the product purchased by the user.

[1623] "AI" refers to artificial intelligence systems that automate specific tasks, analyze data, and make decisions.

[1624] The "refrigerator sub-app" is an auxiliary application for managing and updating information about ingredients in the refrigerator.

[1625] A "user terminal" is an electronic device operated by a user, such as a smartphone or tablet.

[1626] A "physical store" is a physical sales location where products are actually displayed and sold.

[1627] An "emotion engine" is software that analyzes a user's emotions and makes appropriate suggestions based on their state.

[1628] A "recipe" is a document that describes the steps and ingredients for making a particular dish.

[1629] A "common API" is a standardized interface that allows different systems to communicate with each other and exchange data.

[1630] "Sensor data" is information obtained from sensors that detect the temperature, humidity, and presence of objects inside the refrigerator.

[1631] "Camera data" refers to data acquired as video or images of the inside of a refrigerator.

[1632] A "generative AI model" is an artificial intelligence model designed to generate specific outputs based on user input.

[1633] A "prompt sentence" is an input sentence that a generative AI model uses to generate appropriate output.

[1634] 1. System Overview

[1635] This invention is a system that receives information about products purchased by users through an electronic payment app, and then uses AI to determine whether or not to store the products in the refrigerator, automatically registering them in a refrigerator sub-app. It also includes a function to manage information about ingredients in the refrigerator and suggest recipes based on that information, allowing recipes to be personalized based on the user's emotions.

[1636] 2. Program Generation

[1637] The program to realize this system is as follows, but the details of the code are not included. The specific main functions are as follows:

[1638] Receiving purchase information: The user sends product information from the electronic payment app.

[1639] Refrigerated storage decision: AI analyzes product information and determines the need for refrigerated storage.

[1640] Information registration: Register the necessary product information in the refrigerator sub-app.

[1641] User notification: Registration information is sent to the user's device.

[1642] Sentiment Analysis: The emotion engine analyzes user emotions and personalizes recipes.

[1643] Recipe suggestion: Generates recipes based on emotion data and notifies the user.

[1644] 3. Processing Description

[1645] Receiving purchase information

[1646] When a user purchases a product through an electronic payment app, purchased product information (product name, category, quantity, etc.) is generated. The server receives and analyzes this information.

[1647] Refrigerator storage judgment

[1648] The server uses an AI model to analyze the received purchase information and determine which products need to be stored in the refrigerator. The necessary product information is then registered in the refrigerator sub-app.

[1649] Physical store integration and continuous processing

[1650] Information about products purchased at physical stores is automatically sent to a server via the electronic payment app, which links purchase information with information in the refrigerator, reducing the burden on users.

[1651] Refrigerator management

[1652] Sensor and camera data from inside the refrigerator is sent to the server via a common API, which then analyzes the data and updates the information about ingredients in the refrigerator in real time.

[1653] Emotional engine and recipe suggestions

[1654] The emotion engine analyzes the user's facial expressions and voice to generate emotion data. Based on this, the server suggests recipes appropriate for the user's condition. For example, if the user is tired, it will suggest easy-to-make recipes.

[1655] 4. Usage example

[1656] Recipe suggestions based on refrigerator status data and user emotion data

[1657] Current refrigerator data:

[1658] Milk: 1 bottle

[1659] Bread: 2 pieces

[1660] User sentiment data:

[1661] Facial Expression: Tired

[1662] Audio Tone: Low

[1663] Prompt Sentence Examples

[1664] The system suggests suitable recipes based on the current state of the refrigerator and the user's emotional data.

[1665] For example, generate a recipe with the following conditions:

[1666] Refrigerator Data:

[1667] Milk: 1 bottle

[1668] Bread: 2 pieces

[1669] User sentiment data:

[1670] Facial Expression: Tired

[1671] Audio Tone: Low

[1672] Generated recipe example:

[1673] 1. Easy pasta recipes

[1674] 2. Relaxing Salad Recipes

[1675] As described above, the system receives information about purchased items and provides consistent support, from managing items in the refrigerator to suggesting recipes based on the user's emotions. This reduces the burden on the user and provides a personalized experience.

[1676] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1677] Step 1:

[1678] When a user purchases a product through an electronic payment app, purchased product information (product name, category, quantity, etc.) is generated. The user purchases ingredients at a supermarket or online shop and completes the payment through the electronic payment app. At this time, purchased product information is automatically generated.

[1679] Input: Purchased product information (product name, category, quantity, etc.)

[1680] Output: Purchase product information data

[1681] Step 2:

[1682] The terminal sends the purchased product information to the server. The purchased product information is sent from the electronic payment app to the server.

[1683] Input: Purchase product information data

[1684] Output: Purchased product information sent to the server

[1685] Step 3:

[1686] The server uses an AI model to analyze the received purchase information and determine whether or not the product needs to be stored in a refrigerator. For example, it identifies products that need to be refrigerated, such as milk and butter.

[1687] Input: Purchase information sent to the server

[1688] Processing: AI model determines whether refrigeration is necessary

[1689] Output: List of items that need to be stored

[1690] Step 4:

[1691] The server registers product information that needs to be stored in the refrigerator sub-app, which then automatically records these products.

[1692] Input: List of items that need to be stored

[1693] Output: Product information registered in the refrigerator sub-app

[1694] Step 5:

[1695] The server notifies the user of the registration information on their smartphone or other device, allowing them to check the products they have purchased.

[1696] Input: Product information registered in the refrigerator sub-app

[1697] Output: Notification to user terminal

[1698] Step 6:

[1699] The emotion engine is used to analyze the user's emotions. Facial expressions and voice data are acquired from the user's smartphone or tablet, and the emotion engine analyzes them.

[1700] Input: User facial expressions and voice data

[1701] Processing: Sentiment analysis using the emotion engine

[1702] Output: User emotion data

[1703] Step 7:

[1704] The server personalizes recipes based on the user's emotional data: if the user is tired, it suggests simple recipes, and if the user is relaxed, it suggests more elaborate recipes.

[1705] Input: User emotion data, information about ingredients in the refrigerator

[1706] Processing: Recipe generation based on sentiment data

[1707] Output: personalized recipe

[1708] Step 8:

[1709] The server sends personalized recipes to the user's device, which then notifies the user of the recipes on their smartphone or other device, allowing the user to check the appropriate recipes.

[1710] Input: Personalized Recipe

[1711] Output: Recipe notification to user's device

[1712] Examples of specific examples and prompts

[1713] Usage example

[1714] The system suggests suitable recipes based on the current state of the refrigerator and the user's emotional data.

[1715] For example, generate a recipe with the following conditions:

[1716] Refrigerator Data:

[1717] Milk: 1 bottle

[1718] Bread: 2 pieces

[1719] User sentiment data:

[1720] Facial Expression: Tired

[1721] Audio Tone: Low

[1722] Generated recipe example:

[1723] 1. Easy pasta recipes

[1724] 2. Relaxing Salad Recipes

[1725] The above is a specific embodiment of the invention, which allows users to store and manage their purchases automatically and provides a personalized experience based on emotions.

[1726] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1727] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1728] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1729] [Fourth embodiment]

[1730] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1731] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1732] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1733] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1734] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1735] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1736] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1737] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1738] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1739] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1740] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1741] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1742] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1743] 1. System Overview

[1744] The system of this invention receives information about products purchased by users through an electronic payment app, and based on that information, AI determines whether the product needs to be stored in the refrigerator and automatically registers that information in the refrigerator sub-app. It also includes a function to manage information about ingredients in the refrigerator and suggest recipes to users based on that information. This system works in conjunction with refrigerator manufacturers, acquiring sensor and camera data through a common API, allowing it to update information about ingredients in the refrigerator in real time.

[1745] 2. System Configuration and Operation

[1746] Receiving purchase information

[1747] User:

[1748] When a user purchases ingredients at a supermarket or online shop and pays using an electronic payment app, purchased product information is generated.

[1749] Device:

[1750] Purchase information (product name, category, quantity, etc.) is sent from the electronic payment app to the server.

[1751] server:

[1752] The server receives the purchased product information and uses an AI model to determine whether it needs to be stored in a refrigerator.

[1753] Determining whether it can be stored in a refrigerator

[1754] server:

[1755] The AI ​​model analyzes the purchased product information and determines whether it needs to be stored in a refrigerator. For example, it determines that milk and butter are products that need to be stored in a refrigerator.

[1756] Device:

[1757] The information about products that need to be stored in the refrigerator is registered in the refrigerator sub-application, which receives the information from the server. The registered information is also sent to the user's device.

[1758] Refrigerator management and recipe suggestions

[1759] Terminal (refrigerator):

[1760] The storage status of ingredients is detected using cameras and sensors inside the refrigerator, and that information is sent to a server.

[1761] server:

[1762] The system analyzes sensor and camera data received from the refrigerator and updates the information about ingredients in the refrigerator. It then generates recipes based on the updated information.

[1763] Terminal (user terminal):

[1764] The server receives the recipe suggestions and notifies the user. The user checks the recipe, takes out the necessary ingredients from the refrigerator, and starts cooking.

[1765] Collaboration with refrigerator manufacturers

[1766] server:

[1767] We provide refrigerator manufacturers with a common API to collect sensor and camera data from refrigerators, enabling accurate and real-time updates of food ingredients stored in refrigerators. We also monitor API usage and manage license revenue.

[1768] Terminal (refrigerator):

[1769] Information from inside the refrigerator is sent to the server via the API, and management information from the server is displayed inside the refrigerator.

[1770] Specific examples

[1771] Example of operation at time of purchase

[1772] 1. A user selects items (milk, butter, bread) at a supermarket and completes payment using an electronic payment app.

[1773] 2. The device sends the purchased product information to the server, which uses AI to determine whether the milk and butter need to be stored in the refrigerator.

[1774] 3. The registration information is sent to the refrigerator sub-app, and the user stores the purchased milk and butter in the refrigerator.

[1775] Example of refrigerator management and recipe suggestions

[1776] 1. The terminal (refrigerator) detects newly stored ingredients and sends the information to the server.

[1777] 2. The server updates the information about ingredients in the refrigerator and generates a recipe based on the ingredients you have.

[1778] 3. The generated recipe is sent to the user's device, and the user checks the recipe and cooks the ingredients.

[1779] This system allows users to automate the management of their purchases, reducing wasteful consumption and enabling effective meal planning. Furthermore, by collaborating with refrigerator manufacturers, it is possible to update information about ingredients in the refrigerator in real time.

[1780] The processing flow will be explained below.

[1781] Program processing steps

[1782] Receiving and determining purchase information

[1783] Step 1:

[1784] A user selects a product at a supermarket or online shop and completes the purchase process using an electronic payment app.

[1785] Step 2:

[1786] The device (smartphone, etc.) captures information about the purchased product (product name, category, quantity, etc.) and sends it to the server.

[1787] Step 3:

[1788] The server receives the purchased product information sent from the electronic payment application.

[1789] Step 4:

[1790] The server uses an AI model to analyze the received purchase information and determine whether it needs to be stored in a refrigerator.

[1791] For example, it determines that "milk" and "butter" need to be stored in the refrigerator, but "bread" does not.

[1792] Step 5:

[1793] The server generates a list of products that need to be stored in the refrigerator and issues an instruction to register the list in the refrigerator sub-app.

[1794] Step 6:

[1795] The terminal (smartphone, etc.) receives instructions from the server and registers the purchased items in the refrigerator sub-app.

[1796] After the terminal has completed registration of the purchased product information, it notifies the user.

[1797] Refrigerator management and recipe suggestions

[1798] Step 7:

[1799] The food purchased by the user is stored in the refrigerator.

[1800] Step 8:

[1801] The terminal (refrigerator) detects newly stored ingredients using its internal camera and sensors and sends this information to the server.

[1802] Step 9:

[1803] The server receives the data sent from the refrigerator sub-application and updates the information about ingredients in the refrigerator.

[1804] Step 10:

[1805] The server generates recipes based on the ingredients in the refrigerator.

[1806] For example, generate a recipe for "Omelette with butter and eggs."

[1807] Step 11:

[1808] The server transmits the generated recipe information to the user's terminal.

[1809] Step 12:

[1810] The terminal (smartphone, etc.) notifies the user of the recipe suggestions sent from the server.

[1811] Step 13:

[1812] The user checks the recipe, takes the necessary ingredients from the refrigerator, and starts cooking.

[1813] Collaboration with refrigerator manufacturers

[1814] Step 14:

[1815] The server provides a common API to refrigerator manufacturers.

[1816] Step 15:

[1817] The terminal (refrigerator) sends information from the interior camera and sensors to the server via API.

[1818] Step 16:

[1819] The server analyzes the data sent from the refrigerator and automatically updates the information about ingredients in the refrigerator.

[1820] Step 17:

[1821] The server monitors API usage and manages license revenue from refrigerator manufacturers.

[1822] This allows users to automate the management of their purchases and plan meals efficiently, while refrigerator manufacturers can also earn revenue from the use and integration of the system.

[1823] Example 1

[1824] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1825] In modern life, proper storage of purchased ingredients and their efficient use are important issues. In particular, there is a demand for a system that can automatically manage ingredient purchasing information, accurately update the information on ingredients in the refrigerator, and suggest recipes. However, with conventional technologies, it is difficult to efficiently link the management of purchased items with the updating of the information on ingredients in the refrigerator, which places a heavy burden on users.

[1826] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1827] In this invention, the server includes means for receiving purchased item information from the electronic payment app, means for determining using AI whether or not the purchased items need to be stored in the refrigerator, means for registering information on the purchased items determined to need to be stored in the refrigerator in the refrigerator sub-app, means for notifying the user terminal of the registered purchased item information, means for detecting food ingredient information using sensors and cameras inside the refrigerator and sending that information to the server, and means for updating the food ingredient information in the refrigerator based on the food ingredient information received by the server. This automates the management of purchased items and accurately updates the food ingredient information in the refrigerator in real time, enabling users to use ingredients efficiently and plan meals.

[1828] An "electronic payment app" is software that allows users to manage purchase information and make payments online or in physical stores.

[1829] "Purchased product information" is data including detailed information about products purchased by a user, such as product names, categories, and quantities.

[1830] An "AI model" is an artificial intelligence model that uses machine learning algorithms to analyze data and make predictions or decisions for specific tasks.

[1831] The "refrigerator sub-app" is an auxiliary application that manages information about ingredients in the refrigerator and notifies the user's device of this information.

[1832] A "user terminal" is an electronic device that can be directly operated by a user, such as a smartphone or tablet.

[1833] A "sensor" is a device that detects the state inside a refrigerator and transmits that data to the outside.

[1834] The "camera" is a photographic device for capturing image information of the inside of the refrigerator.

[1835] A "server" is a central system that provides data processing and storage functions and integrates and manages data from various devices.

[1836] A "recipe" is an instruction manual that lists the steps and ingredients needed for cooking based on the ingredients in the refrigerator.

[1837] A "database" is a system for efficiently storing and managing large amounts of data.

[1838] A "common API" is a common program interface that enables data exchange between different systems.

[1839] "License revenue" is revenue earned by providing third parties with the right to use specific technology or software.

[1840] The present invention provides a system that efficiently manages ingredients purchased by users, updates ingredient information in the refrigerator in real time, and suggests suitable recipes, allowing users to use purchased food without waste and plan meal preparations accordingly.

[1841] The main components of the system include:

[1842] 1. Electronic payment app: This is software that manages purchase information when a user purchases ingredients. When a user purchases ingredients at a supermarket or online shop and completes payment using an electronic payment app, purchase information is generated. Examples include SmartPay and PayPal.

[1843] 1. Server: Receives purchase information sent from the electronic payment app and uses an AI model (using, for example, TensorFlow or PyTorch) to determine whether the purchased items need to be stored in the refrigerator. The server also manages information about ingredients in the refrigerator and processes the data to provide users with appropriate recipes.

[1844] 1. Refrigerator sub-application: This application manages information about ingredients in the refrigerator and notifies the user. The sub-application receives information about products that need to be stored in the refrigerator from the server and notifies the user's device.

[1845] 1. Sensors and cameras: These are devices used to monitor the state inside the refrigerator and detect the presence of ingredients. Data from these devices is sent to the server and used to update the ingredient information. Examples include built-in image recognition cameras and temperature sensors.

[1846] 1. User terminal: An electronic device that can be directly operated by the user, such as a smartphone or tablet. This receives recipe suggestions sent from the server and notifies the user.

[1847] To illustrate, consider the following scenario:

[1848] 1. Receiving purchase information:

[1849] A user purchases food items such as milk, butter, and bread at a supermarket and completes the payment using an electronic payment app. The device then sends the purchased item information from the electronic payment app to the server.

[1850] 1. Refrigerator storage criteria:

[1851] The server uses an AI model to determine whether or not a product needs to be refrigerated based on the product information. For example, it may determine that milk and butter need to be refrigerated, while bread is recommended to be stored at room temperature.

[1852] 1. Register for the refrigerator sub-app:

[1853] The server sends the result of the judgment to the refrigerator sub-application, which notifies the user's device. The user then stores the milk and butter they bought in the refrigerator.

[1854] 1. Refrigerator management and recipe suggestions:

[1855] The device (refrigerator) uses a camera to recognize newly stored ingredients and sends the information to the server. The server updates the information about ingredients in the refrigerator and generates recipes based on the ingredients it has. For example, it generates a recipe for "cream stew" based on the latest list of ingredients in the refrigerator (milk, butter, eggs, cheese) and sends it to the user's device. The user's device notifies the user of the recipe, and the user begins cooking based on that information.

[1856] An example of a prompt to be input to the generative AI model is, "Please explain the processing flow of a system in which a user purchases milk, butter, and bread at the supermarket, determines the need to store them in the refrigerator based on that information, registers them in the refrigerator sub-app, and then manages the information about ingredients in the refrigerator and suggests recipes."

[1857] As a result, the present invention provides a system that is highly convenient for users, reduces food waste, and assists in efficient meal planning.

[1858] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1859] Processing Steps

[1860] Step 1: Obtain purchase information

[1861] User: Purchases ingredients at a supermarket or online shop and completes payment using an electronic payment app.

[1862] Input: Purchased product information (product name, category, quantity, etc.)

[1863] How it works: A user uses the SuperPay app to buy milk, butter, and bread.

[1864] Output: Purchase product information is generated from the electronic payment app and sent to the terminal.

[1865] Step 2: Submit your purchase information

[1866] Terminal: Sends purchased product information to the server.

[1867] Input: Purchase information obtained from the electronic payment app

[1868] Operation: The terminal sends purchased product information to the server.

[1869] Output: The server receives the purchased product information.

[1870] Step 3: Determine if refrigeration is necessary

[1871] Server: Uses AI models to determine whether purchased items need to be stored in the refrigerator.

[1872] Input: Purchase product information

[1873] How it works: The server uses TensorFlow to determine that milk and butter need to be stored in the refrigerator, but bread can be stored at room temperature.

[1874] Output: A list of products that need to be stored in the refrigerator and those that can be stored at room temperature

[1875] Step 4: Registering the refrigerator sub-app

[1876] Server: Sends information about products that need to be stored in the refrigerator to the refrigerator sub-app.

[1877] Input: List of products that need to be kept in the refrigerator

[1878] Operation: The server sends milk and butter information to the refrigerator sub-app, which then notifies the user's device.

[1879] Output: Product information registered in the refrigerator sub-app and notification to the user

[1880] Step 5: Get information about ingredients in the refrigerator

[1881] Terminal (refrigerator): Detects food ingredient information using sensors and cameras inside the refrigerator and sends that information to the server.

[1882] Input: Sensor data and camera data from inside the refrigerator

[1883] How it works: The refrigerator's camera recognizes newly placed ingredients (milk and butter), and the sensor captures temperature information.

[1884] Output: Sensor data and camera data are sent to the server.

[1885] Step 6: Update the ingredients in your refrigerator

[1886] Server: Updates the information about ingredients in the refrigerator based on the received sensor data and camera data.

[1887] Input: Sensor data and camera data

[1888] Operation: The server updates the ingredient information in the database to keep the ingredient list up to date.

[1889] Output: Updated ingredient information

[1890] Step 7: Recipe suggestions

[1891] Server: Generates recipes using AI based on updated ingredient information.

[1892] Input: Updated ingredient information

[1893] Operation: Generates a recipe for "cream stew" based on the ingredient information held by the server.

[1894] Output: The generated recipe

[1895] Step 8: Notify the user

[1896] Terminal (user terminal): Receives recipe suggestions sent from the server and notifies the user.

[1897] Input: Generated recipe

[1898] Operation: The user terminal notifies the user of the recipe.

[1899] Output: The user receives a recipe notification.

[1900] These steps allow users to automate their purchase management, update their refrigerator with ingredients, and suggest recipes, reducing food waste and enabling efficient meal planning.

[1901] (Application example 1)

[1902] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1903] Conventional refrigerator management systems require users to manually enter information about purchased products, which takes time and effort. They also lack a means to automatically determine whether purchased products need to be stored in the refrigerator and manage them appropriately. Furthermore, recipe suggestions based on refrigerator inventory information are not provided in real time, making it difficult to make effective use of ingredients.

[1904] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1905] In this invention, the server includes means for receiving purchased product information from the electronic payment app, means for determining using AI whether the purchased product needs to be stored in the refrigerator, means for registering information on the purchased product determined to need to be stored in the refrigerator in a refrigerator sub-app, means for notifying the user terminal of the registered purchased product information, and means for automatically reading the purchased product information at the physical store and means for updating inventory information in the refrigerator based on that information, managing ingredients, and suggesting recipes. This allows the user to automatically register information on purchased products in the refrigerator, allowing them to grasp inventory information in the refrigerator in real time and use ingredients efficiently.

[1906] An "electronic payment app" is an application that allows users to pay for goods electronically.

[1907] "Purchased product information" refers to information such as the name, category, and quantity of the product purchased by the user.

[1908] "AI" stands for artificial intelligence, a system that learns and reasons to solve specific problems.

[1909] The "refrigerator sub-app" is an application that manages refrigerator inventory information and provides the user with information on ingredients and recipe suggestions.

[1910] A "user terminal" is an information processing device operated by a user, such as a smartphone, tablet, or personal computer.

[1911] A "physical store" is a store where users can physically purchase products.

[1912] "Means for automatically reading information about purchased items" refers to technology that automatically obtains information about purchased items using sensors, cameras, etc.

[1913] "Inventory information" is information that indicates the current status of ingredients and products in the refrigerator.

[1914] "Recipe suggestion" refers to suggesting cooking methods to the user based on information about ingredients in the refrigerator.

[1915] "Sensor data" is digital data obtained from sensors regarding temperature, humidity, the presence of objects, etc.

[1916] "Camera data" refers to image and video data captured by a camera.

[1917] A "common API" is an application program interface that can be used commonly across multiple systems.

[1918] A "smart device" is an electronic device that can connect to the Internet and allows users to input and display information.

[1919] "Licensing revenue" is revenue earned by permitting the use of a certain technology or product.

[1920] This invention is a system that automatically acquires information about products purchased by a user, updates refrigerator inventory based on that information, and manages ingredients and suggests recipes. This system is configured using the following hardware and software.

[1921] Hardware

[1922] Smartphone: A device that allows users to purchase products and use electronic payment apps.

[1923] Smart glasses: devices that display real-time information while users are shopping in a physical store.

[1924] Refrigerator: A home appliance equipped with sensors and cameras inside that can obtain information about the food inside the refrigerator.

[1925] Server: A central processing unit that processes purchased product information and data from the refrigerator.

[1926] software

[1927] Electronic payment app: An application that allows users to purchase products and make payments.

[1928] Refrigerator sub-app: An application that manages the inventory information in the refrigerator and notifies the user.

[1929] Common API: An interface for receiving and analyzing sensor data and camera data from refrigerator manufacturers.

[1930] AI model: An artificial intelligence model that analyzes purchased product information and determines whether or not it needs to be stored in the refrigerator.

[1931] In this system, users first purchase products at a physical store and pay through an electronic payment app. Purchase information is automatically sent to a smartphone at the time of payment. This information is then sent to a server, where an AI model analyzes it and determines whether the product needs to be stored in a refrigerator.

[1932] Next, product information that needs to be stored in the refrigerator is registered in the refrigerator sub-app and simultaneously notified to the user's device. The user receives this notification and stores the necessary ingredients in the refrigerator.

[1933] The refrigerator uses internal sensors and cameras to detect newly stored ingredients. The detected data is sent to the server via a common API. The server uses this data to update the refrigerator's inventory and generate optimal recipes.

[1934] The generated recipe is sent to the user's device, and the user can start cooking based on the recipe. This allows the user to automatically register purchased product information in the refrigerator, keep track of inventory in the refrigerator in real time, and use ingredients efficiently.

[1935] Specific examples

[1936] For example, suppose a user purchases milk, butter, and bread at a physical store and completes the payment using an electronic payment app. At this time, the purchase information sent from the payment app is sent to a server, where an AI model analyzes it and determines that the milk and butter need to be stored in the refrigerator. This information is registered in the refrigerator sub-app, and the user checks the information on their smartphone. Next, sensors and cameras inside the refrigerator detect the newly placed ingredients and send the information to the server. The server analyzes the data and generates a recipe (e.g., salad, pasta, smoothie) based on the ingredients on hand, and notifies the user.

[1937] Prompt Sentence Examples

[1938] When a user purchases food at a physical store and completes the payment through a payment app on their smartphone, please explain the process by which the purchase information is registered in the refrigerator sub-app. Also, please explain how recipes are suggested based on the information about ingredients in the refrigerator.

[1939] In this way, the present invention greatly improves the user's daily life, allowing for efficient food management and cooking.

[1940] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1941] Step 1:

[1942] A user purchases an item in a physical store and pays using an electronic payment app.

[1943] Input: Information about the product purchased by the user (product name, category, quantity, etc.)

[1944] What happens: A user uses their smartphone to scan an item into a payment app and complete the payment.

[1945] Output: Purchase product information is generated and sent to your smartphone.

[1946] Step 2:

[1947] The smartphone sends the purchased product information to the server.

[1948] Input: Purchase information obtained from your smartphone

[1949] Specific operation: The smartphone obtains the purchased product information from the electronic payment app and sends it to the server as an HTTP request.

[1950] Output: Purchase information is sent to the server.

[1951] Step 3:

[1952] The server uses an AI model to determine whether refrigeration is necessary.

[1953] Input: Purchase information sent to the server

[1954] How it works: The server inputs purchased product information into the AI ​​model and determines whether or not the product needs to be stored in the refrigerator. The AI ​​model makes this determination based on the product name and category.

[1955] Output: A list of items that need to be kept in a refrigerator is generated.

[1956] Step 4:

[1957] The server registers product information that needs to be stored in the refrigerator in the refrigerator sub-app.

[1958] Input: List of products that require refrigeration as determined by the server

[1959] Specific operation: The server sends the registration information to the refrigerator sub-app via API.

[1960] Output: Product information that needs to be stored is registered in the refrigerator sub-app.

[1961] Step 5:

[1962] The refrigerator sub-app notifies the user device of the registration information

[1963] Input: Product information registered in the refrigerator sub-app

[1964] Specific operation: The refrigerator sub-app notifies the user device of the information received from the server. The information is sent to the user using push notifications, etc.

[1965] Output: The registration information for the purchased product is displayed on the user's device.

[1966] Step 6:

[1967] The user places the product in the refrigerator

[1968] Input: Registration information displayed on the user's device

[1969] Specific operation: The user places the purchased items in the refrigerator. The product placement is performed while confirming the registration information.

[1970] Output: The product is stored in the refrigerator.

[1971] Step 7:

[1972] The refrigerator detects the food stored inside using internal sensors and cameras.

[1973] Input: New item placed in refrigerator

[1974] How it works: The refrigerator uses internal sensors and cameras to detect newly added ingredients.

[1975] Output: The detected ingredients information is generated.

[1976] Step 8:

[1977] The refrigerator sends the food information it detects to the server.

[1978] Input: Detected ingredient information

[1979] Specific operation: The refrigerator sends ingredient information to the server using a common API.

[1980] Output: Ingredient information is sent to the server.

[1981] Step 9:

[1982] The server updates the ingredient information and generates the optimal recipe.

[1983] Input: Ingredient information sent to the server

[1984] Specific operation: The server updates the inventory data in the refrigerator based on the received ingredient information and generates the optimal recipe using a generative AI model.

[1985] Output: The optimal recipe information is generated.

[1986] Step 10:

[1987] The server sends the generated recipe to the user's device.

[1988] Input: Generated recipe information

[1989] Specific operation: The server sends the generated recipe information to the user's device. The recipe information is sent to the user using push notifications, etc.

[1990] Output: Recipe information is displayed on the user's device.

[1991] Through the above processing steps, the user can automatically register information about purchased items in the refrigerator, grasp inventory information in the refrigerator in real time, and use ingredients efficiently.

[1992] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1993] 1. System Overview

[1994] The system of this invention receives information about products purchased by users through an electronic payment app, and based on that information, AI determines whether the product needs to be stored in the refrigerator and automatically registers that information in the refrigerator sub-app. It also includes a function to manage information about ingredients in the refrigerator and suggest recipes to users based on that information. This system works in conjunction with refrigerator manufacturers, acquiring sensor and camera data through a common API to update information about ingredients in the refrigerator in real time. Furthermore, by combining it with an emotion engine, recipes are personalized based on the user's emotions, improving the user experience.

[1995] 2. System Configuration and Operation

[1996] Receiving purchase information

[1997] User:

[1998] When a user purchases ingredients at a supermarket or online shop and pays using an electronic payment app, purchased product information is generated.

[1999] Device:

[2000] Purchase information (product name, category, quantity, etc.) is sent from the electronic payment app to the server.

[2001] server:

[2002] The server receives the purchased product information and uses an AI model to determine whether it needs to be stored in a refrigerator.

[2003] Determining whether it can be stored in a refrigerator

[2004] server:

[2005] The AI ​​model analyzes the purchased product information and determines whether it needs to be stored in a refrigerator. For example, milk and butter are determined to be products that need to be stored in a refrigerator.

[2006] Device:

[2007] The information about products that need to be stored in the refrigerator is registered in the refrigerator sub-application, which receives the information from the server. The registered information is also sent to the user's device.

[2008] Refrigerator management and recipe suggestions

[2009] Terminal (refrigerator):

[2010] The storage status of ingredients is detected using cameras and sensors inside the refrigerator, and that information is sent to a server.

[2011] server:

[2012] The system analyzes sensor and camera data received from the refrigerator and updates the information about ingredients in the refrigerator. It then generates recipes based on the updated information.

[2013] Terminal (user terminal):

[2014] The server receives the recipe suggestions and notifies the user. The user checks the recipe, takes out the necessary ingredients from the refrigerator, and starts cooking.

[2015] Collaboration with refrigerator manufacturers

[2016] server:

[2017] We provide refrigerator manufacturers with a common API to collect sensor and camera data from refrigerators, enabling accurate and real-time updates of food ingredients stored in refrigerators. We also monitor API usage and manage license revenue.

[2018] Terminal (refrigerator):

[2019] Information from inside the refrigerator is sent to the server via the API, and management information from the server is displayed inside the refrigerator.

[2020] Incorporating an emotion engine

[2021] User Emotion Recognition and Recipe Personalization

[2022] Device (smartphone, etc.):

[2023] It incorporates an emotion engine that analyzes emotions through facial recognition and voice data of the user. For example, it uses a camera and microphone to analyze facial expressions and voices while the user is operating the device.

[2024] server:

[2025] It receives emotional data sent from the emotion engine and personalizes recipes based on that data, for example, suggesting easy-to-make recipes if the user is tired.

[2026] Terminal (user terminal):

[2027] It notifies users of personalized recipes and makes meal suggestions based on the user's emotions.

[2028] Emotion-based food consumption and management

[2029] server:

[2030] The system continuously monitors the user's emotional data and suggests food consumption and management based on their emotions. For example, if the user is feeling stressed, it will suggest recipes with ingredients that are effective in reducing stress.

[2031] Device (smartphone, etc.):

[2032] It notifies users in a timely manner and provides alerts about food expiration dates and conditions based on emotional data.

[2033] Specific examples

[2034] Example of operation at time of purchase

[2035] 1. A user selects items (milk, butter, bread) at a supermarket and completes the purchase using an electronic payment app.

[2036] 2. The device sends the purchased product information to the server, which uses AI to determine whether the milk and butter need to be stored in the refrigerator.

[2037] 3. The registration information is sent to the refrigerator sub-app, and the user stores the purchased milk and butter in the refrigerator.

[2038] Example of refrigerator management and recipe suggestions

[2039] 1. The terminal (refrigerator) detects newly stored ingredients and sends the information to the server.

[2040] 2. The server updates the information about ingredients in the refrigerator and generates a recipe based on the ingredients you have.

[2041] 3. The generated recipe is sent to the user's device, and the user checks the recipe and cooks the ingredients.

[2042] Example of Emotion Engine in Action

[2043] 1. The device (smartphone, etc.) collects the user's facial recognition and voice data and analyzes it using an emotion engine.

[2044] 2. Based on the emotional data, the server suggests recipes using ingredients that have a relaxing effect, for example, if the user feels like relaxing.

[2045] 3. The user checks the appropriate recipe and uses it to cook.

[2046] This system allows users to automate their purchase management, reducing wasteful consumption and creating effective meal plans. Furthermore, the incorporation of an emotion engine enables personalized meal suggestions that are in line with the user's emotions, which is expected to improve the user experience.

[2047] The processing flow will be explained below.

[2048] Program processing steps

[2049] Receiving and determining purchase information

[2050] Step 1:

[2051] A user selects a product at a supermarket or online shop and completes the purchase process using an electronic payment app.

[2052] Step 2:

[2053] The device (smartphone, etc.) captures information about the purchased product (product name, category, quantity, etc.) and sends it to the server.

[2054] Step 3:

[2055] The server receives the purchased product information sent from the electronic payment application.

[2056] Step 4:

[2057] The server uses an AI model to analyze the received purchase information and determine whether it needs to be stored in a refrigerator.

[2058] For example, it determines that "milk" and "butter" need to be stored in the refrigerator, but "bread" does not.

[2059] Step 5:

[2060] The server generates a list of products that need to be stored in the refrigerator and issues an instruction to register the list in the refrigerator sub-app.

[2061] Step 6:

[2062] The terminal (smartphone, etc.) receives instructions from the server and registers the purchased items in the refrigerator sub-app.

[2063] After the terminal has completed registration of the purchased product information, it notifies the user.

[2064] Refrigerator management and recipe suggestions

[2065] Step 7:

[2066] The food purchased by the user is stored in the refrigerator.

[2067] Step 8:

[2068] The terminal (refrigerator) detects newly stored ingredients using its internal camera and sensors and sends this information to the server.

[2069] Step 9:

[2070] The server receives the data sent from the refrigerator sub-application and updates the information about ingredients in the refrigerator.

[2071] Step 10:

[2072] The server generates recipes based on the ingredients in the refrigerator.

[2073] For example, generate a recipe for "Omelette with butter and eggs."

[2074] Step 11:

[2075] The server transmits the generated recipe information to the user's terminal.

[2076] Step 12:

[2077] The terminal (smartphone, etc.) notifies the user of the recipe suggestions sent from the server.

[2078] Step 13:

[2079] The user checks the recipe, takes the necessary ingredients from the refrigerator, and starts cooking.

[2080] Collaboration with refrigerator manufacturers

[2081] Step 14:

[2082] The server provides a common API to refrigerator manufacturers.

[2083] Step 15:

[2084] The terminal (refrigerator) sends information from the interior camera and sensors to the server via API.

[2085] Step 16:

[2086] The server analyzes the data sent from the refrigerator and automatically updates the information about ingredients in the refrigerator.

[2087] Step 17:

[2088] The server monitors API usage and manages license revenue from refrigerator manufacturers.

[2089] Incorporating an emotion engine

[2090] Step 18:

[2091] The device (smartphone, etc.) collects the user's facial recognition and voice data and analyzes it using an emotion engine.

[2092] Step 19:

[2093] The server uses emotional data to suggest easy-to-make recipes, for example, if the user feels tired.

[2094] Step 20:

[2095] The device (smartphone, etc.) notifies the user of the personalized recipe.

[2096] Step 21:

[2097] The user checks the recipe based on their emotions and uses the recipe to cook.

[2098] Step 22:

[2099] The server continuously monitors emotional data and suggests recipes using ingredients that are effective in reducing stress.

[2100] Step 23:

[2101] The device (smartphone, etc.) notifies the user in a timely manner and provides alerts regarding the expiration date and condition of ingredients based on emotional data.

[2102] This allows users to automate the management of their purchases and create efficient meal plans, and the incorporation of an emotion engine makes it possible to make personalized meal suggestions that are in line with the user's emotions, which is expected to improve the user experience.

[2103] Example 2

[2104] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2105] Currently, there are many grocery management systems and recipe recommendation systems on the market, but these systems have limitations in terms of centralized management and personalized recommendations. In particular, in terms of refrigerator ingredient management and recipe recommendations based on the user's mood, existing systems lack flexibility and do not sufficiently improve the user experience. Furthermore, they lack automated mechanisms for reducing food waste and efficiently planning meals. This invention aims to solve these problems and improve the quality of life of users.

[2106] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2107] In this invention, the server includes: means for receiving purchased product information from an electronic payment app; means for determining whether a purchased product requires refrigeration using AI; and means for registering information about purchased products determined to require refrigeration in a refrigeration appliance app. This allows for efficient management of purchased product information and enables appropriate storage based on the information. The server also includes means for updating food product information stored in the refrigeration appliance based on data from the refrigeration appliance app; means for generating recipes based on the food product information stored in the refrigeration appliance; and means for transmitting the generated recipes to a user terminal. This allows for real-time updates of food product information stored in the refrigeration appliance, and for suggesting recipes suitable for the user based on the updated information. The server also includes means for providing a common API to refrigeration appliance manufacturers; means for receiving sensor data and image data from the refrigeration appliance; means for analyzing the data and updating the food product information stored in the refrigeration appliance; and means for monitoring the usage of the common API and managing license revenue. This allows for efficient collection and management of refrigeration appliance data, enabling appropriate food product management and revenue generation. The server also includes means for recognizing user emotions; means for personalizing recipes based on the emotion data; and means for notifying the user terminal of the personalized recipes. This allows for personalized meal suggestions based on the user's emotions, further improving the user experience.

[2108] 1. An "electronic payment app" is a software application that a user uses to purchase goods online or in-store, and that digitizes the payment process.

[2109] 2. "Purchased Product Information" refers to detailed data of the product purchased by the user, including product name, category, quantity, price, etc.

[2110] 3. "AI" stands for artificial intelligence, which refers to algorithms and models that analyze data and perform specific tasks automatically.

[2111] 4. "Refrigeration equipment app" is software that supports the operation of refrigeration equipment, managing food ingredient information and monitoring storage conditions.

[2112] 5. "User terminal" refers to a device used by a user, such as a smartphone, tablet, PC, or other electronic device.

[2113] 6. "Sensor Data" means information detected by sensors, such as temperature, humidity, and location within refrigeration equipment.

[2114] 7. "Image data" refers to image files taken by a camera placed inside the refrigeration equipment, which are used to visually record the condition and placement of ingredients.

[2115] 8. "Common API" means a common application program interface that can be used by multiple refrigeration equipment manufacturers, enabling data transmission and reception and the sharing of functions.

[2116] 9. "License Revenue" means revenue received from refrigeration equipment manufacturers for use of the provided common API, including API usage fees and service fees.

[2117] 10. "Emotional Data" means information about a user's emotional state obtained through facial recognition or voice analysis, and is data indicating joy, sadness, fatigue, stress, etc.

[2118] 11. "Personalized Cooking Recipes" are cooking recipes that are specifically tailored to a user's individual emotional state and preferences and that are provided as part of meal suggestions to the user.

[2119] 1. System Overview

[2120] The system of this invention receives information about products purchased by users through an electronic payment app, and based on that information, AI determines the need for refrigeration and automatically registers that information in the refrigeration appliance app. It also includes a function to manage food information stored in the refrigeration appliance and suggest recipes to users based on that information. This system works in conjunction with refrigeration appliance manufacturers, acquiring sensor data and image data through a common API to update the food information stored in the refrigeration appliance in real time. Furthermore, by incorporating an emotion engine, recipes are personalized based on the user's emotions, improving the user experience.

[2121] 2. System Configuration and Operation

[2122] Receiving purchase information

[2123] When a user purchases groceries at a supermarket or online shop and pays using an electronic payment app, purchased product information (product name, category, quantity, etc.) is generated within the electronic payment app.

[2124] The terminal (electronic payment app) sends the purchased product information to the server. For example, it is sent to the cloud server in the form of an electronic receipt.

[2125] The server receives the purchased product information and records it in a database.

[2126] Determining whether refrigerated storage is possible

[2127] The server uses AI models to determine the need for refrigeration based on the purchased product information received, for example, milk and perishables.

[2128] The device registers the information about the products that need to be stored sent from the server in the refrigeration equipment app and notifies the user's device, allowing the user to store their purchased items appropriately.

[2129] Refrigeration equipment management and recipe suggestions

[2130] The terminal (refrigeration equipment) uses internal sensors and cameras to monitor the condition of the food and sends the data to a server.

[2131] The server analyzes the received sensor data and image data and updates the information in the refrigeration equipment, for example, identifying the type and quantity of ingredients.

[2132] The server generates usable recipes based on updated ingredient information, taking into account nutritional value and user preferences.

[2133] The terminal (user terminal) receives the recipe suggestions sent from the server and notifies the user, who can then check the recipes and prepare the suggested meals.

[2134] Collaboration with refrigeration equipment manufacturers

[2135] The server provides a common API to refrigeration equipment manufacturers and acquires sensor data and image data, for example, to manage fresh vegetables and dairy products.

[2136] The terminal (refrigeration equipment) sends data to the server via API, which updates the information in the refrigeration equipment in real time, allowing users to manage their ingredients appropriately.

[2137] Specific examples

[2138] Example of operation at time of purchase

[2139] 1. A user selects milk, butter, and bread at a supermarket and completes the purchase using an electronic payment app.

[2140] 2. The terminal (electronic payment app) sends the purchased product information to the server.

[2141] 3. The server uses AI to determine that the milk and butter need to be refrigerated.

[2142] 4. The registration information is sent to the refrigerator app, and the user places the milk and butter in the refrigerator.

[2143] Example of refrigeration equipment management and recipe suggestions

[2144] 1. The terminal (refrigeration equipment) detects newly stored food items and sends the information to the server.

[2145] 2. The server updates the food information in the refrigerator and generates a recipe based on the ingredients it has.

[2146] 3. The generated recipe is sent to the user's device, and the user checks the recipe and cooks the ingredients.

[2147] Example of Emotion Engine in Action

[2148] 1. The device (smartphone, etc.) collects the user's facial recognition and voice data and analyzes it using an emotion engine.

[2149] 2. Based on the emotional data, the server suggests recipes using ingredients that have a relaxing effect if the user feels like relaxing.

[2150] 3. The user checks the appropriate recipe and uses it to cook.

[2151] Example prompts for generative AI models

[2152] Receiving purchase information

[2153] Prompt: Receive information about the milk, butter, and bread purchased by a user at the supermarket from an electronic payment app, and use AI to determine whether they need to be stored in the refrigerator.

[2154] Refrigeration equipment management and recipe suggestions

[2155] Prompt: Send the newly stored ingredients to the server, update the ingredients in the refrigerator, and suggest a recipe.

[2156] Emotion Engine Operation

[2157] Prompt: Collect facial and voice data from the user and suggest recipes suitable for when the user wants to relax.

[2158] By using this system, users can automate the management of their purchases, reduce wasteful consumption, and create effective meal plans. In addition, the incorporation of an emotion engine enables personalized meal suggestions that are in line with the user's emotions, which is expected to improve the user experience.

[2159] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2160] Step 1:

[2161] A user purchases groceries at a supermarket or online shop and pays using an electronic payment app. At this time, product information (product name, category, quantity, etc.) is generated within the electronic payment app. The user inputs their intention to purchase a product, and as a result, product information is generated.

[2162] Step 2:

[2163] The terminal (electronic payment app) sends purchased product information to the server. For example, the purchase information is sent to the cloud server in the form of an electronic receipt. The terminal receives purchased product information as input and sends it to the server as output.

[2164] Step 3:

[2165] The server receives the purchased product information and records it in the database. The server analyzes the purchased product information received as input and saves it in the database. This records the purchased product information.

[2166] Step 4:

[2167] The server uses an AI model to determine the need for refrigerated storage based on the received purchase information. Purchase information is provided to the AI ​​model as input, and the AI ​​analyzes the data to determine the need for refrigerated storage. For example, it may determine that milk or fresh food needs to be stored in a refrigerator.

[2168] Step 5:

[2169] The terminal registers the information about the products that require storage sent from the server in the refrigeration equipment app and notifies the user terminal.The terminal receives information about the products that require refrigeration storage from the server as input, processes it as output to be registered in the refrigeration equipment app, and notifies the user terminal.

[2170] Step 6:

[2171] The terminal (refrigeration equipment) monitors the condition of food using internal sensors and cameras and sends the data to a server. The refrigeration equipment's sensors and cameras acquire food condition data as input and send it to the server as output. For example, sensor data on temperature and humidity, and image data of food.

[2172] Step 7:

[2173] The server analyzes the received sensor data and image data and updates the food ingredient information in the refrigerator. The server receives sensor data and image data as input, analyzes them to identify the type and amount of food ingredients, and updates the food ingredient information in the refrigerator as output.

[2174] Step 8:

[2175] The server generates usable recipes based on updated ingredient information, taking into account nutritional value and user preferences. Using the latest ingredient information as input, the AI ​​model analyzes the data and generates suitable recipes as output.

[2176] Step 9:

[2177] The terminal (user terminal) receives the recipe suggestions sent from the server and notifies the user. It receives the recipe information provided by the server as input and suggests it to the user using the notification function as output. The user checks this recipe and prepares the suggested meal.

[2178] Step 10:

[2179] The device (such as a smartphone) collects the user's facial recognition and voice data and analyzes it using an emotion engine. The user's facial recognition and voice data are taken as input, and the output is analyzed by the emotion engine. For example, data collection using a camera or microphone.

[2180] Step 11:

[2181] The server analyzes the emotional data and suggests recipes suitable for when the user wants to relax or when they are tired. The server receives the emotional data as input, analyzes it with an AI model, and generates suitable recipes as output.

[2182] Step 12:

[2183] The user checks the personalized recipe notified to the device and uses that recipe to cook. The notified recipe information is received as input, and cooking is performed based on that recipe as output. The user's cooking action is the final execution stage.

[2184] (Application example 2)

[2185] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2186] In conventional refrigerator and purchasing information management systems, storing and managing purchased products is often done manually, which is cumbersome for users and makes it difficult to effectively manage ingredients. Furthermore, there is a lack of personalized recipe suggestions based on the user's emotions and condition, which hinders the improvement of the user experience. There is a need for an efficient and user-friendly system that can solve these problems.

[2187] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2188] In this invention, the server includes means for receiving purchased product information from the electronic payment app, means for determining using AI whether the purchased product needs to be stored in the refrigerator, means for registering information on the purchased product determined to need to be stored in the refrigerator in a refrigerator sub-app, means for notifying the user terminal of the registered purchased product information, means for automatically acquiring purchase information from a physical store and linking with the refrigerator, means for personalizing recipes based on the user's emotions using an emotion engine, and means for analyzing the user's emotion data and suggesting appropriate ingredient consumption and recipes. This automates the storage and management of purchased products and makes it possible to realize personalized ingredient consumption and recipe suggestions based on the user's emotions.

[2189] An "electronic payment app" is a software application that receives purchase information electronically and completes payments.

[2190] "Purchased product information" is data including detailed information such as the name, category, and quantity of the product purchased by the user.

[2191] "AI" refers to artificial intelligence systems that automate specific tasks, analyze data, and make decisions.

[2192] The "refrigerator sub-app" is an auxiliary application for managing and updating information about ingredients in the refrigerator.

[2193] A "user terminal" is an electronic device operated by a user, such as a smartphone or tablet.

[2194] A "physical store" is a physical sales location where products are actually displayed and sold.

[2195] An "emotion engine" is software that analyzes a user's emotions and makes appropriate suggestions based on their state.

[2196] A "recipe" is a document that describes the steps and ingredients for making a particular dish.

[2197] A "common API" is a standardized interface that allows different systems to communicate with each other and exchange data.

[2198] "Sensor data" is information obtained from sensors that detect the temperature, humidity, and presence of objects inside the refrigerator.

[2199] "Camera data" refers to data acquired as video or images of the inside of a refrigerator.

[2200] A "generative AI model" is an artificial intelligence model designed to generate specific outputs based on user input.

[2201] A "prompt sentence" is an input sentence that a generative AI model uses to generate appropriate output.

[2202] 1. System Overview

[2203] This invention is a system that receives information about products purchased by users through an electronic payment app, and then uses AI to determine whether or not to store the products in the refrigerator, automatically registering them in a refrigerator sub-app. It also includes a function to manage information about ingredients in the refrigerator and suggest recipes based on that information, allowing recipes to be personalized based on the user's emotions.

[2204] 2. Program Generation

[2205] The program to realize this system is as follows, but the details of the code are not included. The specific main functions are as follows:

[2206] Receiving purchase information: The user sends product information from the electronic payment app.

[2207] Refrigerated storage decision: AI analyzes product information and determines the need for refrigerated storage.

[2208] Information registration: Register the necessary product information in the refrigerator sub-app.

[2209] User notificat...

Claims

1. A means for receiving purchase product information from the electronic payment app; A method to use AI to determine whether purchased products need to be stored in the refrigerator, A means for registering information about purchased items determined to need to be stored in the refrigerator in a refrigerator sub-application; means for notifying a user terminal of registration information of a purchased product; A system including:

2. A way to update the information about ingredients in the refrigerator based on the data from the refrigerator sub-app, A means for generating recipes based on information about ingredients in a refrigerator; means for transmitting the generated recipe to a user terminal; The system of claim 1 further comprising:

3. A means to provide a common API to refrigerator manufacturers, means for receiving sensor data and camera data from the refrigerator; A means of analyzing this data and updating the information on ingredients in the refrigerator; A means to monitor API usage and manage license revenue, The system of claim 1 further comprising:

Images