system

Abstract

We provide the system. [Solution] A means for analyzing item data acquired using image recognition means, A means of updating personal belongings information based on analyzed item data, A means for generating a shopping list that recommends necessary items using updated inventory information and pre-registered cooking method data, A means for outputting the generated shopping list to a display device, A system that includes this.

Description

Technical Field

[0005] ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Many consumers have problems such as forgetting to buy necessary items or purchasing the same items excessively in their daily lives. Such inefficiencies in purchase management may lead to economic waste and an increase in food waste, and may become a factor hindering the realization of a sustainable society. The object of the present invention is to solve these problems and enable consumers to efficiently purchase necessary items.

Means for Solving the Problems

[0005] This invention provides a system that analyzes item data acquired using image recognition means and updates possession information in real time. Furthermore, it includes means for automatically generating a shopping list that recommends necessary items by referring to pre-registered cooking method data based on the updated possession information. In addition, it has a function to calculate the optimal purchase time based on consumption history and notify the user, making it a system that enables efficient management of necessary items. As a result, consumer purchasing behavior can be optimized, and sustainable consumption without waste can be supported.

[0006] "Image recognition means" refers to a technology and apparatus that automatically identifies a specific item from captured image data and analyzes that information.

[0007] "Item data" refers to data obtained through image recognition means, including information about the type and quantity of items.

[0008] "Possessions information" refers to data that records the current type, quantity, and condition of items owned by the user.

[0009] "Cooking method data" refers to information that describes how to prepare various dishes, including the necessary ingredients and procedures.

[0010] The "shopping list generation method" refers to the technology and functionality used to analyze the user's possession information and cooking method data to create a list of necessary items.

[0011] A "display device" is a device that displays generated shopping lists and notification information so that users can check them.

[0012] "Consumption history" refers to a record of items that a user has purchased and used in the past, and is data that indicates the frequency of use and rate of consumption of those items.

[0013] "Calculating the optimal purchase timing" is the process of analyzing consumption trends and inventory levels of goods to determine the best time to purchase them. [Brief explanation of the drawing]

[0014] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0015] Hereinafter, an example of an embodiment of a system according to the technologyof the present disclosure will be described with reference to the accompanyingdrawings. [\

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simplyreferred to as "processor") may be a single arithmetic unit or a combination ofplural arithmetic units. Further, the processor may be a single type ofarithmetic unit or a combination of plural types of arithmetic units. Examplesof arithmetic units include a CPU (Central Processing Unit), a GPU (GraphicsProcessing Unit), a GPGPU (General-Purpose computing on Graphics ProcessingUnits), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) isa memory in which information is temporarily stored and is used as a workmemory by the processor.

[0019] In the following embodiments, the numbered storage is one or morenon-volatile storage devices that store various programs and variousparameters, etc. Examples of non-volatile storage devices include flashmemory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), ormagnetic tapes, and the like.

[0020] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] This invention provides a system that enables efficient item management and shopping assistance using a user's personal device. First, the user takes photos of storage shelves and refrigerators in their home using their device. These images are uploaded to a server, which uses image recognition to automatically identify items and analyze their type and quantity. The results of this analysis are stored in a database as item information.

[0036] The server uses the analyzed inventory information to access the user's registered cooking method database. This database contains recipes for various dishes, detailing the necessary ingredients and procedures. The server compares this cooking method data with the inventory information to generate a shopping list recommending necessary items. The generated shopping list is sent to the user's device, allowing them to easily view it through the device's display.

[0037] Furthermore, the server analyzes past consumption history to understand consumption patterns for each item. This data is used to calculate the optimal purchase time and notify the user's terminal. This allows users to purchase specific items before they run out, optimizing their shopping plans.

[0038] As a concrete example, suppose a user wants to check the ingredients for a pasta dish. The user takes a picture of the pantry with their device, and the server analyzes the image to determine the quantities of pasta and tomato sauce. The server then references a pasta recipe from a recipe database, and if any ingredients are missing, a list is generated and presented to the user. If the server determines that there is a shortage of tomato sauce, it notifies the user to purchase it and encourages them to take this into consideration when shopping next time. This system enables efficient food management within the household and allows for waste-free shopping planning.

[0039] The following describes the processing flow.

[0040] Step 1:

[0041] Users take photos of storage shelves or refrigerators using their devices. These images are then uploaded directly from the device to the server.

[0042] Step 2:

[0043] The server analyzes the received image data using image recognition technology. Through this analysis, it automatically identifies the type and quantity of items present in the image.

[0044] Step 3:

[0045] The server updates the database's inventory information based on the analysis results. This ensures that the latest inventory status of items in the user's home is reflected.

[0046] Step 4:

[0047] The server references the updated inventory information and compares it with the registered cooking method database. This identifies the items needed for the cooking method the user desires.

[0048] Step 5:

[0049] The server generates a shopping list based on the recommended items for the user. This list consists of the missing items.

[0050] Step 6:

[0051] The terminal notifies the user of the generated shopping list. The user can view this list on the terminal's display.

[0052] Step 7:

[0053] The server references consumption history information and calculates the optimal purchase time for each item based on consumption trends. This information is then communicated to the user, prompting them to purchase items at the appropriate time.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] In modern households, inventory management and shopping planning tend to rely on manual processes, which are time-consuming and labor-intensive. Therefore, there is a need for systems that efficiently manage inventory and ensure that necessary items are purchased at the right time. Furthermore, determining the optimal purchase timing, taking past consumption history into account, presents a challenge.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] In this invention, the server includes means for analyzing item information acquired using image processing means, means for updating possession data based on the analyzed item information, means for generating a shopping list that suggests necessary items using the updated possession data and pre-registered cooking information, means for analyzing past recorded consumption history to understand the consumption trend of each item, and means for calculating the optimal purchase time based on the consumption trend and notifying the user. This enables the user to efficiently manage their possessions, create an efficient shopping plan, and purchase necessary items at the optimal time.

[0059] "Image processing means" refers to technology that analyzes images of objects and converts their contents into digital data.

[0060] "Item information" refers to data that includes the type, quantity, and related attributes of an item, obtained by image processing means.

[0061] "Means of analysis" refers to the entire process of examining and organizing acquired item information in detail.

[0062] "Possessions data" refers to information that represents the possession status of items within a household, and includes data such as the names and quantities of items.

[0063] "Cooking information" refers to information that includes recipes for various dishes, detailing specific procedures and required ingredients.

[0064] The "Shopping List" is a list of items that the user should purchase in the future, providing organized information about the necessary items.

[0065] "Consumption history" refers to historical information about items a user has consumed, including data on past usage quantities and frequency.

[0066] The "optimal purchase timing" is calculated based on consumption trends and possession data, and represents the most appropriate time to purchase a new item.

[0067] This invention is a system that uses a user's personal device to efficiently manage the inventory of items in their home and assist with shopping. First, the user takes pictures of their refrigerator and storage shelves using the device's camera. These images are uploaded to a server via the network.

[0068] The server analyzes uploaded images using image processing technologies such as Google® Cloud Vision API and Amazon Rekognition. This automatically recognizes the type and quantity of items present in the image and extracts them as item information. This item information is then stored in a database and managed as household possession data.

[0069] Next, the server refers to a cooking information database that the user has registered in advance. This database contains recipe information for various dishes. The server compares this with the user's inventory data, identifies any missing ingredients, and generates a shopping list suggesting the necessary items. The generated shopping list is transferred to the terminal, and the user can view it through the display device.

[0070] Furthermore, the server analyzes past consumption history data to understand consumption trends for each item. Based on this data, the optimal purchase time is calculated and notified to the user. This allows users to purchase necessary items at the optimal time, enabling them to create shopping plans that reduce waste.

[0071] As a concrete example, consider a scenario where a user wants to make a pasta dish. The user takes a picture of their food shelf with their device, and the server analyzes the image to check the inventory of pasta and tomato sauce. The server then refers to a cooking information database and, if any of the ingredients needed for the pasta recipe are missing, it lists the missing items and presents them to the user. Additionally, if the tomato sauce inventory is low, the server notifies the user of the optimal time to purchase it, allowing them to incorporate this into their next shopping plan.

[0072] A concrete example of a prompt message would be, "Analyze the contents of your refrigerator and check if you have all the ingredients needed for the pasta recipe." This system allows users to manage their belongings more efficiently and shop more systematically.

[0073] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0074] Step 1:

[0075] The user uses their device's camera to take pictures of things like refrigerators and storage shelves in their home. These captured images become the input data. The images are saved on the device and then prepared to be uploaded to the server for processing.

[0076] Step 2:

[0077] The device uploads images it has taken to a server via the internet. The uploaded image data becomes the input, and the server receives this image data. In this process, a secure communication protocol such as HTTPS is used to maintain data security.

[0078] Step 3:

[0079] The server passes the received image data to an image processing system. Specifically, it analyzes the image using image recognition technologies such as Google Cloud Vision API or Amazon Rekognition. Through this analysis, objects in the image are recognized, and information about the objects, such as their type and quantity, is output.

[0080] Step 4:

[0081] The server stores the analyzed item information in a database. The input is the analysis results, and the output is an updated database of household possessions. This database update ensures accurate tracking of the type and quantity of items.

[0082] Step 5:

[0083] The server refers to the registered cooking information database and compares it with the inventory data. The input for this comparison is the inventory data and cooking information, and the output is a list of missing ingredients. The server then generates a shopping list based on this missing information.

[0084] Step 6:

[0085] The server sends the generated shopping list to the terminal. The input is the shopping list, and the output is displayed on the terminal's display device. The user checks this shopping list through the terminal and uses it to help purchase the necessary items.

[0086] Step 7:

[0087] The server analyzes past consumption history. Using data analysis tools, it identifies consumption trends for each item. The input is consumption history, and the output calculates the optimal purchase time for each item.

[0088] Step 8:

[0089] The server notifies the user of the optimal time to purchase. The input for this notification is the calculated purchase time, and the output is an alert or notification message displayed on the user's device. This allows the user to purchase items in a planned manner.

[0090] (Application Example 1)

[0091] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] Current household inventory management and shopping planning present challenges, such as difficulty in understanding consumption patterns and determining the optimal timing for purchases. Furthermore, while online shopping on e-commerce platforms is widespread, integrating these services should enable more efficient shopping, but this is not currently being done sufficiently. In addition, automated purchase suggestions and notifications based on consumption history are not adequately implemented, indicating room for improvement in user convenience.

[0093] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0094] In this invention, the server includes means for analyzing item data acquired using image recognition means, means for updating possession information based on the analyzed item data, and means for generating shopping suggestions that recommend necessary items using the updated possession information and pre-registered cooking procedure data. This enables users to understand the state of their possessions in their home and efficiently purchase necessary items through an e-commerce platform at the optimal time.

[0095] "Image recognition means" refers to technology that automatically identifies the type and quantity of items from an image.

[0096] "Item data" refers to a collection of information about an item obtained through image recognition.

[0097] "Possessions information" is a database containing information about the types and quantities of items owned by the user.

[0098] "Cooking procedure data" refers to data that records information about recipes and necessary ingredients for various dishes.

[0099] "Shopping suggestions" are the output of an algorithm that generates a list of recommended items to purchase based on the user's possession information and cooking procedure data.

[0100] "Communication device" refers to a device or system that enables data transmission and reception with an information processing device.

[0101] An "external information processing device" refers to a device used to process user information and product data in external information systems such as e-commerce platforms.

[0102] An "e-commerce platform" refers to a website or application used for buying and selling goods and services over the internet.

[0103] This system consists of a user's terminal, a server, a communication device, and an external information processing device. The user takes pictures of household items using a smartphone or other device. The terminal has the appropriate application installed, and the captured images are sent from the terminal to the server.

[0104] The server uses image recognition AI (e.g., Google Cloud Vision API) to analyze the received images and identify the type and quantity of items based on the acquired item data. This data is stored in a database as personal belongings information.

[0105] The server matches updated inventory information with cooking procedure data to generate shopping suggestions. These suggestions are sent to the user's terminal via a communication device. Furthermore, the server uses an external information processing device to import product information obtained from e-commerce platforms, providing the user with a seamless shopping experience.

[0106] For example, if a user is making pasta, they might take a picture of their refrigerator with their smartphone. The server checks the inventory of tomato sauce and pasta and suggests any missing items through an e-commerce platform. This allows the user to purchase any missing items in a timely manner.

[0107] This system allows users to efficiently manage their belongings at home and optimize their shopping plans. It also utilizes a generative AI model to generate user-friendly prompts, an example of which is shown below.

[0108] Example of a prompt:

[0109] "Identify the food items in the refrigerator and generate prompts to order any missing items from an online shopping site."

[0110] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0111] Step 1:

[0112] The user takes photos of household items with their device.

[0113] The input is image data taken by the user.

[0114] The device transmits this image to the server via wireless or wired communication. The output is the image data transferred to the server.

[0115] Step 2:

[0116] The server passes the received image data to the image recognition AI.

[0117] The input is image data that exists on the server.

[0118] The server uses image recognition technology to identify the type and quantity of items and performs data processing to obtain the necessary item data.

[0119] The output is the analysis result, including the type and quantity of items.

[0120] Step 3:

[0121] The server saves the analysis results as inventory information in the database.

[0122] The input is item data based on the analysis results.

[0123] The server performs data calculations to update item information and saves the results to the database.

[0124] The output is the most recent inventory information.

[0125] Step 4:

[0126] The server compares the updated inventory information with the cooking procedure data.

[0127] The input consists of updated inventory information and cooking procedure data.

[0128] The server compares this data and performs data processing to identify missing items.

[0129] The output is a list of missing items.

[0130] Step 5:

[0131] The server generates shopping suggestions based on the missing items and sends them to the user's terminal via a communication device.

[0132] The input is a list of missing items.

[0133] The server uses this list to generate purchase suggestions and processes the data to create a shopping list of any missing items.

[0134] The output is in a format where shopping suggestions are displayed on the terminal.

[0135] Step 6:

[0136] The device displays the received shopping suggestions to the user.

[0137] The input is a shopping suggestion sent from the server.

[0138] The device performs an action that visually presents the latest status to the user.

[0139] The output is a visual purchase suggestion.

[0140] Step 7:

[0141] The server uses an external information processing device to retrieve product information from the e-commerce platform.

[0142] The input is data from an e-commerce platform.

[0143] The server collects information on potential purchase items and performs data processing to support seamless purchasing.

[0144] The output is a list of available products.

[0145] Step 8:

[0146] The server analyzes the user's consumption history and calculates the optimal time to make a purchase.

[0147] The input is the consumption history of the items you possess.

[0148] Based on this information, the server analyzes the user's consumption patterns and performs data calculations to suggest the best time to make a purchase.

[0149] The output indicates the recommended purchase time.

[0150] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0151] This invention is a system that streamlines user item management and provides more personalized shopping assistance by utilizing an emotion engine. Users use a terminal to take pictures of storage shelves and refrigerators in their homes and upload these images to a server. The server uses image recognition means to automatically identify the type and quantity of items in the images and obtain the necessary data. This data is recorded in a database and stored as possession information.

[0152] The updated inventory information is then cross-referenced with a pre-registered cooking method database. This database contains recipes and required ingredients for various dishes. Based on the inventory information and cooking method data, the server identifies missing items and generates an optimal shopping list for the user. This shopping list is then notified to the user via the terminal's display device.

[0153] In addition, the present invention incorporates an emotion engine into the system, which can analyze the user's emotions and customize shopping lists and notification content. Specifically, when a user expresses their emotions through voice input or text input from their terminal, the emotion engine analyzes that information and adjusts items on the shopping list or makes new suggestions. For example, if the user is feeling stressed, it can suggest additional items or recipes that can help them relax.

[0154] As a concrete example, when a user takes a picture of their refrigerator, their inventory information is updated. Later, when the user enters "I've been busy and stressed lately," the emotion engine analyzes this input and adds herbal teas and easy-to-make relaxing recipes to their shopping list. The user can then review the updated list and purchase items as needed, gaining a personalized consumption experience. In this way, the system simultaneously achieves effective inventory management and flexible suggestions tailored to the user's psychological needs.

[0155] The following describes the processing flow.

[0156] Step 1:

[0157] Users use their devices to take pictures of storage shelves and refrigerators in their homes. The captured images are uploaded from the device to the server.

[0158] Step 2:

[0159] The server receives the uploaded image and uses image recognition to analyze the type and quantity of items. This identifies each item in the image.

[0160] Step 3:

[0161] The server sends the analysis results to the database, updating it with inventory information. This update reflects the latest inventory status of all items owned by the user.

[0162] Step 4:

[0163] The server uses the updated inventory information and compares it with the cooking recipe database. The cooking recipe database contains pre-registered recipes and the necessary items. This comparison identifies any missing items.

[0164] Step 5:

[0165] The server identifies the missing items and generates a shopping list. The shopping list contains the necessary items and is sent to the user's device.

[0166] Step 6:

[0167] The terminal receives the shopping list from the server and notifies the user, displaying it on the terminal's display device. The user can then review the displayed list.

[0168] Step 7:

[0169] Users use their devices to input their emotions and desires via voice or text. This input information is then analyzed by an emotion engine.

[0170] Step 8:

[0171] The server uses an emotion engine to analyze the user's emotions and preferences, and customizes the shopping list and recipes accordingly. Based on the analysis, it adds new suggestions to the shopping list, such as herbal teas to help relieve stress.

[0172] Step 9:

[0173] The device will redisplay a newly customized shopping list and notify the user, allowing them to make appropriate shopping decisions based on the latest information.

[0174] (Example 2)

[0175] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0176] Current inventory management systems present a complex process for users to efficiently manage their household belongings and then list the items they need. Furthermore, systems capable of providing shopping suggestions tailored to individual users' emotions and circumstances are limited, making it difficult to address individual needs. This results in challenges in providing optimal support for users' purchasing experiences and lifestyles.

[0177] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0178] In this invention, the server includes means for analyzing item data acquired using image recognition means, means for updating management information based on the analyzed item data, means for generating a list of recommended items using the updated management information and pre-registered cooking method data, and means for analyzing emotions from voice or text input and customizing the suggestions in the list. This enables users to manage their items efficiently and receive personalized shopping suggestions that meet their emotions and individual needs.

[0179] "Image recognition means" refers to a technology that identifies items within an image based on a captured image and automatically determines their type and quantity.

[0180] "Item data" refers to information about individual items extracted by image recognition technology, and includes attributes such as type and quantity.

[0181] "Management information" refers to information in a database used to record and update the types and quantities of items owned by the user. This information reflects the latest status of owned items.

[0182] "Cooking method data" is a dataset containing various recipes and information on necessary ingredients, and is used to identify required items by associating them with information on possessions.

[0183] A "recommended list" is a list that provides users with a list of items they are missing or need, to help them with their purchasing decisions.

[0184] "Means of analyzing emotions and customizing suggestions" refers to technology that analyzes the user's emotional expressions in voice or text and adds new suggestions or adjustments to the list of recommendations based on the results.

[0185] This invention is a system that streamlines household item management and provides users with personalized shopping assistance. Implementation involves a user-owned device (such as a smartphone or tablet), a server, a database, and an emotion analysis engine.

[0186] The user takes photos of storage areas and refrigerators in their home using the device. The device compresses the captured images and uploads them to a server via the internet.

[0187] The server processes the received images. Specifically, it uses image processing and machine learning libraries such as OpenCV and TENSORFLOW® to identify items within the images. This extracts data on the type and quantity of items. This item data is stored in a database, keeping the user's possession status up-to-date as management information.

[0188] The server also cross-references the user's inventory information with a cooking method database and processes the data to identify missing items. This cooking method data contains recipes and required ingredients. As a result, the server generates a recommended list for the user, i.e., a shopping list. This generation process uses a generative AI model to provide customized recommendations based on prompt messages.

[0189] Furthermore, users can input their emotions via voice or text through their device. An emotion analysis engine analyzes this information and adjusts the shopping list based on the user's emotions. For example, for a user who is feeling stressed, it can suggest products with relaxing effects or easy-to-make recipes.

[0190] As a concrete example, an example of a prompt message might be, "If a user takes a picture of their refrigerator and feels stressed, what items or recipes should be suggested?" Based on this prompt, the system generates optimal suggestions tailored to the user's emotions and possessions. In this way, the present invention realizes personalized life support that suits the user.

[0191] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0192] Step 1:

[0193] The user takes photos of storage shelves or a refrigerator in their home using their device. The input is the captured image, and the output is the image file saved on the device. Specifically, the user operates the camera app to photograph the desired area.

[0194] Step 2:

[0195] The device uploads the captured image to the server. The input is the image file stored on the device, and the output is the image data sent to the server. Specifically, in this step, the device uses Wi-Fi or mobile data to transfer the image to the server via the internet.

[0196] Step 3:

[0197] The server analyzes the received image using an image recognition library. The input is the image data sent to the server, and the output is the analyzed item data. This item data is a list of items in the image based on their type and quantity, and specifically, item recognition and classification are performed using a convolutional neural network.

[0198] Step 4:

[0199] The server records the analyzed item data in the database. The input is the analyzed item data, and the output is the updated management information in the database. This step compares the data with the existing database to reflect any additions, deletions, or updates to items.

[0200] Step 5:

[0201] The server identifies missing items by cross-referencing updated management information with the cooking method database. Inputs are updated management information and cooking method data, and output is a recommended list (shopping list). This cross-referencing process determines whether all necessary ingredients are available and lists any missing items.

[0202] Step 6:

[0203] The user inputs their emotions into the device via voice or text. The input is the user's voice or text information, and the output is emotion data sent to the device. The specific action involves expressing emotions using a voice input application.

[0204] Step 7:

[0205] The server analyzes emotions expressed in voice or text and customizes the recommended list. The input is emotion data and an existing shopping list, and the output is a shopping list adjusted based on those emotions. This step uses a generative AI model to gain insights from emotions and optimize the list.

[0206] Step 8:

[0207] The terminal notifies the user of the final shopping list. The input is a customized shopping list received from the server, and the output is the list displayed on the user's terminal screen. Specifically, it uses a notification function to inform the user of the updates.

[0208] (Application Example 2)

[0209] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0210] In recent years, there has been a growing demand for efficient inventory management and personalized product recommendations based on user emotions. However, conventional systems have struggled to integrate inventory management and user emotion analysis, making it difficult to provide optimal recommendations for users. This invention aims to provide a system that utilizes image recognition technology and an emotion engine to achieve more accurate inventory management and personalized product recommendations.

[0211] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0212] In this invention, the server includes means for analyzing item data acquired using image recognition means, means for updating possession information based on the analyzed item data, and means for customizing shopping lists and product suggestions using an emotion engine that analyzes the user's emotional information. This enables efficient item management based on the user's actual possessions and personalized product suggestions tailored to the user's emotions at any given time.

[0213] "Image recognition means" refers to a technology that analyzes image data to automatically identify the type and quantity of items.

[0214] "Item data" refers to information about items obtained through image recognition, including their type and quantity.

[0215] "Possession information" refers to information stored in a database about items owned by the user.

[0216] "Cooking method data" refers to a database containing pre-registered information about various cooking recipes and necessary ingredients.

[0217] A "shopping list" is a list that identifies items that are lacking and lists recommended items that need to be purchased.

[0218] An "emotion engine" is a technology that analyzes a user's emotional information and customizes the information accordingly.

[0219] A "display device" is a device that provides users with generated information or shopping lists visually.

[0220] A "learning model" is a predictive model used to improve the accuracy of analysis in image recognition and sentiment analysis.

[0221] "Consumption history" refers to records of items a user has consumed in the past, and is used to predict future consumption patterns.

[0222] "Dynamic recommendations" refer to product recommendations and information provided that change in real time according to the user's current state and emotions.

[0223] This system consists of user terminals, servers, and various software that communicates with them. User terminals refer to devices such as smartphones and computers, while servers are computing devices located in the cloud or local network.

[0224] The server runs various programs, including image recognition and an emotion engine. The image recognition uses TensorFlow and PyTorch, which automatically analyzes item data from images of storage shelves and refrigerators taken by the user, identifying their type and quantity. The user's device is responsible for uploading these images to the server.

[0225] The emotion engine utilizes IBM Watson® and Google Cloud Natural Language API to analyze user emotions. This engine analyzes voice and text information entered by the user via the device to determine the user's emotional state.

[0226] The server updates the user's inventory information based on the analyzed item data and cross-references this information with cooking method data. During this process, it generates a shopping list recommending necessary items. This shopping list is customized according to the user's emotional state and displayed on the user's device.

[0227] For example, if a user enters "I've been busy lately and feeling stressed" into their device, the system analyzes this information using its emotion engine. Based on the analysis, products and recipes effective for stress relief are added to the shopping list. In this way, users can receive personalized suggestions.

[0228] An example of a prompt message could be, "I've been feeling exhausted from work lately. Please suggest some products that will help me relax." This allows for flexible product suggestions tailored to the user's psychological needs.

[0229] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0230] Step 1:

[0231] The user uses the device to take pictures of storage shelves and refrigerators in their home. The device then prepares the captured image data and sends it to the server.

[0232] Step 2:

[0233] The server applies image recognition to the received image data. Specifically, it uses TensorFlow to classify objects within the image and identify their type and quantity. Item data is generated as output of this process.

[0234] Step 3:

[0235] The server updates the inventory information in the database based on the analyzed item data. The updated inventory information is output as data indicating the current status of the user's items.

[0236] Step 4:

[0237] The user inputs emotional information via voice or text on the device. The device then sends this input data to the server.

[0238] Step 5:

[0239] The server uses an emotion engine to analyze emotional information received from the user. Specifically, it uses IBM Watson to quantify or categorize emotional states. The analysis results are output and used for subsequent processing.

[0240] Step 6:

[0241] The server uses the updated inventory information and sentiment analysis results to match them against its internal cooking recipe database. It identifies the necessary items and generates a customized shopping list based on a generative AI model.

[0242] Step 7:

[0243] The server sends the generated shopping list to the user's terminal. The user can visually review the shopping list on their terminal and decide on their next purchase action. This stage also includes information received as a result of prompt messages.

[0244] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0245] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0246] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0247] [Second Embodiment]

[0248] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0249] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0250] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0251] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0252] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0253] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0254] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0255] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0256] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0257] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0258] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0259] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0260] This invention provides a system that enables efficient item management and shopping assistance using a user's personal device. First, the user takes photos of storage shelves and refrigerators in their home using their device. These images are uploaded to a server, which uses image recognition to automatically identify items and analyze their type and quantity. The results of this analysis are stored in a database as item information.

[0261] The server uses the analyzed inventory information to access the user's registered cooking method database. This database contains recipes for various dishes, detailing the necessary ingredients and procedures. The server compares this cooking method data with the inventory information to generate a shopping list recommending necessary items. The generated shopping list is sent to the user's device, allowing them to easily view it through the device's display.

[0262] Furthermore, the server analyzes past consumption history to understand consumption patterns for each item. This data is used to calculate the optimal purchase time and notify the user's terminal. This allows users to purchase specific items before they run out, optimizing their shopping plans.

[0263] As a concrete example, suppose a user wants to check the ingredients for a pasta dish. The user takes a picture of the pantry with their device, and the server analyzes the image to determine the quantities of pasta and tomato sauce. The server then references a pasta recipe from a recipe database, and if any ingredients are missing, a list is generated and presented to the user. If the server determines that there is a shortage of tomato sauce, it notifies the user to purchase it and encourages them to take this into consideration when shopping next time. This system enables efficient food management within the household and allows for waste-free shopping planning.

[0264] The following describes the processing flow.

[0265] Step 1:

[0266] Users take photos of storage shelves or refrigerators using their devices. These images are then uploaded directly from the device to the server.

[0267] Step 2:

[0268] The server analyzes the received image data using image recognition technology. Through this analysis, it automatically identifies the type and quantity of items present in the image.

[0269] Step 3:

[0270] The server updates the database's inventory information based on the analysis results. This ensures that the latest inventory status of items in the user's home is reflected.

[0271] Step 4:

[0272] The server references the updated inventory information and compares it with the registered cooking method database. This identifies the items needed for the cooking method the user desires.

[0273] Step 5:

[0274] The server generates a shopping list based on the recommended items for the user. This list consists of the missing items.

[0275] Step 6:

[0276] The terminal notifies the user of the generated shopping list. The user can view this list on the terminal's display.

[0277] Step 7:

[0278] The server references consumption history information and calculates the optimal purchase time for each item based on consumption trends. This information is then communicated to the user, prompting them to purchase items at the appropriate time.

[0279] (Example 1)

[0280] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0281] In modern households, inventory management of items and shopping planning often rely on manual work, which is time-consuming and laborious. Therefore, there is a need for a mechanism to efficiently manage items and purchase necessary items at appropriate times. Additionally, there is a problem that it is difficult to know the optimal purchase timing considering past consumption history.

[0282] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0283] In this invention, the server includes means for analyzing item information acquired using image processing means, means for updating possession data based on the analyzed item information, means for generating a shopping list that proposes necessary items using the updated possession data and pre-registered cooking information, means for analyzing the consumption history recorded in the past and grasping the consumption trend of each item, and means for calculating the optimal purchase timing based on the consumption trend and notifying the user. As a result, the user can efficiently manage items, make a shopping plan without waste, and purchase necessary items at the optimal timing.

[0284] "Image processing means" is a technology that analyzes an image of an item and converts its content into digital data.

[0285] "Item information" is data including the type, quantity, and related attributes of an item acquired by image processing means.

[0286] "Means for analyzing" refers to a technology that generally refers to the entire process of carefully inspecting and organizing the acquired item information based on the acquired item information.

[0287] "Possession data" is information representing the possession status of items in the household and is data including the name and quantity of the items.

[0288] "Cooking information" includes recipe information related to various dishes and is information in which specific procedures and required ingredients are described.

[0289] The "Shopping List" is a list of items that the user should purchase in the future, providing organized information about the necessary items.

[0290] "Consumption history" refers to historical information about items a user has consumed, including data on past usage quantities and frequency.

[0291] The "optimal purchase timing" is calculated based on consumption trends and possession data, and represents the most appropriate time to purchase a new item.

[0292] This invention is a system that uses a user's personal device to efficiently manage the inventory of items in their home and assist with shopping. First, the user takes pictures of their refrigerator and storage shelves using the device's camera. These images are uploaded to a server via the network.

[0293] The server uses image processing technologies such as Google Cloud Vision API and Amazon Rekognition to analyze uploaded images. This automatically recognizes the type and quantity of items present in the image and extracts them as item information. This item information is then stored in a database and managed as household possession data.

[0294] Next, the server refers to a cooking information database that the user has registered in advance. This database contains recipe information for various dishes. The server compares this with the user's inventory data, identifies any missing ingredients, and generates a shopping list suggesting the necessary items. The generated shopping list is transferred to the terminal, and the user can view it through the display device.

[0295] Furthermore, the server analyzes past consumption history data to understand consumption trends for each item. Based on this data, the optimal purchase time is calculated and notified to the user. This allows users to purchase necessary items at the optimal time, enabling them to create shopping plans that reduce waste.

[0296] As a concrete example, consider a scenario where a user wants to make a pasta dish. The user takes a picture of their food shelf with their device, and the server analyzes the image to check the inventory of pasta and tomato sauce. The server then refers to a cooking information database and, if any of the ingredients needed for the pasta recipe are missing, it lists the missing items and presents them to the user. Additionally, if the tomato sauce inventory is low, the server notifies the user of the optimal time to purchase it, allowing them to incorporate this into their next shopping plan.

[0297] A concrete example of a prompt message would be, "Analyze the contents of your refrigerator and check if you have all the ingredients needed for the pasta recipe." This system allows users to manage their belongings more efficiently and shop more systematically.

[0298] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0299] Step 1:

[0300] The user uses their device's camera to take pictures of things like refrigerators and storage shelves in their home. These captured images become the input data. The images are saved on the device and then prepared to be uploaded to the server for processing.

[0301] Step 2:

[0302] The device uploads images it has taken to a server via the internet. The uploaded image data becomes the input, and the server receives this image data. In this process, a secure communication protocol such as HTTPS is used to maintain data security.

[0303] Step 3:

[0304] The server passes the received image data to the image processing means. Specifically, the image is analyzed using image recognition technologies such as Google Cloud Vision API and Amazon Rekognition. Through this analysis, the items in the image are recognized, and item information such as their types and quantities is output.

[0305] Step 4:

[0306] The server stores the analyzed item information in the database. The input is the analysis result, and as output, the household inventory data is updated. Through this database update, the types and quantities of items are accurately tracked.

[0307] Step 5:

[0308] The server refers to the registered cooking information database and compares it with the inventory data. The inputs for this comparison are the inventory data and the cooking information, and as output, the missing ingredients are listed. The server generates a shopping list based on this shortage information.

[0309] Step 6:

[0310] The server sends the generated shopping list to the terminal. The input is the shopping list, and the output will be displayed on the display device of the terminal. This helps the user to check the shopping list through the terminal and purchase the necessary items.

[0311] Step 7:

[0312] The server analyzes the past consumption history. Using data analysis tools, it identifies the consumption trends of each item. The input is the consumption history, and as output, the optimal purchase time for each item is calculated.

[0313] Step 8:

[0314] The server notifies the user of the optimal time to purchase. The input to this notification is the calculated purchase time, and the output is an alert or notification message displayed on the user's device. This allows the user to purchase items in a planned manner.

[0315] (Application Example 1)

[0316] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0317] Current household inventory management and shopping planning present challenges, such as difficulty in understanding consumption patterns and determining the optimal timing for purchases. Furthermore, while online shopping on e-commerce platforms is widespread, integrating these services should enable more efficient shopping, but this is not currently being done sufficiently. In addition, automated purchase suggestions and notifications based on consumption history are not adequately implemented, indicating room for improvement in user convenience.

[0318] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0319] In this invention, the server includes means for analyzing item data acquired using image recognition means, means for updating possession information based on the analyzed item data, and means for generating shopping suggestions that recommend necessary items using the updated possession information and pre-registered cooking procedure data. This enables users to understand the state of their possessions in their home and efficiently purchase necessary items through an e-commerce platform at the optimal time.

[0320] "Image recognition means" refers to technology that automatically identifies the type and quantity of items from an image.

[0321] "Item data" refers to a collection of information about an item obtained through image recognition.

[0322] "Possessions information" is a database containing information about the types and quantities of items owned by the user.

[0323] "Cooking procedure data" refers to data that records information about recipes and necessary ingredients for various dishes.

[0324] "Shopping suggestions" are the output of an algorithm that generates a list of recommended items to purchase based on the user's possession information and cooking procedure data.

[0325] "Communication device" refers to a device or system that enables data transmission and reception with an information processing device.

[0326] An "external information processing device" refers to a device used to process user information and product data in external information systems such as e-commerce platforms.

[0327] An "e-commerce platform" refers to a website or application used for buying and selling goods and services over the internet.

[0328] This system consists of a user's terminal, a server, a communication device, and an external information processing device. The user takes pictures of household items using a smartphone or other device. The terminal has the appropriate application installed, and the captured images are sent from the terminal to the server.

[0329] The server uses image recognition AI (e.g., Google Cloud Vision API) to analyze the received images and identify the type and quantity of items based on the acquired item data. This data is stored in a database as personal belongings information.

[0330] The server matches updated inventory information with cooking procedure data to generate shopping suggestions. These suggestions are sent to the user's terminal via a communication device. Furthermore, the server uses an external information processing device to import product information obtained from e-commerce platforms, providing the user with a seamless shopping experience.

[0331] For example, if a user is making pasta, they might take a picture of their refrigerator with their smartphone. The server checks the inventory of tomato sauce and pasta and suggests any missing items through an e-commerce platform. This allows the user to purchase any missing items in a timely manner.

[0332] This system allows users to efficiently manage their belongings at home and optimize their shopping plans. It also utilizes a generative AI model to generate user-friendly prompts, an example of which is shown below.

[0333] Example of a prompt:

[0334] "Identify the food items in the refrigerator and generate prompts to order any missing items from an online shopping site."

[0335] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0336] Step 1:

[0337] The user takes photos of household items with their device.

[0338] The input is image data captured by the user.

[0339] The device transmits this image to the server via wireless or wired communication. The output is the image data transferred to the server.

[0340] Step 2:

[0341] The server passes the received image data to the image recognition AI.

[0342] The input is image data that exists on the server.

[0343] The server uses image recognition technology to identify the type and quantity of items and performs data processing to obtain the necessary item data.

[0344] The output is the analysis result, including the type and quantity of items.

[0345] Step 3:

[0346] The server saves the analysis results as inventory information in the database.

[0347] The input is item data based on the analysis results.

[0348] The server performs data calculations to update item information and saves the results to the database.

[0349] The output is the most recent inventory information.

[0350] Step 4:

[0351] The server compares the updated inventory information with the cooking procedure data.

[0352] The input consists of updated inventory information and cooking procedure data.

[0353] The server compares this data and performs data processing to identify missing items.

[0354] The output is a list of missing items.

[0355] Step 5:

[0356] The server generates shopping suggestions based on the missing items and sends them to the user's terminal via a communication device.

[0357] The input is a list of missing items.

[0358] The server uses this list to generate purchase suggestions and processes the data to create a shopping list of any missing items.

[0359] The output is in a format where shopping suggestions are displayed on the terminal.

[0360] Step 6:

[0361] The device displays the received shopping suggestions to the user.

[0362] The input is shopping suggestions sent from the server.

[0363] The device performs an action that visually presents the latest status to the user.

[0364] The output is a visual purchase suggestion.

[0365] Step 7:

[0366] The server uses an external information processing device to retrieve product information from the e-commerce platform.

[0367] The input is data from an e-commerce platform.

[0368] The server collects information on potential purchase items and performs data processing to support seamless purchasing.

[0369] The output is a list of available products.

[0370] Step 8:

[0371] The server analyzes the user's consumption history and calculates the optimal time to make a purchase.

[0372] The input is the consumption history of the items you possess.

[0373] Based on this information, the server analyzes the user's consumption patterns and performs data calculations to suggest the best time to make a purchase.

[0374] The output indicates the recommended purchase time.

[0375] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0376] This invention is a system that streamlines user item management and provides more personalized shopping assistance by utilizing an emotion engine. Users use a terminal to take pictures of storage shelves and refrigerators in their homes and upload these images to a server. The server uses image recognition means to automatically identify the type and quantity of items in the images and obtain the necessary data. This data is recorded in a database and stored as possession information.

[0377] The updated inventory information is then cross-referenced with a pre-registered cooking method database. This database contains recipes and required ingredients for various dishes. Based on the inventory information and cooking method data, the server identifies missing items and generates an optimal shopping list for the user. This shopping list is then notified to the user via the terminal's display device.

[0378] In addition, the present invention incorporates an emotion engine into the system, which can analyze the user's emotions and customize shopping lists and notification content. Specifically, when a user expresses their emotions through voice input or text input from their terminal, the emotion engine analyzes that information and adjusts items on the shopping list or makes new suggestions. For example, if the user is feeling stressed, it can suggest additional items or recipes that can help them relax.

[0379] As a concrete example, when a user takes a picture of their refrigerator, their inventory information is updated. Later, when the user enters "I've been busy and stressed lately," the emotion engine analyzes this input and adds herbal teas and easy-to-make relaxing recipes to their shopping list. The user can then review the updated list and purchase items as needed, gaining a personalized consumption experience. In this way, the system simultaneously achieves effective inventory management and flexible suggestions tailored to the user's psychological needs.

[0380] The following describes the processing flow.

[0381] Step 1:

[0382] Users use their devices to take pictures of storage shelves and refrigerators in their homes. The captured images are uploaded from the device to the server.

[0383] Step 2:

[0384] The server receives the uploaded image and uses image recognition to analyze the type and quantity of items. This identifies each item in the image.

[0385] Step 3:

[0386] The server sends the analysis results to the database, updating it with inventory information. This update reflects the latest inventory status of all items owned by the user.

[0387] Step 4:

[0388] The server uses the updated inventory information and compares it with the cooking recipe database. The cooking recipe database contains pre-registered recipes and the necessary items. This comparison identifies any missing items.

[0389] Step 5:

[0390] The server identifies the missing items and generates a shopping list. The shopping list contains the necessary items and is sent to the user's device.

[0391] Step 6:

[0392] The terminal receives the shopping list from the server and notifies the user, displaying it on the terminal's display device. The user can then review the displayed list.

[0393] Step 7:

[0394] Users use their devices to input their emotions and desires via voice or text. This input information is then analyzed by an emotion engine.

[0395] Step 8:

[0396] The server uses an emotion engine to analyze the user's emotions and preferences, and customizes the shopping list and recipes accordingly. Based on the analysis, it adds new suggestions to the shopping list, such as herbal teas to help relieve stress.

[0397] Step 9:

[0398] The device will redisplay a newly customized shopping list and notify the user, allowing them to make appropriate shopping decisions based on the latest information.

[0399] (Example 2)

[0400] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0401] Current inventory management systems present a complex process for users to efficiently manage their household belongings and then list the items they need. Furthermore, systems capable of providing shopping suggestions tailored to individual users' emotions and circumstances are limited, making it difficult to address individual needs. This results in challenges in providing optimal support for users' purchasing experiences and lifestyles.

[0402] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0403] In this invention, the server includes means for analyzing item data acquired using image recognition means, means for updating management information based on the analyzed item data, means for generating a list of recommended items using the updated management information and pre-registered cooking method data, and means for analyzing emotions from voice or text input and customizing the suggestions in the list. This enables users to manage their items efficiently and receive personalized shopping suggestions that meet their emotions and individual needs.

[0404] "Image recognition means" refers to a technology that identifies items within an image based on a captured image and automatically determines their type and quantity.

[0405] "Item data" refers to information about individual items extracted by image recognition technology, and includes attributes such as type and quantity.

[0406] "Management information" refers to information in a database used to record and update the types and quantities of items owned by the user. This information reflects the latest status of owned items.

[0407] "Cooking method data" is a dataset containing various recipes and information on necessary ingredients, and is used to identify required items by associating them with information on possessions.

[0408] A "recommended list" is a list that provides users with a list of items they are missing or need, to help them with their purchasing decisions.

[0409] "Means of analyzing emotions and customizing suggestions" refers to technology that analyzes the user's emotional expressions in voice or text and adds new suggestions or adjustments to the recommended list based on the results.

[0410] This invention is a system that streamlines household item management and provides users with personalized shopping assistance. Implementation involves a user-owned device (such as a smartphone or tablet), a server, a database, and an emotion analysis engine.

[0411] The user takes photos of storage areas and refrigerators in their home using the device. The device compresses the captured images and uploads them to a server via the internet.

[0412] The server processes the received images. Specifically, it uses image processing and machine learning libraries such as OpenCV and TensorFlow to identify items within the images. This extracts data on the type and quantity of items. This item data is stored in a database, keeping the user's possession status up-to-date as management information.

[0413] The server also cross-references the user's inventory information with a cooking method database and processes the data to identify missing items. This cooking method data contains recipes and required ingredients. As a result, the server generates a recommended list for the user, i.e., a shopping list. This generation process uses a generative AI model to provide customized recommendations based on prompt messages.

[0414] Furthermore, users can input their emotions via voice or text through their device. An emotion analysis engine analyzes this information and adjusts the shopping list based on the user's emotions. For example, for a user who is feeling stressed, it can suggest products with relaxing effects or easy-to-make recipes.

[0415] As a concrete example, an example of a prompt message might be, "If a user takes a picture of their refrigerator and feels stressed, what items or recipes should be suggested?" Based on this prompt, the system generates optimal suggestions tailored to the user's emotions and possessions. In this way, the present invention realizes personalized life support that suits the user.

[0416] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0417] Step 1:

[0418] The user takes photos of storage shelves or a refrigerator in their home using their device. The input is the captured image, and the output is the image file saved on the device. Specifically, the user operates the camera app to photograph the desired area.

[0419] Step 2:

[0420] The device uploads the captured image to the server. The input is the image file stored on the device, and the output is the image data sent to the server. Specifically, in this step, the device uses Wi-Fi or mobile data to transfer the image to the server via the internet.

[0421] Step 3:

[0422] The server analyzes the received image using an image recognition library. The input is the image data sent to the server, and the output is the analyzed item data. This item data is a list of items in the image based on their type and quantity, and specifically, item recognition and classification are performed using a convolutional neural network.

[0423] Step 4:

[0424] The server records the analyzed item data in the database. The input is the analyzed item data, and the output is the updated management information in the database. This step compares the data with the existing database to reflect any additions, deletions, or updates to items.

[0425] Step 5:

[0426] The server identifies missing items by cross-referencing updated management information with the cooking method database. Inputs are updated management information and cooking method data, and output is a recommended list (shopping list). This cross-referencing process determines whether all necessary ingredients are available and lists any missing items.

[0427] Step 6:

[0428] The user inputs their emotions into the device via voice or text. The input is the user's voice or text information, and the output is emotion data sent to the device. The specific action involves expressing emotions using a voice input application.

[0429] Step 7:

[0430] The server analyzes emotions expressed in voice or text and customizes the recommended list. The input is emotion data and an existing shopping list, and the output is a shopping list adjusted based on those emotions. This step uses a generative AI model to gain insights from emotions and optimize the list.

[0431] Step 8:

[0432] The terminal notifies the user of the final shopping list. The input is a customized shopping list received from the server, and the output is the list displayed on the user's terminal screen. Specifically, it uses a notification function to inform the user of the updates.

[0433] (Application Example 2)

[0434] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0435] In recent years, there has been a growing demand for efficient inventory management and personalized product recommendations based on user emotions. However, conventional systems have struggled to integrate inventory management and user emotion analysis, making it difficult to provide optimal recommendations for users. This invention aims to provide a system that utilizes image recognition technology and an emotion engine to achieve more accurate inventory management and personalized product recommendations.

[0436] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0437] In this invention, the server includes means for analyzing item data acquired using image recognition means, means for updating possession information based on the analyzed item data, and means for customizing shopping lists and product suggestions using an emotion engine that analyzes the user's emotional information. This enables efficient item management based on the user's actual possessions and personalized product suggestions tailored to the user's emotions at any given time.

[0438] "Image recognition means" refers to a technology that analyzes image data to automatically identify the type and quantity of items.

[0439] "Item data" refers to information about items obtained through image recognition, including their type and quantity.

[0440] "Possession information" refers to information stored in a database about items owned by the user.

[0441] "Cooking method data" refers to a database containing pre-registered information about various cooking recipes and necessary ingredients.

[0442] A "shopping list" is a list that identifies items that are lacking and lists recommended items that need to be purchased.

[0443] An "emotion engine" is a technology that analyzes a user's emotional information and customizes the information accordingly.

[0444] A "display device" is a device that provides users with generated information or shopping lists visually.

[0445] A "learning model" is a predictive model used to improve the accuracy of analysis in image recognition and sentiment analysis.

[0446] "Consumption history" refers to records of items a user has consumed in the past, and is used to predict future consumption patterns.

[0447] "Dynamic recommendations" refer to product recommendations and information provided that change in real time according to the user's current state and emotions.

[0448] This system consists of user terminals, servers, and various software that communicates with them. User terminals refer to devices such as smartphones and computers, while servers are computing devices located in the cloud or on a local network.

[0449] The server runs various programs, including image recognition and an emotion engine. The image recognition uses TensorFlow and PyTorch, which automatically analyzes item data from images of storage shelves and refrigerators taken by the user, identifying their type and quantity. The user's device is responsible for uploading these images to the server.

[0450] The emotion engine uses IBM Watson and Google Cloud Natural Language API to analyze user emotions. This engine analyzes voice and text information entered by the user via the device to determine the user's emotional state.

[0451] The server updates the user's inventory information based on the analyzed item data and cross-references this information with cooking method data. During this process, it generates a shopping list recommending necessary items. This shopping list is customized according to the user's emotional state and displayed on the user's device.

[0452] For example, if a user enters "I've been busy lately and feeling stressed" into their device, the system analyzes this information using its emotion engine. Based on the analysis, products and recipes effective for stress relief are added to the shopping list. In this way, users can receive personalized suggestions.

[0453] An example of a prompt message could be, "I've been feeling exhausted from work lately. Please suggest some products that will help me relax." This allows for flexible product suggestions tailored to the user's psychological needs.

[0454] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0455] Step 1:

[0456] The user uses the device to take pictures of storage shelves and refrigerators in their home. The device then prepares the captured image data and sends it to the server.

[0457] Step 2:

[0458] The server applies image recognition to the received image data. Specifically, it uses TensorFlow to classify objects within the image and identify their type and quantity. Item data is generated as output of this process.

[0459] Step 3:

[0460] The server updates the inventory information in the database based on the analyzed item data. The updated inventory information is output as data indicating the current status of the user's items.

[0461] Step 4:

[0462] The user inputs emotional information via voice or text on the device. The device then sends this input data to the server.

[0463] Step 5:

[0464] The server uses an emotion engine to analyze emotional information received from the user. Specifically, it uses IBM Watson to quantify or categorize emotional states. The analysis results are output and used for subsequent processing.

[0465] Step 6:

[0466] The server uses updated inventory information and sentiment analysis results to match them against its internal cooking recipe database. It identifies the necessary items and generates a customized shopping list based on a generative AI model.

[0467] Step 7:

[0468] The server sends the generated shopping list to the user's terminal. The user can visually review the shopping list on their terminal and decide on their next purchase action. This stage also includes information received as a result of prompt messages.

[0469] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0470] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0471] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0472] [Third Embodiment]

[0473] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0474] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0475] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0476] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0477] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0478] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0479] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0480] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0481] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0482] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0483] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0484] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0485] This invention provides a system that enables efficient item management and shopping assistance using a user's personal device. First, the user takes photos of storage shelves and refrigerators in their home using their device. These images are uploaded to a server, which uses image recognition to automatically identify items and analyze their type and quantity. The results of this analysis are stored in a database as item information.

[0486] The server uses the analyzed inventory information to access the user's registered cooking method database. This database contains recipes for various dishes, detailing the necessary ingredients and procedures. The server compares this cooking method data with the inventory information to generate a shopping list recommending necessary items. The generated shopping list is sent to the user's device, allowing them to easily view it through the device's display.

[0487] Furthermore, the server analyzes past consumption history to understand consumption patterns for each item. This data is used to calculate the optimal purchase time and notify the user's terminal. This allows users to purchase specific items before they run out, optimizing their shopping plans.

[0488] As a concrete example, suppose a user wants to check the ingredients for a pasta dish. The user takes a picture of the pantry with their device, and the server analyzes the image to determine the quantities of pasta and tomato sauce. The server then references a pasta recipe from a recipe database, and if any ingredients are missing, a list is generated and presented to the user. If the server determines that there is a shortage of tomato sauce, it notifies the user to purchase it and encourages them to take this into consideration when shopping next time. This system enables efficient food management within the household and allows for waste-free shopping planning.

[0489] The following describes the processing flow.

[0490] Step 1:

[0491] Users take photos of storage shelves or refrigerators using their devices. These images are then uploaded directly from the device to the server.

[0492] Step 2:

[0493] The server analyzes the received image data using image recognition technology. Through this analysis, it automatically identifies the type and quantity of items present in the image.

[0494] Step 3:

[0495] The server updates the database's inventory information based on the analysis results. This ensures that the latest inventory status of items in the user's home is reflected.

[0496] Step 4:

[0497] The server references the updated inventory information and compares it with the registered cooking method database. This identifies the items needed for the cooking method the user desires.

[0498] Step 5:

[0499] The server generates a shopping list based on the recommended items for the user. This list consists of the missing items.

[0500] Step 6:

[0501] The terminal notifies the user of the generated shopping list. The user can view this list on the terminal's display.

[0502] Step 7:

[0503] The server references consumption history information and calculates the optimal purchase time for each item based on consumption trends. This information is then communicated to the user, prompting them to purchase items at the appropriate time.

[0504] (Example 1)

[0505] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0506] In modern households, inventory management and shopping planning tend to rely on manual processes, which are time-consuming and labor-intensive. Therefore, there is a need for systems that efficiently manage inventory and ensure that necessary items are purchased at the right time. Furthermore, determining the optimal purchase timing, taking past consumption history into account, presents a challenge.

[0507] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0508] In this invention, the server includes means for analyzing item information acquired using image processing means, means for updating possession data based on the analyzed item information, means for generating a shopping list that suggests necessary items using the updated possession data and pre-registered cooking information, means for analyzing past recorded consumption history to understand the consumption trend of each item, and means for calculating the optimal purchase time based on the consumption trend and notifying the user. This enables the user to efficiently manage their possessions, create an efficient shopping plan, and purchase necessary items at the optimal time.

[0509] "Image processing means" refers to technology that analyzes images of objects and converts their contents into digital data.

[0510] "Item information" refers to data that includes the type, quantity, and related attributes of an item, obtained by image processing means.

[0511] "Means of analysis" refers to the entire process of examining and organizing acquired item information in detail.

[0512] "Possessions data" refers to information that represents the possession status of items within a household, and includes data such as the names and quantities of items.

[0513] "Cooking information" refers to information that includes recipes for various dishes, detailing specific procedures and required ingredients.

[0514] The "Shopping List" is a list of items that the user should purchase in the future, providing organized information about the necessary items.

[0515] "Consumption history" refers to historical information about items a user has consumed, including data on past usage quantities and frequency.

[0516] The "optimal purchase timing" is calculated based on consumption trends and possession data, and represents the most appropriate time to purchase a new item.

[0517] This invention is a system that uses a user's personal device to efficiently manage the inventory of items in their home and assist with shopping. First, the user takes pictures of their refrigerator and storage shelves using the device's camera. These images are uploaded to a server via the network.

[0518] The server uses image processing technologies such as Google Cloud Vision API and Amazon Rekognition to analyze uploaded images. This automatically recognizes the type and quantity of items present in the image and extracts them as item information. This item information is then stored in a database and managed as household possession data.

[0519] Next, the server refers to a cooking information database that the user has registered in advance. This database contains recipe information for various dishes. The server compares this with the user's inventory data, identifies any missing ingredients, and generates a shopping list suggesting the necessary items. The generated shopping list is transferred to the terminal, and the user can view it through the display device.

[0520] Furthermore, the server analyzes past consumption history data to understand consumption trends for each item. Based on this data, the optimal purchase time is calculated and notified to the user. This allows users to purchase necessary items at the optimal time, enabling them to create shopping plans that reduce waste.

[0521] As a concrete example, consider a scenario where a user wants to make a pasta dish. The user takes a picture of their food shelf with their device, and the server analyzes the image to check the inventory of pasta and tomato sauce. The server then refers to a cooking information database and, if any of the ingredients needed for the pasta recipe are missing, it lists the missing items and presents them to the user. Additionally, if the tomato sauce inventory is low, the server notifies the user of the optimal time to purchase it, allowing them to incorporate this into their next shopping plan.

[0522] A concrete example of a prompt message would be, "Analyze the contents of your refrigerator and check if you have all the ingredients needed for the pasta recipe." This system allows users to manage their belongings more efficiently and shop more systematically.

[0523] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0524] Step 1:

[0525] The user uses their device's camera to take pictures of things like refrigerators and storage shelves in their home. These captured images become the input data. The images are saved on the device and then prepared to be uploaded to the server for processing.

[0526] Step 2:

[0527] The device uploads images it has taken to a server via the internet. The uploaded image data becomes the input, and the server receives this image data. In this process, a secure communication protocol such as HTTPS is used to maintain data security.

[0528] Step 3:

[0529] The server passes the received image data to an image processing system. Specifically, it analyzes the image using image recognition technologies such as Google Cloud Vision API or Amazon Rekognition. Through this analysis, objects in the image are recognized, and information about the objects, such as their type and quantity, is output.

[0530] Step 4:

[0531] The server stores the analyzed item information in a database. The input is the analysis results, and the output is an updated database of household possessions. This database update ensures accurate tracking of the type and quantity of items.

[0532] Step 5:

[0533] The server refers to the registered cooking information database and compares it with the inventory data. The input for this comparison is the inventory data and cooking information, and the output is a list of missing ingredients. The server then generates a shopping list based on this missing information.

[0534] Step 6:

[0535] The server sends the generated shopping list to the terminal. The input is the shopping list, and the output is displayed on the terminal's display device. The user checks this shopping list through the terminal and uses it to help purchase the necessary items.

[0536] Step 7:

[0537] The server analyzes past consumption history. Using data analysis tools, it identifies consumption trends for each item. The input is consumption history, and the output calculates the optimal purchase time for each item.

[0538] Step 8:

[0539] The server notifies the user of the optimal time to purchase. The input to this notification is the calculated purchase time, and the output is an alert or notification message displayed on the user's device. This allows the user to purchase items in a planned manner.

[0540] (Application Example 1)

[0541] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0542] Current household inventory management and shopping planning present challenges, such as difficulty in understanding consumption patterns and determining the optimal timing for purchases. Furthermore, while online shopping on e-commerce platforms is widespread, integrating these services should enable more efficient shopping, but this is not currently being done sufficiently. In addition, automated purchase suggestions and notifications based on consumption history are not adequately implemented, indicating room for improvement in user convenience.

[0543] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0544] In this invention, the server includes means for analyzing item data acquired using image recognition means, means for updating possession information based on the analyzed item data, and means for generating shopping suggestions that recommend necessary items using the updated possession information and pre-registered cooking procedure data. This enables users to understand the state of their possessions in their home and efficiently purchase necessary items through an e-commerce platform at the optimal time.

[0545] "Image recognition means" refers to technology that automatically identifies the type and quantity of items from an image.

[0546] "Item data" refers to a collection of information about an item obtained through image recognition.

[0547] "Possessions information" is a database containing information about the types and quantities of items owned by the user.

[0548] "Cooking procedure data" refers to data that records information about recipes and necessary ingredients for various dishes.

[0549] "Shopping suggestions" are the output of an algorithm that generates a list of recommended items to purchase based on the user's possession information and cooking procedure data.

[0550] "Communication device" refers to a device or system that enables data transmission and reception with an information processing device.

[0551] An "external information processing device" refers to a device used to process user information and product data in external information systems such as e-commerce platforms.

[0552] An "e-commerce platform" refers to a website or application used for buying and selling goods and services over the internet.

[0553] This system consists of a user's terminal, a server, a communication device, and an external information processing device. The user takes pictures of household items using a smartphone or other device. The terminal has the appropriate application installed, and the captured images are sent from the terminal to the server.

[0554] The server uses image recognition AI (e.g., Google Cloud Vision API) to analyze the received images and identify the type and quantity of items based on the acquired item data. This data is stored in a database as personal belongings information.

[0555] The server matches updated inventory information with cooking procedure data to generate shopping suggestions. These suggestions are sent to the user's terminal via a communication device. Furthermore, the server uses an external information processing device to import product information obtained from e-commerce platforms, providing the user with a seamless shopping experience.

[0556] For example, if a user is making pasta, they might take a picture of their refrigerator with their smartphone. The server checks the inventory of tomato sauce and pasta and suggests any missing items through an e-commerce platform. This allows the user to purchase any missing items in a timely manner.

[0557] This system allows users to efficiently manage their belongings at home and optimize their shopping plans. It also utilizes a generative AI model to generate user-friendly prompts, an example of which is shown below.

[0558] Example of a prompt:

[0559] "Identify the food items in the refrigerator and generate prompts to order any missing items from an online shopping site."

[0560] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0561] Step 1:

[0562] The user takes photos of household items with their device.

[0563] The input is image data captured by the user.

[0564] The device transmits this image to the server via wireless or wired communication. The output is the image data transferred to the server.

[0565] Step 2:

[0566] The server passes the received image data to the image recognition AI.

[0567] The input is image data that exists on the server.

[0568] The server uses image recognition technology to identify the type and quantity of items and performs data processing to obtain the necessary item data.

[0569] The output is the analysis result, including the type and quantity of items.

[0570] Step 3:

[0571] The server saves the analysis results as inventory information in the database.

[0572] The input is item data based on the analysis results.

[0573] The server performs data calculations to update item information and saves the results to the database.

[0574] The output is the most recent inventory information.

[0575] Step 4:

[0576] The server compares the updated inventory information with the cooking procedure data.

[0577] The input consists of updated inventory information and cooking procedure data.

[0578] The server compares this data and performs data processing to identify missing items.

[0579] The output is a list of missing items.

[0580] Step 5:

[0581] The server generates shopping suggestions based on the missing items and sends them to the user's terminal via a communication device.

[0582] The input is a list of missing items.

[0583] The server uses this list to generate purchase suggestions and processes the data to create a shopping list of any missing items.

[0584] The output is in a format where shopping suggestions are displayed on the terminal.

[0585] Step 6:

[0586] The device displays the received shopping suggestions to the user.

[0587] The input is shopping suggestions sent from the server.

[0588] The device performs an action that visually presents the latest status to the user.

[0589] The output is a visual purchase suggestion.

[0590] Step 7:

[0591] The server uses an external information processing device to retrieve product information from the e-commerce platform.

[0592] The input is data from an e-commerce platform.

[0593] The server collects information on potential purchase items and performs data processing to support seamless purchasing.

[0594] The output is a list of available products.

[0595] Step 8:

[0596] The server analyzes the user's consumption history and calculates the optimal time to make a purchase.

[0597] The input is the consumption history of the items you possess.

[0598] Based on this information, the server analyzes the user's consumption patterns and performs data calculations to suggest the best time to make a purchase.

[0599] The output indicates the recommended purchase time.

[0600] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0601] This invention is a system that streamlines user item management and provides more personalized shopping assistance by utilizing an emotion engine. Users use a terminal to take pictures of storage shelves and refrigerators in their homes and upload these images to a server. The server uses image recognition means to automatically identify the type and quantity of items in the images and obtain the necessary data. This data is recorded in a database and stored as possession information.

[0602] The updated inventory information is then cross-referenced with a pre-registered cooking method database. This database contains recipes and required ingredients for various dishes. Based on the inventory information and cooking method data, the server identifies missing items and generates an optimal shopping list for the user. This shopping list is then notified to the user via the terminal's display device.

[0603] In addition, the present invention incorporates an emotion engine into the system, which can analyze the user's emotions and customize shopping lists and notification content. Specifically, when a user expresses their emotions through voice input or text input from their terminal, the emotion engine analyzes that information and adjusts items on the shopping list or makes new suggestions. For example, if the user is feeling stressed, it can suggest additional items or recipes that can help them relax.

[0604] As a concrete example, when a user takes a picture of their refrigerator, their inventory information is updated. Later, when the user enters "I've been busy and stressed lately," the emotion engine analyzes this input and adds herbal teas and easy-to-make relaxing recipes to their shopping list. The user can then review the updated list and purchase items as needed, gaining a personalized consumption experience. In this way, the system simultaneously achieves effective inventory management and flexible suggestions tailored to the user's psychological needs.

[0605] The following describes the processing flow.

[0606] Step 1:

[0607] Users use their devices to take pictures of storage shelves and refrigerators in their homes. The captured images are uploaded from the device to the server.

[0608] Step 2:

[0609] The server receives the uploaded image and uses image recognition to analyze the type and quantity of items. This identifies each item in the image.

[0610] Step 3:

[0611] The server sends the analysis results to the database, updating it with inventory information. This update reflects the latest inventory status of all items owned by the user.

[0612] Step 4:

[0613] The server uses the updated inventory information and compares it with the cooking recipe database. The cooking recipe database contains pre-registered recipes and the necessary items. This comparison identifies any missing items.

[0614] Step 5:

[0615] The server identifies the missing items and generates a shopping list. The shopping list contains the necessary items and is sent to the user's device.

[0616] Step 6:

[0617] The terminal receives the shopping list from the server and notifies the user, displaying it on the terminal's display device. The user can then review the displayed list.

[0618] Step 7:

[0619] Users use their devices to input their emotions and desires via voice or text. This input information is then analyzed by an emotion engine.

[0620] Step 8:

[0621] The server uses an emotion engine to analyze the user's emotions and preferences, and customizes the shopping list and recipes accordingly. Based on the analysis, it adds new suggestions to the shopping list, such as herbal teas to help relieve stress.

[0622] Step 9:

[0623] The device will redisplay a newly customized shopping list and notify the user, allowing them to make appropriate shopping decisions based on the latest information.

[0624] (Example 2)

[0625] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0626] Current inventory management systems present a complex process for users to efficiently manage their household belongings and then list the items they need. Furthermore, systems capable of providing shopping suggestions tailored to individual users' emotions and circumstances are limited, making it difficult to address individual needs. This results in challenges in providing optimal support for users' purchasing experiences and lifestyles.

[0627] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0628] In this invention, the server includes means for analyzing item data acquired using image recognition means, means for updating management information based on the analyzed item data, means for generating a list of recommended items using the updated management information and pre-registered cooking method data, and means for analyzing emotions from voice or text input and customizing the suggestions in the list. This enables users to manage their items efficiently and receive personalized shopping suggestions that meet their emotions and individual needs.

[0629] "Image recognition means" refers to a technology that identifies items within an image based on a captured image and automatically determines their type and quantity.

[0630] "Item data" refers to information about individual items extracted by image recognition technology, and includes attributes such as type and quantity.

[0631] "Management information" refers to information in a database used to record and update the types and quantities of items owned by the user. This information reflects the latest status of owned items.

[0632] "Cooking method data" is a dataset containing various recipes and information on necessary ingredients, and is used to identify required items by associating them with information on possessions.

[0633] A "recommended list" is a list that provides users with a list of items they are missing or need, to help them with their purchasing decisions.

[0634] "Means of analyzing emotions and customizing suggestions" refers to technology that analyzes the user's emotional expressions in voice or text and adds new suggestions or adjustments to the recommended list based on the results.

[0635] This invention is a system that streamlines household item management and provides users with personalized shopping assistance. Implementation involves a user-owned device (such as a smartphone or tablet), a server, a database, and an emotion analysis engine.

[0636] The user takes photos of storage areas and refrigerators in their home using the device. The device compresses the captured images and uploads them to a server via the internet.

[0637] The server processes the received images. Specifically, it uses image processing and machine learning libraries such as OpenCV and TensorFlow to identify items within the images. This extracts data on the type and quantity of items. This item data is stored in a database, keeping the user's possession status up-to-date as management information.

[0638] The server also cross-references the user's inventory information with a cooking method database and processes the data to identify missing items. This cooking method data contains recipes and required ingredients. As a result, the server generates a recommended list for the user, i.e., a shopping list. This generation process uses a generative AI model to provide customized recommendations based on prompt messages.

[0639] Furthermore, users can input their emotions via voice or text through their device. An emotion analysis engine analyzes this information and adjusts the shopping list based on the user's emotions. For example, for a user who is feeling stressed, it can suggest products with relaxing effects or easy-to-make recipes.

[0640] As a concrete example, an example of a prompt message might be, "If a user takes a picture of their refrigerator and feels stressed, what items or recipes should be suggested?" Based on this prompt, the system generates optimal suggestions tailored to the user's emotions and possessions. In this way, the present invention realizes personalized life support that suits the user.

[0641] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0642] Step 1:

[0643] The user takes photos of storage shelves or a refrigerator in their home using their device. The input is the captured image, and the output is the image file saved on the device. Specifically, the user operates the camera app to photograph the desired area.

[0644] Step 2:

[0645] The device uploads the captured image to the server. The input is the image file stored on the device, and the output is the image data sent to the server. Specifically, in this step, the device uses Wi-Fi or mobile data to transfer the image to the server via the internet.

[0646] Step 3:

[0647] The server analyzes the received image using an image recognition library. The input is the image data sent to the server, and the output is the analyzed item data. This item data is a list of items in the image based on their type and quantity, and specifically, item recognition and classification are performed using a convolutional neural network.

[0648] Step 4:

[0649] The server records the analyzed item data in the database. The input is the analyzed item data, and the output is the updated management information in the database. This step compares the data with the existing database to reflect any additions, deletions, or updates to items.

[0650] Step 5:

[0651] The server identifies missing items by cross-referencing updated management information with the cooking method database. Inputs are updated management information and cooking method data, and output is a recommended list (shopping list). This cross-referencing process determines whether all necessary ingredients are available and lists any missing items.

[0652] Step 6:

[0653] The user inputs their emotions into the device via voice or text. The input is the user's voice or text information, and the output is emotion data sent to the device. The specific action involves expressing emotions using a voice input application.

[0654] Step 7:

[0655] The server analyzes emotions expressed in voice or text and customizes the recommended list. The input is emotion data and an existing shopping list, and the output is a shopping list adjusted based on those emotions. This step uses a generative AI model to gain insights from emotions and optimize the list.

[0656] Step 8:

[0657] The terminal notifies the user of the final shopping list. The input is a customized shopping list received from the server, and the output is the list displayed on the user's terminal screen. Specifically, it uses a notification function to inform the user of the updates.

[0658] (Application Example 2)

[0659] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0660] In recent years, there has been a growing demand for efficient inventory management and personalized product recommendations based on user emotions. However, conventional systems have struggled to integrate inventory management and user emotion analysis, making it difficult to provide optimal recommendations for users. This invention aims to provide a system that utilizes image recognition technology and an emotion engine to achieve more accurate inventory management and personalized product recommendations.

[0661] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0662] In this invention, the server includes means for analyzing item data acquired using image recognition means, means for updating possession information based on the analyzed item data, and means for customizing shopping lists and product suggestions using an emotion engine that analyzes the user's emotional information. This enables efficient item management based on the user's actual possessions and personalized product suggestions tailored to the user's emotions at any given time.

[0663] "Image recognition means" refers to a technology that analyzes image data to automatically identify the type and quantity of items.

[0664] "Item data" refers to information about items obtained through image recognition, including their type and quantity.

[0665] "Possession information" refers to information stored in a database about items owned by the user.

[0666] "Cooking method data" refers to a database containing pre-registered information about various cooking recipes and necessary ingredients.

[0667] A "shopping list" is a list that identifies items that are lacking and lists recommended items that need to be purchased.

[0668] An "emotion engine" is a technology that analyzes a user's emotional information and customizes the information accordingly.

[0669] A "display device" is a device that provides users with generated information or shopping lists visually.

[0670] A "learning model" is a predictive model used to improve the accuracy of analysis in image recognition and sentiment analysis.

[0671] "Consumption history" refers to records of items a user has consumed in the past, and is used to predict future consumption patterns.

[0672] "Dynamic recommendations" refer to product recommendations and information provided that change in real time according to the user's current state and emotions.

[0673] This system consists of user terminals, servers, and various software that communicates with them. User terminals refer to devices such as smartphones and computers, while servers are computing devices located in the cloud or on a local network.

[0674] The server runs various programs, including image recognition and an emotion engine. The image recognition uses TensorFlow and PyTorch, which automatically analyzes item data from images of storage shelves and refrigerators taken by the user, identifying their type and quantity. The user's device is responsible for uploading these images to the server.

[0675] The emotion engine uses IBM Watson and Google Cloud Natural Language API to analyze user emotions. This engine analyzes voice and text information entered by the user via the device to determine the user's emotional state.

[0676] The server updates the user's inventory information based on the analyzed item data and cross-references this information with cooking method data. During this process, it generates a shopping list recommending necessary items. This shopping list is customized according to the user's emotional state and displayed on the user's device.

[0677] For example, if a user enters "I've been busy lately and feeling stressed" into their device, the system analyzes this information using its emotion engine. Based on the analysis, products and recipes effective for stress relief are added to the shopping list. In this way, users can receive personalized suggestions.

[0678] An example of a prompt message could be, "I've been feeling exhausted from work lately. Please suggest some products that will help me relax." This allows for flexible product suggestions tailored to the user's psychological needs.

[0679] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0680] Step 1:

[0681] The user uses the device to take pictures of storage shelves and refrigerators in their home. The device then prepares the captured image data and sends it to the server.

[0682] Step 2:

[0683] The server applies image recognition to the received image data. Specifically, it uses TensorFlow to classify objects within the image and identify their type and quantity. Item data is generated as output of this process.

[0684] Step 3:

[0685] The server updates the inventory information in the database based on the analyzed item data. The updated inventory information is output as data indicating the current status of the user's items.

[0686] Step 4:

[0687] The user inputs emotional information via voice or text on the device. The device then sends this input data to the server.

[0688] Step 5:

[0689] The server uses an emotion engine to analyze emotional information received from the user. Specifically, it uses IBM Watson to quantify or categorize emotional states. The analysis results are output and used for subsequent processing.

[0690] Step 6:

[0691] The server uses updated inventory information and sentiment analysis results to match them against its internal cooking recipe database. It identifies the necessary items and generates a customized shopping list based on a generative AI model.

[0692] Step 7:

[0693] The server sends the generated shopping list to the user's terminal. The user can visually review the shopping list on their terminal and decide on their next purchase action. This stage also includes information received as a result of prompt messages.

[0694] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0695] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0696] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0697] [Fourth Embodiment]

[0698] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0699] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0700] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0701] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0702] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0703] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0704] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0705] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0706] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0707] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0708] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0709] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0710] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0711] This invention provides a system that enables efficient item management and shopping assistance using a user's personal device. First, the user takes photos of storage shelves and refrigerators in their home using their device. These images are uploaded to a server, which uses image recognition to automatically identify items and analyze their type and quantity. The results of this analysis are stored in a database as item information.

[0712] The server uses the analyzed inventory information to access the user's registered cooking method database. This database contains recipes for various dishes, detailing the necessary ingredients and procedures. The server compares this cooking method data with the inventory information to generate a shopping list recommending necessary items. The generated shopping list is sent to the user's device, allowing them to easily view it through the device's display.

[0713] Furthermore, the server analyzes past consumption history to understand consumption patterns for each item. This data is used to calculate the optimal purchase time and notify the user's terminal. This allows users to purchase specific items before they run out, optimizing their shopping plans.

[0714] As a concrete example, suppose a user wants to check the ingredients for a pasta dish. The user takes a picture of the pantry with their device, and the server analyzes the image to determine the quantities of pasta and tomato sauce. The server then references a pasta recipe from a recipe database, and if any ingredients are missing, a list is generated and presented to the user. If the server determines that there is a shortage of tomato sauce, it notifies the user to purchase it and encourages them to take this into consideration when shopping next time. This system enables efficient food management within the household and allows for waste-free shopping planning.

[0715] The following describes the processing flow.

[0716] Step 1:

[0717] Users take photos of storage shelves or refrigerators using their devices. These images are then uploaded directly from the device to the server.

[0718] Step 2:

[0719] The server analyzes the received image data using image recognition technology. Through this analysis, it automatically identifies the type and quantity of items present in the image.

[0720] Step 3:

[0721] The server updates the database's inventory information based on the analysis results. This ensures that the latest inventory status of items in the user's home is reflected.

[0722] Step 4:

[0723] The server references the updated inventory information and compares it with the registered cooking method database. This identifies the items needed for the cooking method the user desires.

[0724] Step 5:

[0725] The server generates a shopping list based on the recommended items for the user. This list consists of the missing items.

[0726] Step 6:

[0727] The terminal notifies the user of the generated shopping list. The user can view this list on the terminal's display.

[0728] Step 7:

[0729] The server references consumption history information and calculates the optimal purchase time for each item based on consumption trends. This information is then communicated to the user, prompting them to purchase items at the appropriate time.

[0730] (Example 1)

[0731] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0732] In modern households, inventory management and shopping planning tend to rely on manual processes, which are time-consuming and labor-intensive. Therefore, there is a need for systems that efficiently manage inventory and ensure that necessary items are purchased at the right time. Furthermore, determining the optimal purchase timing, taking past consumption history into account, presents a challenge.

[0733] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0734] In this invention, the server includes means for analyzing item information acquired using image processing means, means for updating possession data based on the analyzed item information, means for generating a shopping list that suggests necessary items using the updated possession data and pre-registered cooking information, means for analyzing past recorded consumption history to understand the consumption trend of each item, and means for calculating the optimal purchase time based on the consumption trend and notifying the user. This enables the user to efficiently manage their possessions, create an efficient shopping plan, and purchase necessary items at the optimal time.

[0735] "Image processing means" refers to technology that analyzes images of objects and converts their contents into digital data.

[0736] "Item information" refers to data that includes the type, quantity, and related attributes of an item, obtained by image processing means.

[0737] "Means of analysis" refers to the entire process of examining and organizing acquired item information in detail.

[0738] "Possessions data" refers to information that represents the possession status of items within a household, and includes data such as the names and quantities of items.

[0739] "Cooking information" refers to information that includes recipes for various dishes, detailing specific procedures and required ingredients.

[0740] The "Shopping List" is a list of items that the user should purchase in the future, providing organized information about the necessary items.

[0741] "Consumption history" refers to historical information about items a user has consumed, including data on past usage quantities and frequency.

[0742] The "optimal purchase timing" is calculated based on consumption trends and possession data, and represents the most appropriate time to purchase a new item.

[0743] This invention is a system that uses a user's personal device to efficiently manage the inventory of items in their home and assist with shopping. First, the user takes pictures of their refrigerator and storage shelves using the device's camera. These images are uploaded to a server via the network.

[0744] The server uses image processing technologies such as Google Cloud Vision API and Amazon Rekognition to analyze uploaded images. This automatically recognizes the type and quantity of items present in the image and extracts them as item information. This item information is then stored in a database and managed as household possession data.

[0745] Next, the server refers to a cooking information database that the user has registered in advance. This database contains recipe information for various dishes. The server compares this with the user's inventory data, identifies any missing ingredients, and generates a shopping list suggesting the necessary items. The generated shopping list is transferred to the terminal, and the user can view it through the display device.

[0746] Furthermore, the server analyzes past consumption history data to understand consumption trends for each item. Based on this data, the optimal purchase time is calculated and notified to the user. This allows users to purchase necessary items at the optimal time, enabling them to create shopping plans that reduce waste.

[0747] As a concrete example, consider a scenario where a user wants to make a pasta dish. The user takes a picture of their food shelf with their device, and the server analyzes the image to check the inventory of pasta and tomato sauce. The server then refers to a cooking information database and, if any of the ingredients needed for the pasta recipe are missing, it lists the missing items and presents them to the user. Additionally, if the tomato sauce inventory is low, the server notifies the user of the optimal time to purchase it, allowing them to incorporate this into their next shopping plan.

[0748] A concrete example of a prompt message would be, "Analyze the contents of your refrigerator and check if you have all the ingredients needed for the pasta recipe." This system allows users to manage their belongings more efficiently and shop more systematically.

[0749] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0750] Step 1:

[0751] The user uses their device's camera to take pictures of things like refrigerators and storage shelves in their home. These captured images become the input data. The images are saved on the device and then prepared to be uploaded to the server for processing.

[0752] Step 2:

[0753] The device uploads images it has taken to a server via the internet. The uploaded image data becomes the input, and the server receives this image data. In this process, a secure communication protocol such as HTTPS is used to maintain data security.

[0754] Step 3:

[0755] The server passes the received image data to an image processing system. Specifically, it analyzes the image using image recognition technologies such as Google Cloud Vision API or Amazon Rekognition. Through this analysis, objects in the image are recognized, and information about the objects, such as their type and quantity, is output.

[0756] Step 4:

[0757] The server stores the analyzed item information in a database. The input is the analysis results, and the output is an updated database of household possessions. This database update ensures accurate tracking of the type and quantity of items.

[0758] Step 5:

[0759] The server refers to the registered cooking information database and compares it with the inventory data. The input for this comparison is the inventory data and cooking information, and the output is a list of missing ingredients. The server then generates a shopping list based on this missing information.

[0760] Step 6:

[0761] The server sends the generated shopping list to the terminal. The input is the shopping list, and the output is displayed on the terminal's display device. The user checks this shopping list through the terminal and uses it to help purchase the necessary items.

[0762] Step 7:

[0763] The server analyzes past consumption history. Using data analysis tools, it identifies consumption trends for each item. The input is consumption history, and the output calculates the optimal purchase time for each item.

[0764] Step 8:

[0765] The server notifies the user of the optimal time to purchase. The input for this notification is the calculated purchase time, and the output is an alert or notification message displayed on the user's device. This allows the user to purchase items in a planned manner.

[0766] (Application Example 1)

[0767] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0768] Current household inventory management and shopping planning present challenges, such as difficulty in understanding consumption patterns and determining the optimal timing for purchases. Furthermore, while online shopping on e-commerce platforms is widespread, integrating these services should enable more efficient shopping, but this is not currently being done sufficiently. In addition, automated purchase suggestions and notifications based on consumption history are not adequately implemented, indicating room for improvement in user convenience.

[0769] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0770] In this invention, the server includes means for analyzing item data acquired using image recognition means, means for updating possession information based on the analyzed item data, and means for generating shopping suggestions that recommend necessary items using the updated possession information and pre-registered cooking procedure data. This enables users to understand the state of their possessions in their home and efficiently purchase necessary items through an e-commerce platform at the optimal time.

[0771] "Image recognition means" refers to technology that automatically identifies the type and quantity of items from an image.

[0772] "Item data" refers to a collection of information about an item obtained through image recognition.

[0773] "Possessions information" is a database containing information about the types and quantities of items owned by the user.

[0774] "Cooking procedure data" refers to data that records information about recipes and necessary ingredients for various dishes.

[0775] "Shopping suggestions" are the output of an algorithm that generates a list of recommended items to purchase based on the user's possession information and cooking procedure data.

[0776] "Communication device" refers to a device or system that enables data transmission and reception with an information processing device.

[0777] An "external information processing device" refers to a device used to process user information and product data in external information systems such as e-commerce platforms.

[0778] An "e-commerce platform" refers to a website or application used for buying and selling goods and services over the internet.

[0779] This system consists of a user's terminal, a server, a communication device, and an external information processing device. The user takes pictures of household items using a smartphone or other device. The terminal has the appropriate application installed, and the captured images are sent from the terminal to the server.

[0780] The server uses image recognition AI (e.g., Google Cloud Vision API) to analyze the received images and identify the type and quantity of items based on the acquired item data. This data is stored in a database as personal belongings information.

[0781] The server matches updated inventory information with cooking procedure data to generate shopping suggestions. These suggestions are sent to the user's terminal via a communication device. Furthermore, the server uses an external information processing device to import product information obtained from e-commerce platforms, providing the user with a seamless shopping experience.

[0782] For example, if a user is making pasta, they might take a picture of their refrigerator with their smartphone. The server checks the inventory of tomato sauce and pasta and suggests any missing items through an e-commerce platform. This allows the user to purchase any missing items in a timely manner.

[0783] This system allows users to efficiently manage their belongings at home and optimize their shopping plans. It also utilizes a generative AI model to generate user-friendly prompts, an example of which is shown below.

[0784] Example of a prompt:

[0785] "Identify the food items in the refrigerator and generate prompts to order any missing items from an online shopping site."

[0786] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0787] Step 1:

[0788] The user takes photos of household items with their device.

[0789] The input is image data taken by the user.

[0790] The device transmits this image to the server via wireless or wired communication. The output is the image data transferred to the server.

[0791] Step 2:

[0792] The server passes the received image data to the image recognition AI.

[0793] The input is image data that exists on the server.

[0794] The server uses image recognition technology to identify the type and quantity of items and performs data processing to obtain the necessary item data.

[0795] The output is the analysis result, including the type and quantity of items.

[0796] Step 3:

[0797] The server saves the analysis results as inventory information in the database.

[0798] The input is item data based on the analysis results.

[0799] The server performs data calculations to update item information and saves the results to the database.

[0800] The output is the most recent inventory information.

[0801] Step 4:

[0802] The server compares the updated inventory information with the cooking procedure data.

[0803] The input consists of updated inventory information and cooking procedure data.

[0804] The server compares this data and performs data processing to identify missing items.

[0805] The output is a list of missing items.

[0806] Step 5:

[0807] The server generates shopping suggestions based on the missing items and sends them to the user's terminal via a communication device.

[0808] The input is a list of missing items.

[0809] The server uses this list to generate purchase suggestions and processes the data to create a shopping list of any missing items.

[0810] The output is in a format where shopping suggestions are displayed on the terminal.

[0811] Step 6:

[0812] The device displays the received shopping suggestions to the user.

[0813] The input is a shopping suggestion sent from the server.

[0814] The device performs an action that visually presents the latest status to the user.

[0815] The output is a visual purchase suggestion.

[0816] Step 7:

[0817] The server uses an external information processing device to retrieve product information from the e-commerce platform.

[0818] The input is data from an e-commerce platform.

[0819] The server collects information on potential purchase items and performs data processing to support seamless purchasing.

[0820] The output is a list of available products.

[0821] Step 8:

[0822] The server analyzes the user's consumption history and calculates the optimal time to make a purchase.

[0823] The input is the consumption history of the items you possess.

[0824] Based on this information, the server analyzes the user's consumption patterns and performs data calculations to suggest the best time to make a purchase.

[0825] The output indicates the recommended purchase time.

[0826] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0827] This invention is a system that streamlines user item management and provides more personalized shopping assistance by utilizing an emotion engine. Users use a terminal to take pictures of storage shelves and refrigerators in their homes and upload these images to a server. The server uses image recognition means to automatically identify the type and quantity of items in the images and obtain the necessary data. This data is recorded in a database and stored as possession information.

[0828] The updated inventory information is then cross-referenced with a pre-registered cooking method database. This database contains recipes and required ingredients for various dishes. Based on the inventory information and cooking method data, the server identifies missing items and generates an optimal shopping list for the user. This shopping list is then notified to the user via the terminal's display device.

[0829] In addition, the present invention incorporates an emotion engine into the system, which can analyze the user's emotions and customize shopping lists and notification content. Specifically, when a user expresses their emotions through voice input or text input from their terminal, the emotion engine analyzes that information and adjusts items on the shopping list or makes new suggestions. For example, if the user is feeling stressed, it can suggest additional items or recipes that can help them relax.

[0830] As a concrete example, when a user takes a picture of their refrigerator, their inventory information is updated. Later, when the user enters "I've been busy and stressed lately," the emotion engine analyzes this input and adds herbal teas and easy-to-make relaxing recipes to their shopping list. The user can then review the updated list and purchase items as needed, gaining a personalized consumption experience. In this way, the system simultaneously achieves effective inventory management and flexible suggestions tailored to the user's psychological needs.

[0831] The following describes the processing flow.

[0832] Step 1:

[0833] Users use their devices to take pictures of storage shelves and refrigerators in their homes. The captured images are uploaded from the device to the server.

[0834] Step 2:

[0835] The server receives the uploaded image and uses image recognition to analyze the type and quantity of items. This identifies each item in the image.

[0836] Step 3:

[0837] The server sends the analysis results to the database, updating it with inventory information. This update reflects the latest inventory status of all items owned by the user.

[0838] Step 4:

[0839] The server uses the updated inventory information and compares it with the cooking recipe database. The cooking recipe database contains pre-registered recipes and the necessary items. This comparison identifies any missing items.

[0840] Step 5:

[0841] The server identifies the missing items and generates a shopping list. The shopping list contains the necessary items and is sent to the user's device.

[0842] Step 6:

[0843] The terminal receives the shopping list from the server and notifies the user, displaying it on the terminal's display device. The user can then review the displayed list.

[0844] Step 7:

[0845] Users use their devices to input their emotions and desires via voice or text. This input information is then analyzed by an emotion engine.

[0846] Step 8:

[0847] The server uses an emotion engine to analyze the user's emotions and preferences, and customizes the shopping list and recipes accordingly. Based on the analysis, it adds new suggestions to the shopping list, such as herbal teas to help relieve stress.

[0848] Step 9:

[0849] The device will redisplay a newly customized shopping list and notify the user, allowing them to make appropriate shopping decisions based on the latest information.

[0850] (Example 2)

[0851] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0852] Current inventory management systems present a complex process for users to efficiently manage their household belongings and then list the items they need. Furthermore, systems capable of providing shopping suggestions tailored to individual users' emotions and circumstances are limited, making it difficult to address individual needs. This results in challenges in providing optimal support for users' purchasing experiences and lifestyles.

[0853] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0854] In this invention, the server includes means for analyzing item data acquired using image recognition means, means for updating management information based on the analyzed item data, means for generating a list of recommended items using the updated management information and pre-registered cooking method data, and means for analyzing emotions from voice or text input and customizing the suggestions in the list. This enables users to manage their items efficiently and receive personalized shopping suggestions that meet their emotions and individual needs.

[0855] "Image recognition means" refers to a technology that identifies items within an image based on a captured image and automatically determines their type and quantity.

[0856] "Item data" refers to information about individual items extracted by image recognition technology, and includes attributes such as type and quantity.

[0857] "Management information" refers to information in a database used to record and update the types and quantities of items owned by the user. This information reflects the latest status of owned items.

[0858] "Cooking method data" is a dataset containing various recipes and information on necessary ingredients, and is used to identify required items by associating them with information on possessions.

[0859] A "recommended list" is a list that provides users with a list of items they are missing or need, to help them with their purchasing decisions.

[0860] "Means of analyzing emotions and customizing suggestions" refers to technology that analyzes the user's emotional expressions in voice or text and adds new suggestions or adjustments to the list of recommendations based on the results.

[0861] This invention is a system that streamlines household item management and provides users with personalized shopping assistance. Implementation involves a user-owned device (such as a smartphone or tablet), a server, a database, and an emotion analysis engine.

[0862] The user takes photos of storage areas and refrigerators in their home using the device. The device compresses the captured images and uploads them to a server via the internet.

[0863] The server processes the received images. Specifically, it uses image processing and machine learning libraries such as OpenCV and TensorFlow to identify items within the images. This extracts data on the type and quantity of items. This item data is stored in a database, keeping the user's possession status up-to-date as management information.

[0864] The server also cross-references the user's inventory information with a cooking method database and processes the data to identify missing items. This cooking method data contains recipes and required ingredients. As a result, the server generates a recommended list for the user, i.e., a shopping list. This generation process uses a generative AI model to provide customized recommendations based on prompt messages.

[0865] Furthermore, users can input their emotions via voice or text through their device. An emotion analysis engine analyzes this information and adjusts the shopping list based on the user's emotions. For example, for a user who is feeling stressed, it can suggest products with relaxing effects or easy-to-make recipes.

[0866] As a concrete example, an example of a prompt message might be, "If a user takes a picture of their refrigerator and feels stressed, what items or recipes should be suggested?" Based on this prompt, the system generates optimal suggestions tailored to the user's emotions and possessions. In this way, the present invention realizes personalized life support that suits the user.

[0867] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0868] Step 1:

[0869] The user takes photos of storage shelves or a refrigerator in their home using their device. The input is the captured image, and the output is the image file saved on the device. Specifically, the user operates the camera app to photograph the desired area.

[0870] Step 2:

[0871] The device uploads the captured image to the server. The input is the image file stored on the device, and the output is the image data sent to the server. Specifically, in this step, the device uses Wi-Fi or mobile data to transfer the image to the server via the internet.

[0872] Step 3:

[0873] The server analyzes the received image using an image recognition library. The input is the image data sent to the server, and the output is the analyzed item data. This item data is a list of items in the image based on their type and quantity, and specifically, item recognition and classification are performed using a convolutional neural network.

[0874] Step 4:

[0875] The server records the analyzed item data in the database. The input is the analyzed item data, and the output is the updated management information in the database. This step compares the data with the existing database to reflect any additions, deletions, or updates to items.

[0876] Step 5:

[0877] The server identifies missing items by cross-referencing updated management information with the cooking method database. Inputs are updated management information and cooking method data, and output is a recommended list (shopping list). This cross-referencing process determines whether all necessary ingredients are available and lists any missing items.

[0878] Step 6:

[0879] The user inputs their emotions into the device via voice or text. The input is the user's voice or text information, and the output is emotion data sent to the device. The specific action involves expressing emotions using a voice input application.

[0880] Step 7:

[0881] The server analyzes emotions expressed in voice or text and customizes the recommended list. The input is emotion data and an existing shopping list, and the output is a shopping list adjusted based on those emotions. This step uses a generative AI model to gain insights from emotions and optimize the list.

[0882] Step 8:

[0883] The terminal notifies the user of the final shopping list. The input is a customized shopping list received from the server, and the output is the list displayed on the user's terminal screen. Specifically, it uses a notification function to inform the user of the updates.

[0884] (Application Example 2)

[0885] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0886] In recent years, there has been a growing demand for efficient inventory management and personalized product recommendations based on user emotions. However, conventional systems have struggled to integrate inventory management and user emotion analysis, making it difficult to provide optimal recommendations for users. This invention aims to provide a system that utilizes image recognition technology and an emotion engine to achieve more accurate inventory management and personalized product recommendations.

[0887] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0888] In this invention, the server includes means for analyzing item data acquired using image recognition means, means for updating possession information based on the analyzed item data, and means for customizing shopping lists and product suggestions using an emotion engine that analyzes the user's emotional information. This enables efficient item management based on the user's actual possessions and personalized product suggestions tailored to the user's emotions at any given time.

[0889] "Image recognition means" refers to a technology that analyzes image data to automatically identify the type and quantity of items.

[0890] "Item data" refers to information about items obtained through image recognition, including their type and quantity.

[0891] "Possession information" refers to information stored in a database about items owned by the user.

[0892] "Cooking method data" refers to a database containing pre-registered information about various cooking recipes and necessary ingredients.

[0893] A "shopping list" is a list that identifies items that are lacking and lists recommended items that need to be purchased.

[0894] An "emotion engine" is a technology that analyzes a user's emotional information and customizes the information accordingly.

[0895] A "display device" is a device that provides users with generated information or shopping lists visually.

[0896] A "learning model" is a predictive model used to improve the accuracy of analysis in image recognition and sentiment analysis.

[0897] "Consumption history" refers to records of items a user has consumed in the past, and is used to predict future consumption patterns.

[0898] "Dynamic recommendations" refer to product recommendations and information provided that change in real time according to the user's current state and emotions.

[0899] This system consists of user terminals, servers, and various software that communicates with them. User terminals refer to devices such as smartphones and computers, while servers are computing devices located in the cloud or on a local network.

[0900] The server runs various programs, including image recognition and an emotion engine. The image recognition uses TensorFlow and PyTorch, which automatically analyzes item data from images of storage shelves and refrigerators taken by the user, identifying their type and quantity. The user's device is responsible for uploading these images to the server.

[0901] The emotion engine uses IBM Watson and Google Cloud Natural Language API to analyze user emotions. This engine analyzes voice and text information entered by the user via the device to determine the user's emotional state.

[0902] The server updates the user's inventory information based on the analyzed item data and cross-references this information with cooking method data. During this process, it generates a shopping list recommending necessary items. This shopping list is customized according to the user's emotional state and displayed on the user's device.

[0903] For example, if a user enters "I've been busy lately and feeling stressed" into their device, the system analyzes this information using its emotion engine. Based on the analysis, products and recipes effective for stress relief are added to the shopping list. In this way, users can receive personalized suggestions.

[0904] An example of a prompt message could be, "I've been feeling exhausted from work lately. Please suggest some products that will help me relax." This allows for flexible product suggestions tailored to the user's psychological needs.

[0905] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0906] Step 1:

[0907] The user uses the device to take pictures of storage shelves and refrigerators in their home. The device then prepares the captured image data and sends it to the server.

[0908] Step 2:

[0909] The server applies image recognition to the received image data. Specifically, it uses TensorFlow to classify objects within the image and identify their type and quantity. Item data is generated as output of this process.

[0910] Step 3:

[0911] The server updates the inventory information in the database based on the analyzed item data. The updated inventory information is output as data indicating the current status of the user's items.

[0912] Step 4:

[0913] The user inputs emotional information via voice or text on the device. The device then sends this input data to the server.

[0914] Step 5:

[0915] The server uses an emotion engine to analyze emotional information received from the user. Specifically, it uses IBM Watson to quantify or categorize emotional states. The analysis results are output and used for subsequent processing.

[0916] Step 6:

[0917] The server uses updated inventory information and sentiment analysis results to match them against its internal cooking recipe database. It identifies the necessary items and generates a customized shopping list based on a generative AI model.

[0918] Step 7:

[0919] The server sends the generated shopping list to the user's terminal. The user can visually review the shopping list on their terminal and decide on their next purchase action. This stage also includes information received as a result of prompt messages.

[0920] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0921] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0922] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0923] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0924] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0925] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0926] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0927] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0928] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0929] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0930] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0931] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0932] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0933] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0934] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0935] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0936] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0937] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0938] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0939] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0940] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0941] The following is further disclosed regarding the embodiments described above.

[0942] (Claim 1)

[0943] A means for analyzing item data acquired using image recognition means,

[0944] A means of updating personal belongings information based on analyzed item data,

[0945] A means for generating a shopping list that recommends necessary items using updated inventory information and pre-registered cooking method data,

[0946] A means for outputting the generated shopping list to a display device,

[0947] A system that includes this.

[0948] (Claim 2)

[0949] The system according to claim 1, comprising a learning model that improves the accuracy of analyzing item data acquired by image recognition means.

[0950] (Claim 3)

[0951] The system according to claim 1, further comprising means for calculating the optimal purchase time based on the consumption history of the user's possessions and notifying the user.

[0952] "Example 1"

[0953] (Claim 1)

[0954] A means for analyzing item information acquired using image processing means,

[0955] A means of updating possession data based on analyzed item information,

[0956] A means for generating a shopping list that suggests necessary items using updated inventory data and pre-registered cooking information,

[0957] A means for outputting the generated shopping list to a display device,

[0958] A means of analyzing past consumption history to understand consumption trends for each item,

[0959] A means of calculating the optimal purchase time based on consumption trends and notifying the user,

[0960] A system that includes this.

[0961] (Claim 2)

[0962] The system according to claim 1, comprising a learning model that improves the accuracy of analyzing item information acquired by image processing means.

[0963] (Claim 3)

[0964] The system according to claim 1, comprising means for optimizing the accuracy and speed of analyzing item information using a generative AI model.

[0965] "Application Example 1"

[0966] (Claim 1)

[0967] A means for analyzing item data acquired using image recognition means,

[0968] A means of updating personal belongings information based on analyzed item data,

[0969] A means for generating shopping suggestions that recommend necessary items using updated inventory information and pre-registered cooking procedure data,

[0970] A means for transmitting the generated shopping suggestions to an external information processing device via a communication device,

[0971] A means of combining product information obtained from an external information processing device to make purchase suggestions on an e-commerce platform,

[0972] A system that includes this.

[0973] (Claim 2)

[0974] The system according to claim 1, comprising an educational program for improving the accuracy of analyzing item data acquired by image recognition means.

[0975] (Claim 3)

[0976] The system according to claim 1, further comprising means for calculating the optimal purchase time based on the consumption history of the user's belongings and notifying a communication device of the result.

[0977] "Example 2 of combining an emotion engine"

[0978] (Claim 1)

[0979] A means for analyzing item data acquired using image recognition means,

[0980] A means of updating management information based on analyzed item data,

[0981] A means for generating a list of recommended items using updated management information and pre-registered cooking method data,

[0982] A means for outputting the generated list to a display device,

[0983] A means of analyzing emotions from voice or text input and customizing suggestions in a list,

[0984] A system that includes this.

[0985] (Claim 2)

[0986] The system according to claim 1, comprising a learning model that improves the accuracy of analyzing item data acquired by image recognition means.

[0987] (Claim 3)

[0988] The system according to claim 1, further comprising means for calculating the optimal purchase time based on the consumption history of management information and notifying the user.

[0989] "Application example 2 when combining with an emotional engine"

[0990] (Claim 1)

[0991] A means for analyzing item data acquired using image recognition means,

[0992] A means of updating personal belongings information based on analyzed item data,

[0993] A means for generating a shopping list that recommends necessary items using updated inventory information and pre-registered cooking method data,

[0994] A means of customizing shopping lists and product suggestions using an emotion engine that analyzes user sentiment information,

[0995] A means for outputting the generated shopping list to a display device,

[0996] A system that includes this.

[0997] (Claim 2)

[0998] The system according to claim 1, comprising a learning model that improves the accuracy of analyzing item data acquired by image recognition means, and an optimized learning algorithm for improving the accuracy of recommendations based on user sentiment analysis.

[0999] (Claim 3)

[1000] The system according to claim 1, further comprising means for calculating the optimal purchase time based on the consumption history of the user's possessions and notifying the user, and means for making dynamic suggestions based on the user's current emotional state. [Explanation of Symbols]

[1001] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for analyzing item data acquired using image recognition means, A means of updating personal belongings information based on analyzed item data, A means for generating a shopping list that recommends necessary items using updated inventory information and pre-registered cooking method data, A means for outputting the generated shopping list to a display device, A system that includes this.

2. The system according to claim 1, comprising a learning model that improves the accuracy of analyzing item data acquired by image recognition means.

3. The system according to claim 1, further comprising means for calculating the optimal purchase time based on the consumption history of the user's belongings and notifying the user.

Images