system

Abstract

We provide the system. [Solution] A means of recognizing household items using an input device and acquiring related data, Means for storing data on the aforementioned articles and organizing and analyzing it based on predetermined conditions, A means for acquiring information based on the user's preferences and health status, and for generating suggestions corresponding to the said goods and user information, A means of presenting the aforementioned proposal to the user, obtaining additional information through dialogue, and making improvements, A means of providing product information visually using a smart device and generating visual alerts based on expiration dates, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern households, the problem of food waste due to insufficient food ingredient management is increasing. Also, since the selection of recipes for efficiently utilizing available food ingredients is limited, it is difficult to easily create meals that suit consumers' preferences. Furthermore, when there are specific health needs or dietary restrictions, users face the problem of more complex recipe selection. These situations lead to a waste of time and labor, resulting in inefficient food ingredient consumption and an increase in food loss. There is a need to solve such problems and provide a sustainable and healthy diet.

Means for Solving the Problems

[0005] This invention provides a system that uses an input device to recognize household items and acquire related data. The acquired item data is organized and analyzed based on predetermined conditions. It also acquires information based on the user's preferences and health status and generates optimal suggestions tailored to the items and user information. The suggestions include items to be used preferentially based on their expiration dates and are further presented to the user. The user can provide additional information through dialogue and improve the suggestions. By using this system, it is possible to achieve efficient food management and automated meal suggestions tailored to individual needs, thereby reducing food waste and improving dietary habits.

[0006] An "input device" refers to a hardware or software configuration for recognizing household items and acquiring data related to them.

[0007] "Item data" refers to a dataset that compiles information such as type, quantity, and expiration date related to items recognized using an input device.

[0008] "Prescribed conditions" refers to the standards and rules used when organizing and analyzing item data.

[0009] "Preference" refers to a user's personal taste regarding ingredients, seasonings, and types of cuisine.

[0010] "Health status" refers to health-related information such as the user's diet needs, allergies, and calorie restrictions.

[0011] "Suggestions" refer to recipes and meal menus generated based on acquired item data and user information.

[0012] "Dialogue" refers to two-way communication between the user and the system, and is used to improve suggestions.

[0013] "Best before date" refers to the date by which an item can be consumed safely and at its expected quality.

[0014] A "system" refers to a series of technical configurations aimed at efficient food management and improvement of dietary habits through the acquisition, analysis, generation, and presentation of product data. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14]It is a sequence diagram showing the processing flow of a data processing system in Application Example 2 when a sentiment engine is combined.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] This invention provides a system for efficiently managing household items and generating suggestions. The following describes in detail how this system is implemented.

[0037] First, the user uses a smartphone or dedicated device as a terminal to scan items in the refrigerator or pantry with an input device. This system uses barcode and image recognition technology to recognize the items and automatically acquires item data such as type, quantity, and expiration date. This data is sent from the terminal to a server and stored in a central database.

[0038] Next, the user enters details such as their individual preferences, health status, and eating habits into an application on their device. The server uses this information to create a database of individual user information.

[0039] The server analyzes acquired item data and user information to generate optimal suggestions. These suggestions include recipes that take available ingredients into account and menus using items nearing their expiration date. Users can view these suggestions through their devices and, if adjustments are needed, can contact the server using an interactive dialogue function. The server incorporates user feedback to improve recipes and suggestions.

[0040] As a concrete example, suppose a user scans tomatoes, chicken, and onions in their refrigerator, and the system suggests a recipe using the tomatoes and chicken. In this case, the server prioritizes the tomatoes, which are nearing their expiration date, and includes them in the suggestion. The user prepares the dish based on the suggested recipe, asking the system for cooking tips if needed. Furthermore, the information presented here, based on the remaining ingredients, helps the user generate a shopping list and streamline their next grocery shopping trip.

[0041] In this way, users can enjoy meals tailored to their individual preferences and health needs while making efficient use of household items. This system reduces food waste and contributes to promoting sustainable eating habits.

[0042] The following describes the processing flow.

[0043] Step 1:

[0044] The user launches an application on their device and scans household items using an input device. The device then reads the barcode or image of the item and retrieves item data, including its type, quantity, and expiration date.

[0045] Step 2:

[0046] The terminal sends the acquired item data to the cloud. This data reaches the server and is organized and stored in a central database. The server updates the item data, keeping the inventory status constantly up-to-date.

[0047] Step 3:

[0048] The user navigates to the user profile section within the application on their device and enters their preferences, health information, and dietary restrictions. The server then creates a user-specific profile and retains this information.

[0049] Step 4:

[0050] The server analyzes stored item data and checks expiration dates. It identifies items that should be used as a priority and generates alerts and notifications when necessary.

[0051] Step 5:

[0052] The server utilizes a generation AI to create recipes using available items, based on the user's profile. In particular, it actively incorporates items nearing their expiration date into the recipes.

[0053] Step 6:

[0054] The server sends generated recipes and meal suggestions to the terminal and presents them to the user. The terminal then shows the displayed suggestions to the user, allowing the user to choose their preferred meal from multiple options.

[0055] Step 7:

[0056] When a user asks a question or requests customization, the device sends it to the server. The server then prepares answers and advice based on the request, improving the user's experience.

[0057] Step 8:

[0058] After a recipe is suggested, the user can provide feedback via their device. The server captures this feedback and uses it to improve the accuracy of future suggestions.

[0059] (Example 1)

[0060] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0061] In managing household supplies, there are problems such as food waste due to expired food and difficulty in managing ingredients that meet the individual dietary needs of users. Furthermore, it is difficult to keep track of daily food shortages and how to use them appropriately, making efficient meal management and shopping planning a challenge.

[0062] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0063] In this invention, the server includes means for recognizing household items using input power and acquiring related information; means for storing the item information and organizing and analyzing it based on predetermined criteria; and means for acquiring information based on the user's preferences and health status and using a generative machine learning model to generate suggestions corresponding to the items and user information. This enables the management of expiration dates and the generation of meal suggestions optimized for the user.

[0064] "Input power" refers to the energy source used to recognize information about an object, enabling the operation of a device.

[0065] "Item information" refers to detailed data about household items, including information such as type, quantity, and expiration date.

[0066] "Prescribed standards" refer to the rules and conditions that are followed when organizing and analyzing product information, and are set to ensure data consistency and usefulness.

[0067] "Generative machine learning models" refer to AI technology used to generate optimal suggestions for users by utilizing collected data.

[0068] "Expiration date" refers to the period during which food products and other items can be used safely and deliciously, and is the period during which use or consumption is recommended.

[0069] A "meal plan" refers to a menu and schedule of meals designed based on the user's dietary needs, and is a means of efficiently managing daily eating habits.

[0070] "Purchase records" refer to data that records the purchase of necessary items based on a meal plan, and are useful for formulating future shopping plans.

[0071] This invention is a system for efficiently managing household items and generating suggestions. This system mainly consists of terminals and servers, each performing a specific role. The following describes a specific method for implementing this invention.

[0072] First, users use devices such as smartphones or dedicated devices to manage items in their homes. Using these devices, users scan items in their refrigerators and pantries. The devices recognize items using barcode scanners and cameras, and utilize barcode recognition and image recognition technologies to obtain item information (type, quantity, expiration date, etc.). This item information is sent from the device to a server and stored in a central database for later analysis and suggestion generation.

[0073] Next, the user enters their preferences and health information into an application installed on the device. This information includes preferences for specific foods, allergy information, and health status. The device sends the entered user information to a server, which then creates an individual user database based on this information.

[0074] The server analyzes the received item information and individual user information, and uses a generation AI model to generate optimal suggestions for the user. These suggestions might include recipes that prioritize the use of ingredients nearing their expiration date, or meal menus tailored to the user's health condition. The terminal then displays the generated suggestions to the user.

[0075] For example, if the refrigerator contains tomatoes, chicken, and onions, the server will suggest a recipe using the tomatoes that are nearing their expiration date. This reduces food waste while supporting the user's desired meal.

[0076] Another example of a prompt message is: "Please suggest the best recipe using the tomatoes, chicken, and onions I have in my refrigerator. Please also consider menus that prioritize ingredients that are nearing their expiration date." In this way, users can adjust the suggestions and obtain additional information through interactive dialogue, receiving optimal meal suggestions tailored to their individual needs.

[0077] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0078] Step 1:

[0079] Users use a terminal to manage household items, scanning items in their refrigerators and pantries. Input can be barcodes or images of the items. The terminal uses a barcode scanner or camera to acquire barcode images of items and inputs that data. Based on this input, barcode recognition technology and image recognition technology are used to identify item information such as type, quantity, and expiration date. This item information is then output.

[0080] Step 2:

[0081] The terminal transfers the acquired item information to the server. The server receives this information as input and stores it in a central database. Specifically, it uses a database management system to systematically manage and update the information. The stored item information is used for subsequent analysis and proposal generation.

[0082] Step 3:

[0083] Users input their individual preferences and health information using a dedicated application on their device. This includes favorite foods, allergy information, and calorie restrictions. The device receives user information as input and sends it to a server. Based on this transmitted data, the server organizes the information for each user and creates an individual database. This data is used to generate suggestions tailored to the user's needs.

[0084] Step 4:

[0085] The server integrates stored item information and user information and performs analysis using a generative AI model. Specifically, it evaluates the expiration dates of items and the user's dietary preferences to generate optimized suggestions. The output of this process is a list of meal plans and recipes suggested to the user.

[0086] Step 5:

[0087] The terminal presents the user with suggestions from the server. The user passively receives this output and, if necessary, interacts with the server via the terminal to request further information or adjustments. Specifically, the user can select options on the screen and ask additional questions.

[0088] Step 6:

[0089] The server takes user feedback as input, re-evaluates the accuracy of the suggestions using the generative AI model, and makes improvements. This process ensures that future suggestions are better suited to the user's needs. Specifically, it updates the AI ​​model's training data and improves the accuracy of the suggestion generation algorithm.

[0090] (Application Example 1)

[0091] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0092] Managing household items is cumbersome due to the need to check their type, quantity, and expiration dates, and there is a significant problem of food waste, especially food that expires. Furthermore, it is difficult to suggest meals that take into account individual user preferences and health conditions. In addition, current systems lack a means to instantly check item information visually, making it difficult to plan efficient consumption and shopping.

[0093] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0094] In this invention, the server includes means for recognizing household items using an input device and acquiring related data; means for storing the data on the items and organizing and analyzing it based on predetermined conditions; means for acquiring information based on the user's preferences and health status and generating suggestions corresponding to the items and user information; and means for providing item information visually using a smart device and generating visual alerts based on expiration dates. This enables the user to efficiently manage household items and plan appropriate consumption and shopping based on expiration dates.

[0095] An "input device" is a device used to acquire information about an item, and includes barcode readers and cameras.

[0096] "Related data" refers to information about the items, such as their type, quantity, and expiration date.

[0097] "Predetermined conditions" refer to criteria or baseline values ​​that are set in advance for organizing and analyzing data.

[0098] "User preferences" refers to a user's personal tastes and preferences.

[0099] "Health status" refers to information about the user's physical health and dietary restrictions.

[0100] "Suggestions" refer to recipes and consumption plans generated based on the acquired data.

[0101] A "smart device" is a device that can display and operate digital information, and includes smartphones and smart glasses.

[0102] "Visual presentation" refers to a method of displaying information directly in the user's field of vision, allowing for visual confirmation.

[0103] A "visual alert" refers to a visually displayed warning message or highlight.

[0104] "Expiration date" refers to information indicating the period by which an item can be safely consumed.

[0105] This invention aims to realize a system that optimizes the management of household items and provides appropriate suggestions to users. A specific embodiment of this system is described below.

[0106] The server first receives data transmitted from an input device. This input device includes a barcode reader and a camera, which are used to recognize items in the refrigerator or pantry and obtain related data (type, quantity, expiration date).

[0107] The device transmits data to the server using the user's smartphone or smart glasses. At this stage, the user inputs their individual preferences and health status, and this information is also transmitted to the server.

[0108] The server uses an AI model to generate optimal meal suggestions based on acquired item data and user information. These suggestions include recipes that take available ingredients into consideration and menus that prioritize items nearing their expiration date.

[0109] Users receive these suggestions visually via smart devices. For example, smart glasses overlay recipe suggestions directly into their field of vision when they open the refrigerator. Ingredients nearing their expiration date are highlighted with special visual effects.

[0110] For example, when a user takes tomatoes out of the refrigerator, a prompt such as "Please suggest a tomato soup recipe. Is this recipe suitable for tonight's dinner?" appears on the smart glasses. This allows the user to efficiently use ingredients based on the selected suggestion.

[0111] In this way, the system can reduce waste of goods within the home and promote a diet tailored to the user's health and preferences.

[0112] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0113] Step 1:

[0114] The user uses a terminal to scan items in the refrigerator or pantry using an input device. Input options include barcodes attached to the items or recognized image data. The scanned data is automatically converted into relevant information such as item type, quantity, and expiration date. This data is then sent to the server.

[0115] Step 2:

[0116] The server stores the received item data in a central database. At this stage, the data is organized and analyzed according to predetermined conditions based on the information stored in the database. Specifically, it identifies items nearing their expiration date and sets their priority.

[0117] Step 3:

[0118] Users input information about their preferences and health status through their device. This includes allergy information, dietary restrictions, and preferred food types. This information is sent to the server and stored in a database as individual user information.

[0119] Step 4:

[0120] The server uses a generative AI model to generate optimal suggestions from item data and user information. The input data includes item information and user information, and the server uses this data to select recipes and suggest future shopping lists. This output is then formatted in a user-friendly manner.

[0121] Step 5:

[0122] Users receive suggestions visually through their smart devices. At this stage, suggested recipes and ingredients nearing their expiration date are displayed on smart glasses or similar devices. Items that should be consumed as a priority are highlighted, particularly as visual alerts. This display may also appear as interactive prompts on the device.

[0123] Step 6:

[0124] Users either take action based on the suggested information or send feedback to the server if they need more information. The server then uses this feedback to improve the suggestions and incorporate it into future suggestions. This process enhances the accuracy of the suggestions and user satisfaction.

[0125] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0126] This invention provides an advanced suggestion system that takes user emotions into consideration, from managing household items to suggesting meals. A detailed embodiment of this invention is described below.

[0127] This system begins with the user scanning an item using a terminal. The terminal recognizes the item's barcode or image, retrieves the item data, and sends it to a server. This data is stored in a central database. Simultaneously, the user enters personal information, including preferences and health status, into the terminal, which is also stored on the server.

[0128] Furthermore, this system incorporates an emotion engine that analyzes the user's emotions in real time. The emotion engine evaluates the emotional state based on inputs such as voice, facial expressions, and text input that the user makes while operating the device. This allows the server to recognize the user's current emotional state and adjust the content of its suggestions accordingly.

[0129] Specifically, the server combines available item data, user preferences, and emotional data to generate optimized recipes and meal suggestions using generative AI. For example, if the emotional engine determines that the user is stressed, it can generate suggestions using ingredients that have a relaxing effect.

[0130] The suggested recipes and menus are displayed on the device, allowing the user to review them. Furthermore, when the user responds to a suggestion, the emotion engine analyzes that emotion and sends it to the server as new feedback. This allows the server to further improve future suggestions and increase user satisfaction.

[0131] In this system, the generation of shopping lists could also be influenced by emotions. For example, it could be possible to customize suggestions, such as encouraging adventurous grocery shopping when the user is in a positive emotional state.

[0132] In this way, the system supports a sustainable and satisfying diet by efficiently utilizing goods and providing personalized suggestions that respond to the user's emotions.

[0133] The following describes the processing flow.

[0134] Step 1:

[0135] The user opens an application on the device and scans household items with the input device. The device uses barcode and image recognition technology to obtain item data, including the type, quantity, and expiration date of the items.

[0136] Step 2:

[0137] The terminal sends the acquired item data to the server. The server stores the received data in a database and updates the inventory information.

[0138] Step 3:

[0139] Users input individual information such as food preferences, health status, and dietary restrictions on their devices. The device sends this information to the server, which then updates the user profile database.

[0140] Step 4:

[0141] The emotion engine operates, analyzing the user's emotions in real time from the voice, facial expressions, and text they emit while using the device. The analysis results are sent to the server.

[0142] Step 5:

[0143] The server uses the received emotional data to identify the user's current emotional state. Based on this information, the generative AI operates to generate recipes and meal suggestions that take into account item data, preferences, and the user's emotional state.

[0144] Step 6:

[0145] The generated suggestions are sent to the device, allowing the user to visually review the suggested recipes and menus. Sentiment analysis may also be used to suggest meals with relaxing effects.

[0146] Step 7:

[0147] Users provide feedback on the presented suggestions via their device. The emotion engine analyzes this feedback and sends the emotional response to the server.

[0148] Step 8:

[0149] The server analyzes user feedback and emotional responses, and incorporates these into future suggestions. This allows for more personalized suggestions and improves user satisfaction.

[0150] (Example 2)

[0151] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0152] In modern households, consumers are expected to manage numerous items and plan meals, but effectively utilizing available items and providing optimal suggestions tailored to individual health conditions and emotions is challenging. Furthermore, suggesting meals based on an individual's health and psychological state presents an even more complex challenge. Therefore, there is a need for personalized and effective meal management systems that meet individual needs.

[0153] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0154] In this invention, the server includes means for identifying objects in the home environment using an input device and acquiring corresponding information; means for storing the information of the objects and organizing and analyzing it based on set conditions; means for evaluating the user's preferences, health information, and emotional state in real time and generating suggestions according to the object information, user information, and emotional information; means for utilizing the generation AI model and using prompt sentences to make optimized recipe and meal suggestions; and means for displaying the suggestions to the user, acquiring additional information from the dialogue and reactions, and making improvements. This makes it possible to manage meals in a way that is adapted to the health condition and emotions of each individual user while making optimal use of available items.

[0155] An "input device" is a device used to identify objects in a home and acquire related information.

[0156] "Information" refers to various types of data, including data related to objects, as well as data concerning a user's health status, preferences, and emotional state.

[0157] An "object" refers to something tangible, such as household goods or food, that is managed within the home.

[0158] "Storage" means securely saving acquired information and maintaining a state where it can be accessed as needed.

[0159] "Analysis" is the process of organizing acquired information and analyzing and judging it according to a specific purpose.

[0160] "User" refers to a consumer or individual within a household who uses the system.

[0161] "Preferences" refer to information about a user's personal likes and priorities.

[0162] "Health information" refers to information related to the user's physical and mental health.

[0163] "Emotional state" refers to the user's real-time psychological and emotional state.

[0164] A "generative AI model" is an artificial intelligence algorithm or program used for making suggestions or performing analyses.

[0165] A "prompt statement" is an instruction given to a generative AI model to tell it to produce a specific output.

[0166] A "recipe" is a list of steps and ingredients for preparing a meal or dish.

[0167] "Suggestions" refer to recommendations or options presented to users.

[0168] "Dialogue" refers to two-way communication between the user and the system.

[0169] "Response" refers to the opinions and actions that users show towards a proposal or system.

[0170] "Improvement" refers to enhancing system suggestions and functions based on feedback from users.

[0171] This invention provides a system that allows users to manage their home life in a more efficient and personalized way. Detailed embodiments for carrying out the invention are described below.

[0172] The system uses input devices, including cameras and barcode scanners, to identify home appliances and other identifiable physical objects. This automatically acquires and digitizes information about the items. The terminal then transmits this information as packets to the server.

[0173] The server stores acquired item information in a central database, and also incorporates health information, preference information, and real-time emotional information entered by users. This allows for centralized management of a wide range of information related to the user.

[0174] Emotional information is acquired through voice recognition and facial expression analysis software provided by the device. This allows users to receive customized information that is most appropriate for them when receiving recipes or meal suggestions.

[0175] The server uses a generative AI model to optimize suggestions. Instructions are given to the AI ​​model using prompts. For example, if the server sends a prompt such as, "Please suggest a recipe for a dish that will make me feel happy using the ingredients I have in my refrigerator," the AI ​​model will generate a recipe based on the user's emotional state and the ingredients they possess.

[0176] Recipes and meal suggestions generated by the server are displayed to the user on their device. Users can review the suggestions and try the dishes, and the emotion engine analyzes their reactions to the results. This allows the server to improve the accuracy of future suggestions. Specifically, it can suggest using chamomile tea when the user wants to relax.

[0177] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0178] Step 1:

[0179] The terminal identifies objects in the home using a camera or barcode scanner. The input is visual information of the object, and the terminal uses image processing algorithms to identify the type of object and extract its unique ID and related data. This data is packetized as item identification information and sent to the server.

[0180] Step 2:

[0181] The server receives item information sent from the terminal and stores it in the central database. The input consists of item information packets, which the server classifies, organizes, and stores in the database. During this process, the current inventory information is also updated. The output is the updated inventory master.

[0182] Step 3:

[0183] Users input health information, preference information, and emotional information through their devices. This includes text input about their health status and favorite foods, as well as voice and facial expression data indicating their emotional state. The device analyzes this data using an emotion engine and sends it to the server as a dataset.

[0184] Step 4:

[0185] The server uses an emotion engine to analyze the user's emotional state. The server analyzes the input voice, facial expressions, and text data to evaluate the user's emotional state. This information is processed together with other user information. The output is data indicating the user's emotional state.

[0186] Step 5:

[0187] The server uses a generative AI model to input prompts based on received item information, user information, and sentiment data, and creates optimized suggestions. An example prompt might be, "Please suggest a relaxing recipe using ingredients found in the refrigerator," and the generative AI model would output recipe data.

[0188] Step 6:

[0189] The server sends the generated recipe to the terminal and displays it to the user. The output is a visualized meal suggestion, which the user can refer to and select a dish from.

[0190] Step 7:

[0191] Users cook according to a recipe and provide feedback on the process and results via their device. This includes inputting comments via voice or text, which are then analyzed by an emotion engine on the device. The feedback is sent to the server as new data.

[0192] Step 8:

[0193] The server analyzes the collected feedback and makes improvements to enhance the accuracy of future proposals. This analysis prepares the readjusted proposal data for the next cycle. The output is the improved proposal algorithm.

[0194] (Application Example 2)

[0195] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0196] There is a need for a system that can efficiently manage household items and, based on that, provide meal suggestions tailored to the user's emotional state. While conventional systems could manage items and make suggestions based on user preferences, they could not consider the user's real-time emotional state, making it difficult to provide highly satisfying suggestions.

[0197] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0198] In this invention, the server includes means for recognizing household items using an input device and acquiring related data; means for storing the data on the items and organizing and analyzing it based on predetermined conditions; means for acquiring information based on the user's preferences and health status and generating suggestions corresponding to the items and user information; means for analyzing the user's emotional state and adjusting the content of the suggestions based on the emotional state; and means for presenting the suggestions to the user, acquiring additional information through dialogue, and making improvements. This enables personalized meal suggestions that take into account the user's item management and emotional state.

[0199] An "input device" is a device used to recognize household items and acquire information related to those items.

[0200] "Item data" refers to data that includes information for identifying an item and detailed information related to the item.

[0201] "Prescribed conditions" refer to specific criteria or rules set for organizing and analyzing item data.

[0202] "User preferences" refers to information about ingredients, dishes, or personal tendencies and preferences regarding food that users particularly like.

[0203] "Health status" refers to information related to the user's physical health, specifically including allergies, dietary restrictions, and required nutrients.

[0204] "Adjusting the suggested menu" is the process of optimizing meal suggestions based on the user's real-time emotional state.

[0205] "Emotional state" refers to information that indicates the user's current psychological or emotional condition.

[0206] "Means of obtaining additional information and making improvements through dialogue" refers to a process of checking users' reactions to proposals and using the feedback received to make future proposals more suitable for users.

[0207] The system that realizes this invention consists of a series of processes to enable household item management and meal suggestions based on the user's emotions. The system is mainly composed of an input device, a server, and a user terminal.

[0208] The device, for example, takes the form of a smartphone or smart speaker, and acquires item data by photographing or scanning items. This item data includes information obtained using barcodes and image recognition technology. This data records the items and is transmitted to a central server.

[0209] The server stores and analyzes received item data and user preference and health information provided through the device. An emotion engine is integrated into the server to analyze the user's emotional state in real time from voice, facial expressions, and text input as they operate the device. This data is used to assess the emotional state and forms the basis for adjusting meal recommendations.

[0210] The generative AI model generates optimized recipes based on available item data on the server, user preferences, and emotional data. For example, if a user is looking to relax after returning home from work, the generative AI might suggest a relaxing herbal tea chicken salad.

[0211] Users can view suggested recipes on their devices and, if necessary, automatically order ingredients by linking with a local refrigerator locker. Furthermore, user feedback on the suggestions is used to improve future suggestions.

[0212] For example, a prompt might take the form of "Please suggest a relaxing recipe for when the user is feeling stressed." By using this prompt, the generating AI can suggest the most suitable meal.

[0213] In this way, the system provides personalized meal suggestions that respond to the user's emotions, along with the efficient use of goods.

[0214] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0215] Step 1:

[0216] The device scans household items and retrieves data related to those items. It uses barcodes or image data of items as input and generates item identification information and associated data as output. Specifically, the device reads barcodes using a camera or scanner and retrieves the corresponding data from a cloud service or on-device database.

[0217] Step 2:

[0218] The server stores, organizes, and analyzes the acquired item data. It receives item identification information as input and generates organized database entries as output. In this step, the server uses a database management system to record item data and organizes it into categories based on predetermined rules.

[0219] Step 3:

[0220] Users input information about their preferences and health status through their devices. Inputs include forms and voice input data, and output generates user profile information. This process involves using an application on the device to ask about the user's preferences and sending the entered responses to a server.

[0221] Step 4:

[0222] The server acquires real-time emotion data to analyze the user's emotional state. It uses voice, facial expressions, and text data as input and generates emotion state data as output. Specifically, it uses an emotion engine to analyze the voice and images input by the user and quantify the emotional tendencies.

[0223] Step 5:

[0224] The server uses a generative AI model to suggest optimal recipes based on item data, preferences, health information, and emotional data. It uses each dataset as input and generates personalized recipes as output. In this step, the generative AI is instructed with the prompt "Suggest a relaxing recipe for when the user is feeling stressed," and the generated recipes are sent to the user's terminal.

[0225] Step 6:

[0226] Users view the suggested recipes on their devices and provide feedback. The system receives the suggested recipes as input and generates improvement feedback as output. Specifically, users evaluate the recipes, send the feedback to the server information system, and contribute to improving the accuracy of future suggestions.

[0227] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0228] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0229] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0230] [Second Embodiment]

[0231] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0232] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0233] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0234] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0235] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0236] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0237] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0238] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0239] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0240] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0241] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0242] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0243] This invention provides a system for efficiently managing household items and generating suggestions. The following describes in detail how this system is implemented.

[0244] First, the user uses a smartphone or dedicated device as a terminal to scan items in the refrigerator or pantry with an input device. This system uses barcode and image recognition technology to recognize the items and automatically acquires item data such as type, quantity, and expiration date. This data is sent from the terminal to a server and stored in a central database.

[0245] Next, the user enters details such as their individual preferences, health status, and eating habits into an application on their device. The server uses this information to create a database of individual user information.

[0246] The server analyzes acquired item data and user information to generate optimal suggestions. These suggestions include recipes that take available ingredients into account and menus using items nearing their expiration date. Users can view these suggestions through their devices and, if adjustments are needed, can contact the server using an interactive dialogue function. The server incorporates user feedback to improve recipes and suggestions.

[0247] As a concrete example, suppose a user scans tomatoes, chicken, and onions in their refrigerator, and the system suggests a recipe using the tomatoes and chicken. In this case, the server prioritizes the tomatoes, which are nearing their expiration date, and includes them in the suggestion. The user prepares the dish based on the suggested recipe, asking the system for cooking tips if needed. Furthermore, the information presented here, based on the remaining ingredients, helps the user generate a shopping list and streamline their next grocery shopping trip.

[0248] In this way, users can enjoy meals tailored to their individual preferences and health needs while making efficient use of household items. This system reduces food waste and contributes to promoting sustainable eating habits.

[0249] The following describes the processing flow.

[0250] Step 1:

[0251] The user launches an application on their device and scans household items using an input device. The device then reads the barcode or image of the item and retrieves item data, including its type, quantity, and expiration date.

[0252] Step 2:

[0253] The terminal sends the acquired item data to the cloud. This data reaches the server and is organized and stored in a central database. The server updates the item data, keeping the inventory status constantly up-to-date.

[0254] Step 3:

[0255] The user navigates to the user profile section within the application on their device and enters their preferences, health information, and dietary restrictions. The server then creates a user-specific profile and retains this information.

[0256] Step 4:

[0257] The server analyzes stored item data and checks expiration dates. It identifies items that should be used as a priority and generates alerts and notifications when necessary.

[0258] Step 5:

[0259] The server utilizes a generation AI to create recipes using available items, based on the user's profile. In particular, it actively incorporates items nearing their expiration date into the recipes.

[0260] Step 6:

[0261] The server sends generated recipes and meal suggestions to the terminal and presents them to the user. The terminal then shows the displayed suggestions to the user, allowing the user to choose their preferred meal from multiple options.

[0262] Step 7:

[0263] When a user asks a question or requests customization, the device sends it to the server. The server then prepares answers and advice based on the request, improving the user's experience.

[0264] Step 8:

[0265] After a recipe is suggested, the user can provide feedback via their device. The server captures this feedback and uses it to improve the accuracy of future suggestions.

[0266] (Example 1)

[0267] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0268] In managing household supplies, there are problems such as food waste due to expired food and difficulty in managing ingredients that meet the individual dietary needs of users. Furthermore, it is difficult to keep track of daily food shortages and how to use them appropriately, making efficient meal management and shopping planning a challenge.

[0269] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0270] In this invention, the server includes means for recognizing household items using input power and acquiring related information; means for storing the item information and organizing and analyzing it based on predetermined criteria; and means for acquiring information based on the user's preferences and health status and using a generative machine learning model to generate suggestions corresponding to the items and user information. This enables the management of expiration dates and the generation of meal suggestions optimized for the user.

[0271] "Input power" refers to the energy source used to recognize information about an object, enabling the operation of a device.

[0272] "Item information" refers to detailed data about household items, including information such as type, quantity, and expiration date.

[0273] "Prescribed standards" refer to the rules and conditions that are followed when organizing and analyzing product information, and are set to ensure data consistency and usefulness.

[0274] "Generative machine learning models" refer to AI technology used to generate optimal suggestions for users by utilizing collected data.

[0275] "Expiration date" refers to the period during which food products and other items can be used safely and deliciously, and is the period during which use or consumption is recommended.

[0276] A "meal plan" refers to a menu and schedule of meals designed based on the user's dietary needs, and is a means of efficiently managing daily eating habits.

[0277] "Purchase records" refer to data that records the purchase of necessary items based on a meal plan, and are useful for formulating future shopping plans.

[0278] This invention is a system for efficiently managing household items and generating suggestions. This system mainly consists of terminals and servers, each performing a specific role. The following describes a specific method for implementing this invention.

[0279] First, users use devices such as smartphones or dedicated devices to manage items in their homes. Using these devices, users scan items in their refrigerators and pantries. The devices recognize items using barcode scanners and cameras, and utilize barcode recognition and image recognition technologies to obtain item information (type, quantity, expiration date, etc.). This item information is sent from the device to a server and stored in a central database for later analysis and suggestion generation.

[0280] Next, the user enters their preferences and health information into an application installed on the device. This information includes preferences for specific foods, allergy information, and health status. The device sends the entered user information to a server, which then creates an individual user database based on this information.

[0281] The server analyzes the received item information and the user's individual information, and uses the generated AI model to generate an optimal proposal for the user. For example, it includes recipes that prioritize the use of ingredients approaching their expiration dates and meal menus that take into account the user's health status. The terminal displays the generated proposal to the user.

[0282] As a specific example, when there are tomatoes, chicken, and onions in the refrigerator, the server proposes a recipe that utilizes the tomatoes with a near expiration date. This makes it possible to support the user's desired meals while reducing food waste.

[0283] Also, examples of prompt sentences include the following. "Please propose an optimal recipe using the tomatoes, chicken, and onions in the refrigerator. Also consider menus that prioritize ingredients with a near expiration date." In this way, through interactive dialogue, the user can adjust the proposal and obtain additional information, and receive an optimal meal proposal according to individual needs.

[0284] The flow of the specific process in Example 1 will be described using FIG. 11.

[0285] Step 1:

[0286] The user uses the terminal to manage household items and scans the items in the refrigerator and pantry. The inputs include barcodes and images of the items. The terminal uses a barcode scanner or camera to obtain a barcode image of the item and inputs the data. Based on this input, barcode recognition technology and image recognition technology are used to identify item information such as the type, quantity, and expiration date of the item. This item information is output.

[0287] Step 2:

[0288] The terminal transfers the acquired item information to the server. The server receives this information as input and stores it in a central database. Specifically, it uses a database management system to systematically manage and update the information. The stored item information is used for subsequent analysis and proposal generation.

[0289] Step 3:

[0290] Users input their individual preferences and health information using a dedicated application on their device. This includes favorite foods, allergy information, and calorie restrictions. The device receives user information as input and sends it to a server. Based on this transmitted data, the server organizes the information for each user and creates an individual database. This data is used to generate suggestions tailored to the user's needs.

[0291] Step 4:

[0292] The server integrates stored item information and user information and performs analysis using a generative AI model. Specifically, it evaluates the expiration dates of items and the user's dietary preferences to generate optimized suggestions. The output of this process is a list of meal plans and recipes suggested to the user.

[0293] Step 5:

[0294] The terminal presents the user with suggestions from the server. The user passively receives this output and, if necessary, interacts with the server via the terminal to request further information or adjustments. Specifically, the user can select options on the screen and ask additional questions.

[0295] Step 6:

[0296] The server takes user feedback as input, re-evaluates the accuracy of the suggestions using the generative AI model, and makes improvements. This process ensures that future suggestions are better suited to the user's needs. Specifically, it updates the AI ​​model's training data and improves the accuracy of the suggestion generation algorithm.

[0297] (Application Example 1)

[0298] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0299] Managing household items is cumbersome due to the need to check their type, quantity, and expiration dates, and there is a significant problem of food waste, especially food that expires. Furthermore, it is difficult to suggest meals that take into account individual user preferences and health conditions. In addition, current systems lack a means to instantly check item information visually, making it difficult to plan efficient consumption and shopping.

[0300] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0301] In this invention, the server includes means for recognizing household items using an input device and acquiring related data; means for storing the data on the items and organizing and analyzing it based on predetermined conditions; means for acquiring information based on the user's preferences and health status and generating suggestions corresponding to the items and user information; and means for providing item information visually using a smart device and generating visual alerts based on expiration dates. This enables the user to efficiently manage household items and plan appropriate consumption and shopping based on expiration dates.

[0302] An "input device" is a device used to acquire information about an item, and includes barcode readers and cameras.

[0303] "Related data" refers to information such as the type, quantity, expiration date, etc. of an item.

[0304] "Predetermined conditions" are the criteria and reference values set in advance for sorting and analyzing data.

[0305] "User preferences" refer to the personal hobbies and preferences of the user.

[0306] "Health status" is information indicating the physical health of the user and the restrictions related to diet.

[0307] "Proposal" refers to the recipes and consumption plans generated based on the acquired data.

[0308] "Smart device" is a device that can display and operate digital information, including smartphones and smart glasses.

[0309] "Visually provided" refers to a method of directly displaying information within the user's field of vision for visual confirmation.

[0310] "Visual alert" refers to a message or highlight for attracting attention that is visually displayed.

[0311] "Consumption deadline" is information indicating the deadline within which an item can be safely consumed.

[0312] This invention realizes a system that optimizes the management of items within a household and makes appropriate proposals to users. The specific embodiments of this system will be described below.

[0313] The server first receives data transmitted from the input device. This input device includes a barcode reader and a camera, which recognize the items in the refrigerator and pantry and acquire related data (type, quantity, expiration date).

[0314] The device transmits data to the server using the user's smartphone or smart glasses. At this stage, the user inputs their individual preferences and health status, and this information is also transmitted to the server.

[0315] The server uses an AI model to generate optimal meal suggestions based on acquired item data and user information. These suggestions include recipes that take available ingredients into consideration and menus that prioritize items nearing their expiration date.

[0316] Users receive these suggestions visually via smart devices. For example, smart glasses overlay recipe suggestions directly into their field of vision when they open the refrigerator. Ingredients nearing their expiration date are highlighted with special visual effects.

[0317] For example, when a user takes tomatoes out of the refrigerator, a prompt such as "Please suggest a tomato soup recipe. Is this recipe suitable for tonight's dinner?" appears on the smart glasses. This allows the user to efficiently use ingredients based on the selected suggestion.

[0318] In this way, the system can reduce waste of goods within the home and promote a diet tailored to the user's health and preferences.

[0319] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0320] Step 1:

[0321] The user uses a terminal to scan items in the refrigerator or pantry using an input device. Input options include barcodes attached to the items or recognized image data. The scanned data is automatically converted into relevant information such as item type, quantity, and expiration date. This data is then sent to the server.

[0322] Step 2:

[0323] The server stores the received item data in a central database. At this stage, the data is organized and analyzed according to predetermined conditions based on the information stored in the database. Specifically, it identifies items nearing their expiration date and sets their priority.

[0324] Step 3:

[0325] Users input information about their preferences and health status through their device. This includes allergy information, dietary restrictions, and preferred food types. This information is sent to the server and stored in a database as individual user information.

[0326] Step 4:

[0327] The server uses a generative AI model to generate optimal suggestions from item data and user information. The input data includes item information and user information, and the server uses this data to select recipes and suggest future shopping lists. This output is then formatted in a user-friendly manner.

[0328] Step 5:

[0329] Users receive suggestions visually through their smart devices. At this stage, suggested recipes and ingredients nearing their expiration date are displayed on smart glasses or similar devices. Items that should be consumed as a priority are highlighted, particularly as visual alerts. This display may also appear as interactive prompts on the device.

[0330] Step 6:

[0331] Users either take action based on the suggested information or send feedback to the server if they need more information. The server then uses this feedback to improve the suggestions and incorporate it into future suggestions. This process enhances the accuracy of the suggestions and user satisfaction.

[0332] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0333] This invention provides an advanced suggestion system that takes user emotions into consideration, from managing household items to suggesting meals. A detailed embodiment of this invention is described below.

[0334] This system begins with the user scanning an item using a terminal. The terminal recognizes the item's barcode or image, retrieves the item data, and sends it to a server. This data is stored in a central database. Simultaneously, the user enters personal information, including preferences and health status, into the terminal, which is also stored on the server.

[0335] Furthermore, this system incorporates an emotion engine that analyzes the user's emotions in real time. The emotion engine evaluates the emotional state based on inputs such as voice, facial expressions, and text input that the user makes while operating the device. This allows the server to recognize the user's current emotional state and adjust the content of its suggestions accordingly.

[0336] Specifically, the server combines available item data, user preferences, and emotional data to generate optimized recipes and meal suggestions using generative AI. For example, if the emotional engine determines that the user is stressed, it can generate suggestions using ingredients that have a relaxing effect.

[0337] The suggested recipes and menus are displayed on the device, allowing the user to review them. Furthermore, when the user responds to a suggestion, the emotion engine analyzes that emotion and sends it to the server as new feedback. This allows the server to further improve future suggestions and increase user satisfaction.

[0338] In this system, the generation of shopping lists could also be influenced by emotions. For example, it could be possible to customize suggestions, such as encouraging adventurous grocery shopping when the user is in a positive emotional state.

[0339] In this way, the system supports a sustainable and satisfying diet by efficiently utilizing goods and providing personalized suggestions that respond to the user's emotions.

[0340] The following describes the processing flow.

[0341] Step 1:

[0342] The user opens an application on the device and scans household items with the input device. The device uses barcode and image recognition technology to obtain item data, including the type, quantity, and expiration date of the items.

[0343] Step 2:

[0344] The terminal sends the acquired item data to the server. The server stores the received data in a database and updates the inventory information.

[0345] Step 3:

[0346] Users input individual information such as food preferences, health status, and dietary restrictions on their devices. The device sends this information to the server, which then updates the user profile database.

[0347] Step 4:

[0348] The emotion engine operates, analyzing the user's emotions in real time from the voice, facial expressions, and text they emit while using the device. The analysis results are sent to the server.

[0349] Step 5:

[0350] The server uses the received emotional data to identify the user's current emotional state. Based on this information, the generative AI operates to generate recipes and meal suggestions that take into account item data, preferences, and the user's emotional state.

[0351] Step 6:

[0352] The generated suggestions are sent to the device, allowing the user to visually review the suggested recipes and menus. Sentiment analysis may also be used to suggest meals with relaxing effects.

[0353] Step 7:

[0354] Users provide feedback on the presented suggestions via their device. The emotion engine analyzes this feedback and sends the emotional response to the server.

[0355] Step 8:

[0356] The server analyzes user feedback and emotional responses, and incorporates these into future suggestions. This allows for more personalized suggestions and improves user satisfaction.

[0357] (Example 2)

[0358] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0359] In modern households, consumers are expected to manage numerous items and plan meals, but effectively utilizing available items and providing optimal suggestions tailored to individual health conditions and emotions is challenging. Furthermore, suggesting meals based on an individual's health and psychological state presents an even more complex challenge. Therefore, there is a need for personalized and effective meal management systems that meet individual needs.

[0360] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0361] In this invention, the server includes means for identifying objects in the home environment using an input device and acquiring corresponding information; means for storing the information of the objects and organizing and analyzing it based on set conditions; means for evaluating the user's preferences, health information, and emotional state in real time and generating suggestions according to the object information, user information, and emotional information; means for utilizing the generation AI model and using prompt sentences to make optimized recipe and meal suggestions; and means for displaying the suggestions to the user, acquiring additional information from the dialogue and reactions, and making improvements. This makes it possible to manage meals in a way that is adapted to the health condition and emotions of each individual user while making optimal use of available items.

[0362] An "input device" is a device used to identify objects in a home and acquire related information.

[0363] "Information" refers to various types of data, including data related to objects, as well as data concerning a user's health status, preferences, and emotional state.

[0364] An "object" refers to something tangible, such as household goods or food, that is managed within the home.

[0365] "Storage" means securely saving acquired information and maintaining a state where it can be accessed as needed.

[0366] "Analysis" is the process of organizing acquired information and analyzing and judging it according to a specific purpose.

[0367] "User" refers to a consumer or individual within a household who uses the system.

[0368] "Preferences" refer to information about a user's personal likes and priorities.

[0369] "Health information" refers to information related to the user's physical and mental health.

[0370] "Emotional state" refers to the user's real-time psychological and emotional state.

[0371] A "generative AI model" is an artificial intelligence algorithm or program used for making suggestions or performing analyses.

[0372] A "prompt statement" is an instruction given to a generative AI model to tell it to produce a specific output.

[0373] A "recipe" is a list of steps and ingredients for preparing a meal or dish.

[0374] "Suggestions" refer to recommendations or options presented to users.

[0375] "Dialogue" refers to two-way communication between the user and the system.

[0376] "Response" refers to the opinions and actions that users show towards a proposal or system.

[0377] "Improvement" refers to enhancing system suggestions and functions based on feedback from users.

[0378] This invention provides a system that allows users to manage their home life in a more efficient and personalized way. Detailed embodiments for carrying out the invention are described below.

[0379] The system uses input devices, including cameras and barcode scanners, to identify home appliances and other identifiable physical objects. This automatically acquires and digitizes information about the items. The terminal then transmits this information as packets to the server.

[0380] The server stores acquired item information in a central database, and also incorporates health information, preference information, and real-time emotional information entered by users. This allows for centralized management of a wide range of information related to the user.

[0381] Emotional information is acquired through voice recognition and facial expression analysis software provided by the device. This allows users to receive customized information that is most appropriate for them when receiving recipes or meal suggestions.

[0382] The server uses a generative AI model to optimize suggestions. Instructions are given to the AI ​​model using prompts. For example, if the server sends a prompt such as, "Please suggest a recipe for a dish that will make me feel happy using the ingredients I have in my refrigerator," the AI ​​model will generate a recipe based on the user's emotional state and the ingredients they possess.

[0383] Recipes and meal suggestions generated by the server are displayed to the user on their device. Users can review the suggestions and try the dishes, and the emotion engine analyzes their reactions to the results. This allows the server to improve the accuracy of future suggestions. Specifically, it can suggest using chamomile tea when the user wants to relax.

[0384] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0385] Step 1:

[0386] The terminal identifies objects in the home using a camera or barcode scanner. The input is visual information of the object, and the terminal uses image processing algorithms to identify the type of object and extract its unique ID and related data. This data is packetized as item identification information and sent to the server.

[0387] Step 2:

[0388] The server receives item information sent from the terminal and stores it in the central database. The input consists of item information packets, which the server classifies, organizes, and stores in the database. During this process, the current inventory information is also updated. The output is the updated inventory master.

[0389] Step 3:

[0390] Users input health information, preference information, and emotional information through their devices. This includes text input about their health status and favorite foods, as well as voice and facial expression data indicating their emotional state. The device analyzes this data using an emotion engine and sends it to the server as a dataset.

[0391] Step 4:

[0392] The server uses an emotion engine to analyze the user's emotional state. The server analyzes the input voice, facial expressions, and text data to evaluate the user's emotional state. This information is processed together with other user information. The output is data indicating the user's emotional state.

[0393] Step 5:

[0394] The server uses a generative AI model to input prompts based on received item information, user information, and sentiment data, and creates optimized suggestions. An example prompt might be, "Please suggest a relaxing recipe using ingredients found in the refrigerator," and the generative AI model would output recipe data.

[0395] Step 6:

[0396] The server sends the generated recipe to the terminal and displays it to the user. The output is a visualized meal suggestion, which the user can refer to and select a dish from.

[0397] Step 7:

[0398] Users cook according to a recipe and provide feedback on the process and results via their device. This includes inputting comments via voice or text, which are then analyzed by an emotion engine on the device. The feedback is sent to the server as new data.

[0399] Step 8:

[0400] The server analyzes the collected feedback and makes improvements to enhance the accuracy of future proposals. This analysis prepares the readjusted proposal data for the next cycle. The output is the improved proposal algorithm.

[0401] (Application Example 2)

[0402] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0403] There is a need for a system that can efficiently manage household items and, based on that, provide meal suggestions tailored to the user's emotional state. While conventional systems could manage items and make suggestions based on user preferences, they could not consider the user's real-time emotional state, making it difficult to provide highly satisfying suggestions.

[0404] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0405] In this invention, the server includes means for recognizing household items using an input device and acquiring related data; means for storing the data on the items and organizing and analyzing it based on predetermined conditions; means for acquiring information based on the user's preferences and health status and generating suggestions corresponding to the items and user information; means for analyzing the user's emotional state and adjusting the content of the suggestions based on the emotional state; and means for presenting the suggestions to the user, acquiring additional information through dialogue, and making improvements. This enables personalized meal suggestions that take into account the user's item management and emotional state.

[0406] An "input device" is a device used to recognize household items and acquire information related to those items.

[0407] "Item data" refers to data that includes information for identifying an item and detailed information related to the item.

[0408] "Prescribed conditions" refer to specific criteria or rules set for organizing and analyzing item data.

[0409] "User preferences" refers to information about ingredients, dishes, or personal tendencies and preferences regarding food that users particularly like.

[0410] "Health status" refers to information related to the user's physical health, specifically including allergies, dietary restrictions, and required nutrients.

[0411] "Adjusting the suggested menu" is the process of optimizing meal suggestions based on the user's real-time emotional state.

[0412] "Emotional state" refers to information that indicates the user's current psychological or emotional condition.

[0413] "Means of obtaining additional information and making improvements through dialogue" refers to a process of checking users' reactions to proposals and using the feedback received to make future proposals more suitable for users.

[0414] The system that realizes this invention consists of a series of processes to enable household item management and meal suggestions based on the user's emotions. The system is mainly composed of an input device, a server, and a user terminal.

[0415] The device, for example, takes the form of a smartphone or smart speaker, and acquires item data by photographing or scanning items. This item data includes information obtained using barcodes and image recognition technology. This data records the items and is transmitted to a central server.

[0416] The server stores and analyzes received item data and user preference and health information provided through the device. An emotion engine is integrated into the server to analyze the user's emotional state in real time from voice, facial expressions, and text input as they operate the device. This data is used to assess the emotional state and forms the basis for adjusting meal recommendations.

[0417] The generative AI model generates optimized recipes based on available item data on the server, user preferences, and emotional data. For example, if a user is looking to relax after returning home from work, the generative AI might suggest a relaxing herbal tea chicken salad.

[0418] Users can view suggested recipes on their devices and, if necessary, automatically order ingredients by linking with a local refrigerator locker. Furthermore, user feedback on the suggestions is used to improve future suggestions.

[0419] For example, a prompt might take the form of "Please suggest a relaxing recipe for when the user is feeling stressed." By using this prompt, the generating AI can suggest the most suitable meal.

[0420] In this way, the system provides personalized meal suggestions that respond to the user's emotions, along with the efficient use of goods.

[0421] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0422] Step 1:

[0423] The device scans household items and retrieves data related to those items. It uses barcodes or image data of items as input and generates item identification information and associated data as output. Specifically, the device reads barcodes using a camera or scanner and retrieves the corresponding data from a cloud service or on-device database.

[0424] Step 2:

[0425] The server stores, organizes, and analyzes the acquired item data. It receives item identification information as input and generates organized database entries as output. In this step, the server uses a database management system to record item data and organizes it into categories based on predetermined rules.

[0426] Step 3:

[0427] Users input information about their preferences and health status through their devices. Inputs include forms and voice input data, and output generates user profile information. This process involves using an application on the device to ask about the user's preferences and sending the entered responses to a server.

[0428] Step 4:

[0429] The server acquires real-time emotion data to analyze the user's emotional state. It uses voice, facial expressions, and text data as input and generates emotion state data as output. Specifically, it uses an emotion engine to analyze the voice and images input by the user and quantify the emotional tendencies.

[0430] Step 5:

[0431] The server uses a generative AI model to suggest optimal recipes based on item data, preferences, health information, and emotional data. It uses each dataset as input and generates personalized recipes as output. In this step, the generative AI is instructed with the prompt "Suggest a relaxing recipe for when the user is feeling stressed," and the generated recipes are sent to the user's terminal.

[0432] Step 6:

[0433] Users view the suggested recipes on their devices and provide feedback. The system receives the suggested recipes as input and generates improvement feedback as output. Specifically, users evaluate the recipes, send the feedback to the server information system, and contribute to improving the accuracy of future suggestions.

[0434] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0435] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0436] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0437] [Third Embodiment]

[0438] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0439] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0440] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0441] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0442] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0443] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0444] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0445] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0446] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0447] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0448] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0449] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0450] This invention provides a system for efficiently managing household items and generating suggestions. The following describes in detail how this system is implemented.

[0451] First, the user uses a smartphone or dedicated device as a terminal to scan items in the refrigerator or pantry with an input device. This system uses barcode and image recognition technology to recognize the items and automatically acquires item data such as type, quantity, and expiration date. This data is sent from the terminal to a server and stored in a central database.

[0452] Next, the user enters details such as their individual preferences, health status, and eating habits into an application on their device. The server uses this information to create a database of individual user information.

[0453] The server analyzes acquired item data and user information to generate optimal suggestions. These suggestions include recipes that take available ingredients into account and menus using items nearing their expiration date. Users can view these suggestions through their devices and, if adjustments are needed, can contact the server using an interactive dialogue function. The server incorporates user feedback to improve recipes and suggestions.

[0454] As a concrete example, suppose a user scans tomatoes, chicken, and onions in their refrigerator, and the system suggests a recipe using the tomatoes and chicken. In this case, the server prioritizes the tomatoes, which are nearing their expiration date, and includes them in the suggestion. The user prepares the dish based on the suggested recipe, asking the system for cooking tips if needed. Furthermore, the information presented here, based on the remaining ingredients, helps the user generate a shopping list and streamline their next grocery shopping trip.

[0455] In this way, users can enjoy meals tailored to their individual preferences and health needs while making efficient use of household items. This system reduces food waste and contributes to promoting sustainable eating habits.

[0456] The following describes the processing flow.

[0457] Step 1:

[0458] The user launches an application on their device and scans household items using an input device. The device then reads the barcode or image of the item and retrieves item data, including its type, quantity, and expiration date.

[0459] Step 2:

[0460] The terminal sends the acquired item data to the cloud. This data reaches the server and is organized and stored in a central database. The server updates the item data, keeping the inventory status constantly up-to-date.

[0461] Step 3:

[0462] The user navigates to the user profile section within the application on their device and enters their preferences, health information, and dietary restrictions. The server then creates a user-specific profile and retains this information.

[0463] Step 4:

[0464] The server analyzes stored item data and checks expiration dates. It identifies items that should be used as a priority and generates alerts and notifications when necessary.

[0465] Step 5:

[0466] The server utilizes a generation AI to create recipes using available items, based on the user's profile. In particular, it actively incorporates items nearing their expiration date into the recipes.

[0467] Step 6:

[0468] The server sends generated recipes and meal suggestions to the terminal and presents them to the user. The terminal then shows the displayed suggestions to the user, allowing the user to choose their preferred meal from multiple options.

[0469] Step 7:

[0470] When a user asks a question or requests customization, the device sends it to the server. The server then prepares answers and advice based on the request, improving the user's experience.

[0471] Step 8:

[0472] After a recipe is suggested, the user can provide feedback via their device. The server captures this feedback and uses it to improve the accuracy of future suggestions.

[0473] (Example 1)

[0474] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0475] In managing household supplies, there are problems such as food waste due to expired food and difficulty in managing ingredients that meet the individual dietary needs of users. Furthermore, it is difficult to keep track of daily food shortages and how to use them appropriately, making efficient meal management and shopping planning a challenge.

[0476] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0477] In this invention, the server includes means for recognizing household items using input power and acquiring related information; means for storing the item information and organizing and analyzing it based on predetermined criteria; and means for acquiring information based on the user's preferences and health status and using a generative machine learning model to generate suggestions corresponding to the items and user information. This enables the management of expiration dates and the generation of meal suggestions optimized for the user.

[0478] "Input power" refers to the energy source used to recognize information about an object, enabling the operation of a device.

[0479] "Item information" refers to detailed data about household items, including information such as type, quantity, and expiration date.

[0480] "Prescribed standards" refer to the rules and conditions that are followed when organizing and analyzing product information, and are set to ensure data consistency and usefulness.

[0481] "Generative machine learning models" refer to AI technology used to generate optimal suggestions for users by utilizing collected data.

[0482] "Expiration date" refers to the period during which food products and other items can be used safely and deliciously, and is the period during which use or consumption is recommended.

[0483] A "meal plan" refers to a menu and schedule of meals designed based on the user's dietary needs, and is a means of efficiently managing daily eating habits.

[0484] "Purchase records" refer to data that records the purchase of necessary items based on a meal plan, and are useful for formulating future shopping plans.

[0485] This invention is a system for efficiently managing household items and generating suggestions. This system mainly consists of terminals and servers, each performing a specific role. The following describes a specific method for implementing this invention.

[0486] First, users use devices such as smartphones or dedicated devices to manage items in their homes. Using these devices, users scan items in their refrigerators and pantries. The devices recognize items using barcode scanners and cameras, and utilize barcode recognition and image recognition technologies to obtain item information (type, quantity, expiration date, etc.). This item information is sent from the device to a server and stored in a central database for later analysis and suggestion generation.

[0487] Next, the user enters their preferences and health information into an application installed on the device. This information includes preferences for specific foods, allergy information, and health status. The device sends the entered user information to a server, which then creates an individual user database based on this information.

[0488] The server analyzes the received item information and individual user information, and uses a generation AI model to generate optimal suggestions for the user. These suggestions might include recipes that prioritize the use of ingredients nearing their expiration date, or meal menus tailored to the user's health condition. The terminal then displays the generated suggestions to the user.

[0489] For example, if the refrigerator contains tomatoes, chicken, and onions, the server will suggest a recipe using the tomatoes that are nearing their expiration date. This reduces food waste while supporting the user's desired meal.

[0490] Another example of a prompt message is: "Please suggest the best recipe using the tomatoes, chicken, and onions I have in my refrigerator. Please also consider menus that prioritize ingredients that are nearing their expiration date." In this way, users can adjust the suggestions and obtain additional information through interactive dialogue, receiving optimal meal suggestions tailored to their individual needs.

[0491] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0492] Step 1:

[0493] Users use a terminal to manage household items, scanning items in their refrigerators and pantries. Input can be barcodes or images of the items. The terminal uses a barcode scanner or camera to acquire barcode images of items and inputs that data. Based on this input, barcode recognition technology and image recognition technology are used to identify item information such as type, quantity, and expiration date. This item information is then output.

[0494] Step 2:

[0495] The terminal transfers the acquired item information to the server. The server receives this information as input and stores it in a central database. Specifically, it uses a database management system to systematically manage and update the information. The stored item information is used for subsequent analysis and proposal generation.

[0496] Step 3:

[0497] Users input their individual preferences and health information using a dedicated application on their device. This includes favorite foods, allergy information, and calorie restrictions. The device receives user information as input and sends it to a server. Based on this transmitted data, the server organizes the information for each user and creates an individual database. This data is used to generate suggestions tailored to the user's needs.

[0498] Step 4:

[0499] The server integrates stored item information and user information and performs analysis using a generative AI model. Specifically, it evaluates the expiration dates of items and the user's dietary preferences to generate optimized suggestions. The output of this process is a list of meal plans and recipes suggested to the user.

[0500] Step 5:

[0501] The terminal presents the user with suggestions from the server. The user passively receives this output and, if necessary, interacts with the server via the terminal to request further information or adjustments. Specifically, the user can select options on the screen and ask additional questions.

[0502] Step 6:

[0503] The server takes user feedback as input, re-evaluates the accuracy of the suggestions using the generative AI model, and makes improvements. This process ensures that future suggestions are better suited to the user's needs. Specifically, it updates the AI ​​model's training data and improves the accuracy of the suggestion generation algorithm.

[0504] (Application Example 1)

[0505] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0506] Managing household items is cumbersome due to the need to check their type, quantity, and expiration dates, and there is a significant problem of food waste, especially food that expires. Furthermore, it is difficult to suggest meals that take into account individual user preferences and health conditions. In addition, current systems lack a means to instantly check item information visually, making it difficult to plan efficient consumption and shopping.

[0507] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0508] In this invention, the server includes means for recognizing household items using an input device and acquiring related data; means for storing the data on the items and organizing and analyzing it based on predetermined conditions; means for acquiring information based on the user's preferences and health status and generating suggestions corresponding to the items and user information; and means for providing item information visually using a smart device and generating visual alerts based on expiration dates. This enables the user to efficiently manage household items and plan appropriate consumption and shopping based on expiration dates.

[0509] An "input device" is a device used to acquire information about an item, and includes barcode readers and cameras.

[0510] "Related data" refers to information about the items, such as their type, quantity, and expiration date.

[0511] "Predetermined conditions" refer to criteria or baseline values ​​that are set in advance for organizing and analyzing data.

[0512] "User preferences" refers to a user's personal tastes and preferences.

[0513] "Health status" refers to information about the user's physical health and dietary restrictions.

[0514] "Suggestions" refer to recipes and consumption plans generated based on the acquired data.

[0515] A "smart device" is a device that can display and operate digital information, and includes smartphones and smart glasses.

[0516] "Visual presentation" refers to a method of displaying information directly in the user's field of vision, allowing for visual confirmation.

[0517] A "visual alert" refers to a visually displayed warning message or highlight.

[0518] "Expiration date" refers to information indicating the period by which an item can be safely consumed.

[0519] This invention aims to realize a system that optimizes the management of household items and provides appropriate suggestions to users. A specific embodiment of this system is described below.

[0520] The server first receives data transmitted from an input device. This input device includes a barcode reader and a camera, which are used to recognize items in the refrigerator or pantry and obtain related data (type, quantity, expiration date).

[0521] The device transmits data to the server using the user's smartphone or smart glasses. At this stage, the user inputs their individual preferences and health status, and this information is also transmitted to the server.

[0522] The server uses an AI model to generate optimal meal suggestions based on acquired item data and user information. These suggestions include recipes that take available ingredients into consideration and menus that prioritize items nearing their expiration date.

[0523] Users receive these suggestions visually via smart devices. For example, smart glasses overlay recipe suggestions directly into their field of vision when they open the refrigerator. Ingredients nearing their expiration date are highlighted with special visual effects.

[0524] For example, when a user takes tomatoes out of the refrigerator, a prompt such as "Please suggest a tomato soup recipe. Is this recipe suitable for tonight's dinner?" appears on the smart glasses. This allows the user to efficiently use ingredients based on the selected suggestion.

[0525] In this way, the system can reduce waste of goods within the home and promote a diet tailored to the user's health and preferences.

[0526] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0527] Step 1:

[0528] The user uses a terminal to scan items in the refrigerator or pantry using an input device. Input options include barcodes attached to the items or recognized image data. The scanned data is automatically converted into relevant information such as item type, quantity, and expiration date. This data is then sent to the server.

[0529] Step 2:

[0530] The server stores the received item data in a central database. At this stage, the data is organized and analyzed according to predetermined conditions based on the information stored in the database. Specifically, it identifies items nearing their expiration date and sets their priority.

[0531] Step 3:

[0532] Users input information about their preferences and health status through their device. This includes allergy information, dietary restrictions, and preferred food types. This information is sent to the server and stored in a database as individual user information.

[0533] Step 4:

[0534] The server uses a generative AI model to generate optimal suggestions from item data and user information. The input data includes item information and user information, and the server uses this data to select recipes and suggest future shopping lists. This output is then formatted in a user-friendly manner.

[0535] Step 5:

[0536] Users receive suggestions visually through their smart devices. At this stage, suggested recipes and ingredients nearing their expiration date are displayed on smart glasses or similar devices. Items that should be consumed as a priority are highlighted, particularly as visual alerts. This display may also appear as interactive prompts on the device.

[0537] Step 6:

[0538] Users either take action based on the suggested information or send feedback to the server if they need more information. The server then uses this feedback to improve the suggestions and incorporate it into future suggestions. This process enhances the accuracy of the suggestions and user satisfaction.

[0539] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0540] This invention provides an advanced suggestion system that takes user emotions into consideration, from managing household items to suggesting meals. A detailed embodiment of this invention is described below.

[0541] This system begins with the user scanning an item using a terminal. The terminal recognizes the item's barcode or image, retrieves the item data, and sends it to a server. This data is stored in a central database. Simultaneously, the user enters personal information, including preferences and health status, into the terminal, which is also stored on the server.

[0542] Furthermore, this system incorporates an emotion engine that analyzes the user's emotions in real time. The emotion engine evaluates the emotional state based on inputs such as voice, facial expressions, and text input that the user makes while operating the device. This allows the server to recognize the user's current emotional state and adjust the content of its suggestions accordingly.

[0543] Specifically, the server combines available item data, user preferences, and emotional data to generate optimized recipes and meal suggestions using generative AI. For example, if the emotional engine determines that the user is stressed, it can generate suggestions using ingredients that have a relaxing effect.

[0544] The suggested recipes and menus are displayed on the device, allowing the user to review them. Furthermore, when the user responds to a suggestion, the emotion engine analyzes that emotion and sends it to the server as new feedback. This allows the server to further improve future suggestions and increase user satisfaction.

[0545] In this system, the generation of shopping lists could also be influenced by emotions. For example, it could be possible to customize suggestions, such as encouraging adventurous grocery shopping when the user is in a positive emotional state.

[0546] In this way, the system supports a sustainable and satisfying diet by efficiently utilizing goods and providing personalized suggestions that respond to the user's emotions.

[0547] The following describes the processing flow.

[0548] Step 1:

[0549] The user opens an application on the device and scans household items with the input device. The device uses barcode and image recognition technology to obtain item data, including the type, quantity, and expiration date of the items.

[0550] Step 2:

[0551] The terminal sends the acquired item data to the server. The server stores the received data in a database and updates the inventory information.

[0552] Step 3:

[0553] Users input individual information such as food preferences, health status, and dietary restrictions on their devices. The device sends this information to the server, which then updates the user profile database.

[0554] Step 4:

[0555] The emotion engine operates, analyzing the user's emotions in real time from the voice, facial expressions, and text they emit while using the device. The analysis results are sent to the server.

[0556] Step 5:

[0557] The server uses the received emotional data to identify the user's current emotional state. Based on this information, the generative AI operates to generate recipes and meal suggestions that take into account item data, preferences, and the user's emotional state.

[0558] Step 6:

[0559] The generated suggestions are sent to the device, allowing the user to visually review the suggested recipes and menus. Sentiment analysis may also be used to suggest meals with relaxing effects.

[0560] Step 7:

[0561] Users provide feedback on the presented suggestions via their device. The emotion engine analyzes this feedback and sends the emotional response to the server.

[0562] Step 8:

[0563] The server analyzes user feedback and emotional responses, and incorporates these into future suggestions. This allows for more personalized suggestions and improves user satisfaction.

[0564] (Example 2)

[0565] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0566] In modern households, consumers are expected to manage numerous items and plan meals, but effectively utilizing available items and providing optimal suggestions tailored to individual health conditions and emotions is challenging. Furthermore, suggesting meals based on an individual's health and psychological state presents an even more complex challenge. Therefore, there is a need for personalized and effective meal management systems that meet individual needs.

[0567] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0568] In this invention, the server includes means for identifying objects in the home environment using an input device and acquiring corresponding information; means for storing the information of the objects and organizing and analyzing it based on set conditions; means for evaluating the user's preferences, health information, and emotional state in real time and generating suggestions according to the object information, user information, and emotional information; means for utilizing the generation AI model and using prompt sentences to make optimized recipe and meal suggestions; and means for displaying the suggestions to the user, acquiring additional information from the dialogue and reactions, and making improvements. This makes it possible to manage meals in a way that is adapted to the health condition and emotions of each individual user while making optimal use of available items.

[0569] An "input device" is a device used to identify objects in a home and acquire related information.

[0570] "Information" refers to various types of data, including data related to objects, as well as data concerning a user's health status, preferences, and emotional state.

[0571] An "object" refers to something tangible, such as household goods or food, that is managed within the home.

[0572] "Storage" means securely saving acquired information and maintaining a state where it can be accessed as needed.

[0573] "Analysis" is the process of organizing acquired information and analyzing and judging it according to a specific purpose.

[0574] "User" refers to a consumer or individual within a household who uses the system.

[0575] "Preferences" refer to information about a user's personal likes and priorities.

[0576] "Health information" refers to information related to the user's physical and mental health.

[0577] "Emotional state" refers to the user's real-time psychological and emotional state.

[0578] A "generative AI model" is an artificial intelligence algorithm or program used for making suggestions or performing analyses.

[0579] A "prompt statement" is an instruction given to a generative AI model to tell it to produce a specific output.

[0580] A "recipe" is a list of steps and ingredients for preparing a meal or dish.

[0581] "Suggestions" refer to recommendations or options presented to users.

[0582] "Dialogue" refers to two-way communication between the user and the system.

[0583] "Response" refers to the opinions and actions that users show towards a proposal or system.

[0584] "Improvement" refers to enhancing system suggestions and functions based on feedback from users.

[0585] This invention provides a system that allows users to manage their home life in a more efficient and personalized way. Detailed embodiments for carrying out the invention are described below.

[0586] The system uses input devices, including cameras and barcode scanners, to identify home appliances and other identifiable physical objects. This automatically acquires and digitizes information about the items. The terminal then transmits this information as packets to the server.

[0587] The server stores acquired item information in a central database, and also incorporates health information, preference information, and real-time emotional information entered by users. This allows for centralized management of a wide range of information related to the user.

[0588] Emotional information is acquired through voice recognition and facial expression analysis software provided by the device. This allows users to receive customized information that is most appropriate for them when receiving recipes or meal suggestions.

[0589] The server uses a generative AI model to optimize suggestions. Instructions are given to the AI ​​model using prompts. For example, if the server sends a prompt such as, "Please suggest a recipe for a dish that will make me feel happy using the ingredients I have in my refrigerator," the AI ​​model will generate a recipe based on the user's emotional state and the ingredients they possess.

[0590] Recipes and meal suggestions generated by the server are displayed to the user on their device. Users can review the suggestions and try the dishes, and the emotion engine analyzes their reactions to the results. This allows the server to improve the accuracy of future suggestions. Specifically, it can suggest using chamomile tea when the user wants to relax.

[0591] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0592] Step 1:

[0593] The terminal identifies objects in the home using a camera or barcode scanner. The input is visual information of the object, and the terminal uses image processing algorithms to identify the type of object and extract its unique ID and related data. This data is packetized as item identification information and sent to the server.

[0594] Step 2:

[0595] The server receives item information sent from the terminal and stores it in the central database. The input consists of item information packets, which the server classifies, organizes, and stores in the database. During this process, the current inventory information is also updated. The output is the updated inventory master.

[0596] Step 3:

[0597] Users input health information, preference information, and emotional information through their devices. This includes text input about their health status and favorite foods, as well as voice and facial expression data indicating their emotional state. The device analyzes this data using an emotion engine and sends it to the server as a dataset.

[0598] Step 4:

[0599] The server uses an emotion engine to analyze the user's emotional state. The server analyzes the input voice, facial expressions, and text data to evaluate the user's emotional state. This information is processed together with other user information. The output is data indicating the user's emotional state.

[0600] Step 5:

[0601] The server uses a generative AI model to input prompts based on received item information, user information, and sentiment data, and creates optimized suggestions. An example prompt might be, "Please suggest a relaxing recipe using ingredients found in the refrigerator," and the generative AI model would output recipe data.

[0602] Step 6:

[0603] The server sends the generated recipe to the terminal and displays it to the user. The output is a visualized meal suggestion, which the user can refer to and select a dish from.

[0604] Step 7:

[0605] Users cook according to a recipe and provide feedback on the process and results via their device. This includes inputting comments via voice or text, which are then analyzed by an emotion engine on the device. The feedback is sent to the server as new data.

[0606] Step 8:

[0607] The server analyzes the collected feedback and makes improvements to enhance the accuracy of future proposals. This analysis prepares the readjusted proposal data for the next cycle. The output is the improved proposal algorithm.

[0608] (Application Example 2)

[0609] Next, we will explain Application Example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0610] There is a need for a system that can efficiently manage household items and, based on that, provide meal suggestions tailored to the user's emotional state. While conventional systems could manage items and make suggestions based on user preferences, they could not consider the user's real-time emotional state, making it difficult to provide highly satisfying suggestions.

[0611] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0612] In this invention, the server includes means for recognizing household items using an input device and acquiring related data; means for storing the data on the items and organizing and analyzing it based on predetermined conditions; means for acquiring information based on the user's preferences and health status and generating suggestions corresponding to the items and user information; means for analyzing the user's emotional state and adjusting the content of the suggestions based on the emotional state; and means for presenting the suggestions to the user, acquiring additional information through dialogue, and making improvements. This enables personalized meal suggestions that take into account the user's item management and emotional state.

[0613] An "input device" is a device used to recognize household items and acquire information related to those items.

[0614] "Item data" refers to data that includes information for identifying an item and detailed information related to the item.

[0615] "Prescribed conditions" refer to specific criteria or rules set for organizing and analyzing item data.

[0616] "User preferences" refers to information about ingredients, dishes, or personal tendencies and preferences regarding food that users particularly like.

[0617] "Health status" refers to information related to the user's physical health, specifically including allergies, dietary restrictions, and required nutrients.

[0618] "Adjusting the suggested menu" is the process of optimizing meal suggestions based on the user's real-time emotional state.

[0619] "Emotional state" refers to information that indicates the user's current psychological or emotional condition.

[0620] "Means of obtaining additional information and making improvements through dialogue" refers to a process of checking users' reactions to proposals and using the feedback received to make future proposals more suitable for users.

[0621] The system that realizes this invention consists of a series of processes to enable household item management and meal suggestions based on the user's emotions. The system is mainly composed of an input device, a server, and a user terminal.

[0622] The device, for example, takes the form of a smartphone or smart speaker, and acquires item data by photographing or scanning items. This item data includes information obtained using barcodes and image recognition technology. This data records the items and is transmitted to a central server.

[0623] The server stores and analyzes received item data and user preference and health information provided through the device. An emotion engine is integrated into the server to analyze the user's emotional state in real time from voice, facial expressions, and text input as they operate the device. This data is used to assess the emotional state and forms the basis for adjusting meal recommendations.

[0624] The generative AI model generates optimized recipes based on available item data on the server, user preferences, and emotional data. For example, if a user is looking to relax after returning home from work, the generative AI might suggest a relaxing herbal tea chicken salad.

[0625] Users can view suggested recipes on their devices and, if necessary, automatically order ingredients by linking with a local refrigerator locker. Furthermore, user feedback on the suggestions is used to improve future suggestions.

[0626] For example, a prompt might take the form of "Please suggest a relaxing recipe for when the user is feeling stressed." By using this prompt, the generating AI can suggest the most suitable meal.

[0627] In this way, the system provides personalized meal suggestions that respond to the user's emotions, along with the efficient use of goods.

[0628] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0629] Step 1:

[0630] The device scans household items and retrieves data related to those items. It uses barcodes or image data of items as input and generates item identification information and associated data as output. Specifically, the device reads barcodes using a camera or scanner and retrieves the corresponding data from a cloud service or on-device database.

[0631] Step 2:

[0632] The server stores, organizes, and analyzes the acquired item data. It receives item identification information as input and generates organized database entries as output. In this step, the server uses a database management system to record item data and organizes it into categories based on predetermined rules.

[0633] Step 3:

[0634] Users input information about their preferences and health status through their devices. Inputs include forms and voice input data, and output generates user profile information. This process involves using an application on the device to ask about the user's preferences and sending the entered responses to a server.

[0635] Step 4:

[0636] The server acquires real-time emotion data to analyze the user's emotional state. It uses voice, facial expressions, and text data as input and generates emotion state data as output. Specifically, it uses an emotion engine to analyze the voice and images input by the user and quantify the emotional tendencies.

[0637] Step 5:

[0638] The server uses a generative AI model to suggest optimal recipes based on item data, preferences, health information, and emotional data. It uses each dataset as input and generates personalized recipes as output. In this step, the generative AI is instructed with the prompt "Suggest a relaxing recipe for when the user is feeling stressed," and the generated recipes are sent to the user's terminal.

[0639] Step 6:

[0640] Users view the suggested recipes on their devices and provide feedback. The system receives the suggested recipes as input and generates improvement feedback as output. Specifically, users evaluate the recipes, send the feedback to the server information system, and contribute to improving the accuracy of future suggestions.

[0641] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0642] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0643] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0644] [Fourth Embodiment]

[0645] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0646] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0647] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0648] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0649] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0650] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0651] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0652] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0653] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0654] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0655] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0656] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0657] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0658] This invention provides a system for efficiently managing household items and generating suggestions. The following describes in detail how this system is implemented.

[0659] First, the user uses a smartphone or dedicated device as a terminal to scan items in the refrigerator or pantry with an input device. This system uses barcode and image recognition technology to recognize the items and automatically acquires item data such as type, quantity, and expiration date. This data is sent from the terminal to a server and stored in a central database.

[0660] Next, the user enters details such as their individual preferences, health status, and eating habits into an application on their device. The server uses this information to create a database of individual user information.

[0661] The server analyzes acquired item data and user information to generate optimal suggestions. These suggestions include recipes that take available ingredients into account and menus using items nearing their expiration date. Users can view these suggestions through their devices and, if adjustments are needed, can contact the server using an interactive dialogue function. The server incorporates user feedback to improve recipes and suggestions.

[0662] As a concrete example, suppose a user scans tomatoes, chicken, and onions in their refrigerator, and the system suggests a recipe using the tomatoes and chicken. In this case, the server prioritizes the tomatoes, which are nearing their expiration date, and includes them in the suggestion. The user prepares the dish based on the suggested recipe, asking the system for cooking tips if needed. Furthermore, the information presented here, based on the remaining ingredients, helps the user generate a shopping list and streamline their next grocery shopping trip.

[0663] In this way, users can enjoy meals tailored to their individual preferences and health needs while making efficient use of household items. This system reduces food waste and contributes to promoting sustainable eating habits.

[0664] The following describes the processing flow.

[0665] Step 1:

[0666] The user launches an application on their device and scans household items using an input device. The device then reads the barcode or image of the item and retrieves item data, including its type, quantity, and expiration date.

[0667] Step 2:

[0668] The terminal sends the acquired item data to the cloud. This data reaches the server and is organized and stored in a central database. The server updates the item data, keeping the inventory status constantly up-to-date.

[0669] Step 3:

[0670] The user navigates to the user profile section within the application on their device and enters their preferences, health information, and dietary restrictions. The server then creates a user-specific profile and retains this information.

[0671] Step 4:

[0672] The server analyzes stored item data and checks expiration dates. It identifies items that should be used as a priority and generates alerts and notifications when necessary.

[0673] Step 5:

[0674] The server utilizes a generation AI to create recipes using available items, based on the user's profile. In particular, it actively incorporates items nearing their expiration date into the recipes.

[0675] Step 6:

[0676] The server sends generated recipes and meal suggestions to the terminal and presents them to the user. The terminal then shows the displayed suggestions to the user, allowing the user to choose their preferred meal from multiple options.

[0677] Step 7:

[0678] When a user asks a question or requests customization, the device sends it to the server. The server then prepares answers and advice based on the request, improving the user's experience.

[0679] Step 8:

[0680] After a recipe is suggested, the user can provide feedback via their device. The server captures this feedback and uses it to improve the accuracy of future suggestions.

[0681] (Example 1)

[0682] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0683] In managing household supplies, there are problems such as food waste due to expired food and difficulty in managing ingredients that meet the individual dietary needs of users. Furthermore, it is difficult to keep track of daily food shortages and how to use them appropriately, making efficient meal management and shopping planning a challenge.

[0684] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0685] In this invention, the server includes means for recognizing household items using input power and acquiring related information; means for storing the item information and organizing and analyzing it based on predetermined criteria; and means for acquiring information based on the user's preferences and health status and using a generative machine learning model to generate suggestions corresponding to the items and user information. This enables the management of expiration dates and the generation of meal suggestions optimized for the user.

[0686] "Input power" refers to the energy source used to recognize information about an object, enabling the operation of a device.

[0687] "Item information" refers to detailed data about household items, including information such as type, quantity, and expiration date.

[0688] "Prescribed standards" refer to the rules and conditions that are followed when organizing and analyzing product information, and are set to ensure data consistency and usefulness.

[0689] "Generative machine learning models" refer to AI technology used to generate optimal suggestions for users by utilizing collected data.

[0690] "Expiration date" refers to the period during which food products and other items can be used safely and deliciously, and is the period during which use or consumption is recommended.

[0691] A "meal plan" refers to a menu and schedule of meals designed based on the user's dietary needs, and is a means of efficiently managing daily eating habits.

[0692] "Purchase records" refer to data that records the purchase of necessary items based on a meal plan, and are useful for formulating future shopping plans.

[0693] This invention is a system for efficiently managing household items and generating suggestions. This system mainly consists of terminals and servers, each performing a specific role. The following describes a specific method for implementing this invention.

[0694] First, users use devices such as smartphones or dedicated devices to manage items in their homes. Using these devices, users scan items in their refrigerators and pantries. The devices recognize items using barcode scanners and cameras, and utilize barcode recognition and image recognition technologies to obtain item information (type, quantity, expiration date, etc.). This item information is sent from the device to a server and stored in a central database for later analysis and suggestion generation.

[0695] Next, the user enters their preferences and health information into an application installed on the device. This information includes preferences for specific foods, allergy information, and health status. The device sends the entered user information to a server, which then creates an individual user database based on this information.

[0696] The server analyzes the received item information and individual user information, and uses a generation AI model to generate optimal suggestions for the user. These suggestions might include recipes that prioritize the use of ingredients nearing their expiration date, or meal menus tailored to the user's health condition. The terminal then displays the generated suggestions to the user.

[0697] For example, if the refrigerator contains tomatoes, chicken, and onions, the server will suggest a recipe using the tomatoes that are nearing their expiration date. This reduces food waste while supporting the user's desired meal.

[0698] Another example of a prompt message is: "Please suggest the best recipe using the tomatoes, chicken, and onions I have in my refrigerator. Please also consider menus that prioritize ingredients that are nearing their expiration date." In this way, users can adjust the suggestions and obtain additional information through interactive dialogue, receiving optimal meal suggestions tailored to their individual needs.

[0699] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0700] Step 1:

[0701] Users use a terminal to manage household items, scanning items in their refrigerators and pantries. Input can be barcodes or images of the items. The terminal uses a barcode scanner or camera to acquire barcode images of items and inputs that data. Based on this input, barcode recognition technology and image recognition technology are used to identify item information such as type, quantity, and expiration date. This item information is then output.

[0702] Step 2:

[0703] The terminal transfers the acquired item information to the server. The server receives this information as input and stores it in a central database. Specifically, it uses a database management system to systematically manage and update the information. The stored item information is used for subsequent analysis and proposal generation.

[0704] Step 3:

[0705] Users input their individual preferences and health information using a dedicated application on their device. This includes favorite foods, allergy information, and calorie restrictions. The device receives user information as input and sends it to a server. Based on this transmitted data, the server organizes the information for each user and creates an individual database. This data is used to generate suggestions tailored to the user's needs.

[0706] Step 4:

[0707] The server integrates stored item information and user information and performs analysis using a generative AI model. Specifically, it evaluates the expiration dates of items and the user's dietary preferences to generate optimized suggestions. The output of this process is a list of meal plans and recipes suggested to the user.

[0708] Step 5:

[0709] The terminal presents the user with suggestions from the server. The user passively receives this output and, if necessary, interacts with the server via the terminal to request further information or adjustments. Specifically, the user can select options on the screen and ask additional questions.

[0710] Step 6:

[0711] The server takes user feedback as input, re-evaluates the accuracy of the suggestions using the generative AI model, and makes improvements. This process ensures that future suggestions are better suited to the user's needs. Specifically, it updates the AI ​​model's training data and improves the accuracy of the suggestion generation algorithm.

[0712] (Application Example 1)

[0713] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0714] Managing household items is cumbersome due to the need to check their type, quantity, and expiration dates, and there is a significant problem of food waste, especially food that expires. Furthermore, it is difficult to suggest meals that take into account individual user preferences and health conditions. In addition, current systems lack a means to instantly check item information visually, making it difficult to plan efficient consumption and shopping.

[0715] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0716] In this invention, the server includes means for recognizing household items using an input device and acquiring related data; means for storing the data on the items and organizing and analyzing it based on predetermined conditions; means for acquiring information based on the user's preferences and health status and generating suggestions corresponding to the items and user information; and means for providing item information visually using a smart device and generating visual alerts based on expiration dates. This enables the user to efficiently manage household items and plan appropriate consumption and shopping based on expiration dates.

[0717] An "input device" is a device used to acquire information about an item, and includes barcode readers and cameras.

[0718] "Related data" refers to information about the items, such as their type, quantity, and expiration date.

[0719] "Predetermined conditions" refer to criteria or baseline values ​​that are set in advance for organizing and analyzing data.

[0720] "User preferences" refers to a user's personal tastes and preferences.

[0721] "Health status" refers to information about the user's physical health and dietary restrictions.

[0722] "Suggestions" refer to recipes and consumption plans generated based on the acquired data.

[0723] A "smart device" is a device that can display and operate digital information, and includes smartphones and smart glasses.

[0724] "Visual presentation" refers to a method of displaying information directly in the user's field of vision, allowing for visual confirmation.

[0725] A "visual alert" refers to a visually displayed warning message or highlight.

[0726] "Expiration date" refers to information indicating the period by which an item can be safely consumed.

[0727] This invention aims to realize a system that optimizes the management of household items and provides appropriate suggestions to users. A specific embodiment of this system is described below.

[0728] The server first receives data transmitted from an input device. This input device includes a barcode reader and a camera, which are used to recognize items in the refrigerator or pantry and obtain related data (type, quantity, expiration date).

[0729] The device transmits data to the server using the user's smartphone or smart glasses. At this stage, the user inputs their individual preferences and health status, and this information is also transmitted to the server.

[0730] The server uses an AI model to generate optimal meal suggestions based on acquired item data and user information. These suggestions include recipes that take available ingredients into consideration and menus that prioritize items nearing their expiration date.

[0731] Users receive these suggestions visually via smart devices. For example, smart glasses overlay recipe suggestions directly into their field of vision when they open the refrigerator. Ingredients nearing their expiration date are highlighted with special visual effects.

[0732] For example, when a user takes tomatoes out of the refrigerator, a prompt such as "Please suggest a tomato soup recipe. Is this recipe suitable for tonight's dinner?" appears on the smart glasses. This allows the user to efficiently use ingredients based on the selected suggestion.

[0733] In this way, the system can reduce waste of goods within the home and promote a diet tailored to the user's health and preferences.

[0734] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0735] Step 1:

[0736] The user uses a terminal to scan items in the refrigerator or pantry using an input device. Input options include barcodes attached to the items or recognized image data. The scanned data is automatically converted into relevant information such as item type, quantity, and expiration date. This data is then sent to the server.

[0737] Step 2:

[0738] The server stores the received item data in a central database. At this stage, the data is organized and analyzed according to predetermined conditions based on the information stored in the database. Specifically, it identifies items nearing their expiration date and sets their priority.

[0739] Step 3:

[0740] Users input information about their preferences and health status through their device. This includes allergy information, dietary restrictions, and preferred food types. This information is sent to the server and stored in a database as individual user information.

[0741] Step 4:

[0742] The server uses a generative AI model to generate optimal suggestions from item data and user information. The input data includes item information and user information, and the server uses this data to select recipes and suggest future shopping lists. This output is then formatted in a user-friendly manner.

[0743] Step 5:

[0744] Users receive suggestions visually through their smart devices. At this stage, suggested recipes and ingredients nearing their expiration date are displayed on smart glasses or similar devices. Items that should be consumed as a priority are highlighted, particularly as visual alerts. This display may also appear as interactive prompts on the device.

[0745] Step 6:

[0746] Users either take action based on the suggested information or send feedback to the server if they need more information. The server then uses this feedback to improve the suggestions and incorporate it into future suggestions. This process enhances the accuracy of the suggestions and user satisfaction.

[0747] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0748] This invention provides an advanced suggestion system that takes user emotions into consideration, from managing household items to suggesting meals. A detailed embodiment of this invention is described below.

[0749] This system begins with the user scanning an item using a terminal. The terminal recognizes the item's barcode or image, retrieves the item data, and sends it to a server. This data is stored in a central database. Simultaneously, the user enters personal information, including preferences and health status, into the terminal, which is also stored on the server.

[0750] Furthermore, this system incorporates an emotion engine that analyzes the user's emotions in real time. The emotion engine evaluates the emotional state based on inputs such as voice, facial expressions, and text input that the user makes while operating the device. This allows the server to recognize the user's current emotional state and adjust the content of its suggestions accordingly.

[0751] Specifically, the server combines available item data, user preferences, and emotional data to generate optimized recipes and meal suggestions using generative AI. For example, if the emotional engine determines that the user is stressed, it can generate suggestions using ingredients that have a relaxing effect.

[0752] The suggested recipes and menus are displayed on the device, allowing the user to review them. Furthermore, when the user responds to a suggestion, the emotion engine analyzes that emotion and sends it to the server as new feedback. This allows the server to further improve future suggestions and increase user satisfaction.

[0753] In this system, the generation of shopping lists could also be influenced by emotions. For example, it could be possible to customize suggestions, such as encouraging adventurous grocery shopping when the user is in a positive emotional state.

[0754] In this way, the system supports a sustainable and satisfying diet by efficiently utilizing goods and providing personalized suggestions that respond to the user's emotions.

[0755] The following describes the processing flow.

[0756] Step 1:

[0757] The user opens an application on the device and scans household items with the input device. The device uses barcode and image recognition technology to obtain item data, including the type, quantity, and expiration date of the items.

[0758] Step 2:

[0759] The terminal sends the acquired item data to the server. The server stores the received data in a database and updates the inventory information.

[0760] Step 3:

[0761] Users input individual information such as food preferences, health status, and dietary restrictions on their devices. The device sends this information to the server, which then updates the user profile database.

[0762] Step 4:

[0763] The emotion engine operates, analyzing the user's emotions in real time from the voice, facial expressions, and text they emit while using the device. The analysis results are sent to the server.

[0764] Step 5:

[0765] The server uses the received emotional data to identify the user's current emotional state. Based on this information, the generative AI operates to generate recipes and meal suggestions that take into account item data, preferences, and the user's emotional state.

[0766] Step 6:

[0767] The generated suggestions are sent to the device, allowing the user to visually review the suggested recipes and menus. Sentiment analysis may also be used to suggest meals with relaxing effects.

[0768] Step 7:

[0769] Users provide feedback on the presented suggestions via their device. The emotion engine analyzes this feedback and sends the emotional response to the server.

[0770] Step 8:

[0771] The server analyzes user feedback and emotional responses, and incorporates these into future suggestions. This allows for more personalized suggestions and improves user satisfaction.

[0772] (Example 2)

[0773] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0774] In modern households, consumers are expected to manage numerous items and plan meals, but effectively utilizing available items and providing optimal suggestions tailored to individual health conditions and emotions is challenging. Furthermore, suggesting meals based on an individual's health and psychological state presents an even more complex challenge. Therefore, there is a need for personalized and effective meal management systems that meet individual needs.

[0775] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0776] In this invention, the server includes means for identifying objects in the home environment using an input device and acquiring corresponding information; means for storing the information of the objects and organizing and analyzing it based on set conditions; means for evaluating the user's preferences, health information, and emotional state in real time and generating suggestions according to the object information, user information, and emotional information; means for utilizing the generation AI model and using prompt sentences to make optimized recipe and meal suggestions; and means for displaying the suggestions to the user, acquiring additional information from the dialogue and reactions, and making improvements. This makes it possible to manage meals in a way that is adapted to the health condition and emotions of each individual user while making optimal use of available items.

[0777] An "input device" is a device used to identify objects in a home and acquire related information.

[0778] "Information" refers to various types of data, including data related to objects, as well as data concerning a user's health status, preferences, and emotional state.

[0779] An "object" refers to something tangible, such as household goods or food, that is managed within the home.

[0780] "Storage" means securely saving acquired information and maintaining a state where it can be accessed as needed.

[0781] "Analysis" is the process of organizing acquired information and analyzing and judging it according to a specific purpose.

[0782] "User" refers to a consumer or individual within a household who uses the system.

[0783] "Preferences" refer to information about a user's personal likes and priorities.

[0784] "Health information" refers to information related to the user's physical and mental health.

[0785] "Emotional state" refers to the user's real-time psychological and emotional state.

[0786] A "generative AI model" is an artificial intelligence algorithm or program used for making suggestions or performing analyses.

[0787] A "prompt statement" is an instruction given to a generative AI model to tell it to produce a specific output.

[0788] A "recipe" is a list of steps and ingredients for preparing a meal or dish.

[0789] "Suggestions" refer to recommendations or options presented to users.

[0790] "Dialogue" refers to two-way communication between the user and the system.

[0791] "Response" refers to the opinions and actions that users show towards a proposal or system.

[0792] "Improvement" refers to enhancing system suggestions and functions based on feedback from users.

[0793] This invention provides a system that allows users to manage their home life in a more efficient and personalized way. Detailed embodiments for carrying out the invention are described below.

[0794] The system uses input devices, including cameras and barcode scanners, to identify home appliances and other identifiable physical objects. This automatically acquires and digitizes information about the items. The terminal then transmits this information as packets to the server.

[0795] The server stores acquired item information in a central database, and also incorporates health information, preference information, and real-time emotional information entered by users. This allows for centralized management of a wide range of information related to the user.

[0796] Emotional information is acquired through voice recognition and facial expression analysis software provided by the device. This allows users to receive customized information that is most appropriate for them when receiving recipes or meal suggestions.

[0797] The server uses a generative AI model to optimize suggestions. Instructions are given to the AI ​​model using prompts. For example, if the server sends a prompt such as, "Please suggest a recipe for a dish that will make me feel happy using the ingredients I have in my refrigerator," the AI ​​model will generate a recipe based on the user's emotional state and the ingredients they possess.

[0798] Recipes and meal suggestions generated by the server are displayed to the user on their device. Users can review the suggestions and try the dishes, and the emotion engine analyzes their reactions to the results. This allows the server to improve the accuracy of future suggestions. Specifically, it can suggest using chamomile tea when the user wants to relax.

[0799] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0800] Step 1:

[0801] The terminal identifies objects in the home using a camera or barcode scanner. The input is visual information of the object, and the terminal uses image processing algorithms to identify the type of object and extract its unique ID and related data. This data is packetized as item identification information and sent to the server.

[0802] Step 2:

[0803] The server receives item information sent from the terminal and stores it in the central database. The input consists of item information packets, which the server classifies, organizes, and stores in the database. During this process, the current inventory information is also updated. The output is the updated inventory master.

[0804] Step 3:

[0805] Users input health information, preference information, and emotional information through their devices. This includes text input about their health status and favorite foods, as well as voice and facial expression data indicating their emotional state. The device analyzes this data using an emotion engine and sends it to the server as a dataset.

[0806] Step 4:

[0807] The server uses an emotion engine to analyze the user's emotional state. The server analyzes the input voice, facial expressions, and text data to evaluate the user's emotional state. This information is processed together with other user information. The output is data indicating the user's emotional state.

[0808] Step 5:

[0809] The server uses a generative AI model to input prompts based on received item information, user information, and sentiment data, and creates optimized suggestions. An example prompt might be, "Please suggest a relaxing recipe using ingredients found in the refrigerator," and the generative AI model would output recipe data.

[0810] Step 6:

[0811] The server sends the generated recipe to the terminal and displays it to the user. The output is a visualized meal suggestion, which the user can refer to and select a dish from.

[0812] Step 7:

[0813] Users cook according to a recipe and provide feedback on the process and results via their device. This includes inputting comments via voice or text, which are then analyzed by an emotion engine on the device. The feedback is sent to the server as new data.

[0814] Step 8:

[0815] The server analyzes the collected feedback and makes improvements to enhance the accuracy of future proposals. This analysis prepares the readjusted proposal data for the next cycle. The output is the improved proposal algorithm.

[0816] (Application Example 2)

[0817] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0818] There is a need for a system that can efficiently manage household items and, based on that, provide meal suggestions tailored to the user's emotional state. While conventional systems could manage items and make suggestions based on user preferences, they could not consider the user's real-time emotional state, making it difficult to provide highly satisfying suggestions.

[0819] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0820] In this invention, the server includes means for recognizing household items using an input device and acquiring related data; means for storing the data on the items and organizing and analyzing it based on predetermined conditions; means for acquiring information based on the user's preferences and health status and generating suggestions corresponding to the items and user information; means for analyzing the user's emotional state and adjusting the content of the suggestions based on the emotional state; and means for presenting the suggestions to the user, acquiring additional information through dialogue, and making improvements. This enables personalized meal suggestions that take into account the user's item management and emotional state.

[0821] An "input device" is a device used to recognize household items and acquire information related to those items.

[0822] "Item data" refers to data that includes information for identifying an item and detailed information related to the item.

[0823] "Prescribed conditions" refer to specific criteria or rules set for organizing and analyzing item data.

[0824] "User preferences" refers to information about ingredients, dishes, or personal tendencies and preferences regarding food that users particularly like.

[0825] "Health status" refers to information related to the user's physical health, specifically including allergies, dietary restrictions, and required nutrients.

[0826] "Adjusting the suggested menu" is the process of optimizing meal suggestions based on the user's real-time emotional state.

[0827] "Emotional state" refers to information that indicates the user's current psychological or emotional condition.

[0828] "Means of obtaining additional information and making improvements through dialogue" refers to a process of checking users' reactions to proposals and using the feedback received to make future proposals more suitable for users.

[0829] The system that realizes this invention consists of a series of processes to enable household item management and meal suggestions based on the user's emotions. The system is mainly composed of an input device, a server, and a user terminal.

[0830] The device, for example, takes the form of a smartphone or smart speaker, and acquires item data by photographing or scanning items. This item data includes information obtained using barcodes and image recognition technology. This data records the items and is transmitted to a central server.

[0831] The server stores and analyzes received item data and user preference and health information provided through the device. An emotion engine is integrated into the server to analyze the user's emotional state in real time from voice, facial expressions, and text input as they operate the device. This data is used to assess the emotional state and forms the basis for adjusting meal recommendations.

[0832] The generative AI model generates optimized recipes based on available item data on the server, user preferences, and emotional data. For example, if a user is looking to relax after returning home from work, the generative AI might suggest a relaxing herbal tea chicken salad.

[0833] Users can view suggested recipes on their devices and, if necessary, automatically order ingredients by linking with a local refrigerator locker. Furthermore, user feedback on the suggestions is used to improve future suggestions.

[0834] For example, a prompt might take the form of "Please suggest a relaxing recipe for when the user is feeling stressed." By using this prompt, the generating AI can suggest the most suitable meal.

[0835] In this way, the system provides personalized meal suggestions that respond to the user's emotions, along with the efficient use of goods.

[0836] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0837] Step 1:

[0838] The device scans household items and retrieves data related to those items. It uses barcodes or image data of items as input and generates item identification information and associated data as output. Specifically, the device reads barcodes using a camera or scanner and retrieves the corresponding data from a cloud service or on-device database.

[0839] Step 2:

[0840] The server stores, organizes, and analyzes the acquired item data. It receives item identification information as input and generates organized database entries as output. In this step, the server uses a database management system to record item data and organizes it into categories based on predetermined rules.

[0841] Step 3:

[0842] Users input information about their preferences and health status through their devices. Inputs include forms and voice input data, and output generates user profile information. This process involves using an application on the device to ask about the user's preferences and sending the entered responses to a server.

[0843] Step 4:

[0844] The server acquires real-time emotion data to analyze the user's emotional state. It uses voice, facial expressions, and text data as input and generates emotion state data as output. Specifically, it uses an emotion engine to analyze the voice and images input by the user and quantify the emotional tendencies.

[0845] Step 5:

[0846] The server uses a generative AI model to suggest optimal recipes based on item data, preferences, health information, and emotional data. It uses each dataset as input and generates personalized recipes as output. In this step, the generative AI is instructed with the prompt "Suggest a relaxing recipe for when the user is feeling stressed," and the generated recipes are sent to the user's terminal.

[0847] Step 6:

[0848] Users view the suggested recipes on their devices and provide feedback. The system receives the suggested recipes as input and generates improvement feedback as output. Specifically, users evaluate the recipes, send the feedback to the server information system, and contribute to improving the accuracy of future suggestions.

[0849] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0850] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0851] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0852] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0853] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0854] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0855] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0856] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0857] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0858] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0859] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0860] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0861] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0862] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0863] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0864] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0865] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0866] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0867] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0868] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0869] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0870] The following is further disclosed regarding the embodiments described above.

[0871] (Claim 1)

[0872] A means of recognizing household items using an input device and acquiring related data,

[0873] Means for storing data on the aforementioned articles and organizing and analyzing it based on predetermined conditions,

[0874] A means for acquiring information based on the user's preferences and health status, and for generating suggestions corresponding to the said goods and user information,

[0875] A means of presenting the aforementioned proposal to the user, obtaining additional information through dialogue, and making improvements,

[0876] A system that includes this.

[0877] (Claim 2)

[0878] The system according to claim 1, characterized in that the proposed items include items to be used preferentially based on their expiration date.

[0879] (Claim 3)

[0880] The system according to claim 1, characterized in that the proposal includes a recipe configured based on predetermined dietary needs, and generates a shopping record using information on missing items based on the recipe.

[0881] "Example 1"

[0882] (Claim 1)

[0883] A means of recognizing household items using input power and acquiring related information,

[0884] Means for storing the aforementioned item information and organizing and analyzing it based on predetermined standards,

[0885] A means for acquiring information based on the user's preferences and health status, and for using a generative machine learning model to generate suggestions corresponding to the said items and user information,

[0886] A means of presenting the aforementioned proposal to the user, obtaining additional information through dialogue, and making improvements,

[0887] A system that includes this.

[0888] (Claim 2)

[0889] The system according to claim 1, characterized in that the proposed items include items to be used preferentially based on their expiration date.

[0890] (Claim 3)

[0891] The system according to claim 1, wherein the proposal includes a meal plan configured based on predetermined dietary needs, and generates a purchase record using information on shortages of items based on the meal plan.

[0892] "Application Example 1"

[0893] (Claim 1)

[0894] A means of recognizing household items using an input device and acquiring related data,

[0895] Means for storing data on the aforementioned articles and organizing and analyzing it based on predetermined conditions,

[0896] A means for acquiring information based on the user's preferences and health status, and for generating suggestions corresponding to the said goods and user information,

[0897] A means of presenting the aforementioned proposal to the user, obtaining additional information through dialogue, and making improvements,

[0898] A means of providing product information visually using a smart device and generating visual alerts based on expiration dates,

[0899] A system that includes this.

[0900] (Claim 2)

[0901] The system according to claim 1, characterized in that the proposed items include items to be used preferentially based on their expiration date and are displayed through a visual device.

[0902] (Claim 3)

[0903] The system according to claim 1, characterized in that the proposal includes cooking instructions configured based on predetermined meal needs, and generates a purchase record using information on shortages of items based on the instructions.

[0904] "Example 2 of combining an emotion engine"

[0905] (Claim 1)

[0906] A means for identifying objects in the home environment using an input device and obtaining corresponding information,

[0907] Means for storing information about the aforementioned object and organizing and analyzing it based on set conditions,

[0908] A means for evaluating the user's preferences and health information, as well as their emotional state in real time, and generating suggestions based on the aforementioned object information, user information, and emotional information,

[0909] A means for providing optimized recipes and meal suggestions using prompt sentences, utilizing the aforementioned generation AI model,

[0910] A means of displaying the aforementioned proposal to the user, obtaining additional information from the dialogue and responses, and making improvements,

[0911] A system that includes this.

[0912] (Claim 2)

[0913] The system according to claim 1, characterized in that the proposal includes an object that is used preferentially based on its expiration date.

[0914] (Claim 3)

[0915] The system according to claim 1, characterized in that the proposal includes a recipe configured based on a specific dietary requirement, and generates a purchase record using information on missing items based on the recipe.

[0916] "Application example 2 when combining with an emotional engine"

[0917] (Claim 1)

[0918] A means of recognizing household items using an input device and acquiring related data,

[0919] Means for storing data on the aforementioned articles and organizing and analyzing it based on predetermined conditions,

[0920] A means for acquiring information based on the user's preferences and health status, and for generating suggestions corresponding to the said goods and user information,

[0921] A means for analyzing the user's emotional state and adjusting the proposed content based on the said emotional state,

[0922] A means of presenting the aforementioned proposal to the user, obtaining additional information through dialogue, and making improvements,

[0923] A system that includes this.

[0924] (Claim 2)

[0925] The system according to claim 1, characterized in that the proposed items include items to be used preferentially based on their expiration date.

[0926] (Claim 3)

[0927] The system according to claim 1, characterized in that the proposal includes a recipe configured based on predetermined dietary needs, and generates a shopping record using information on missing items based on the recipe. [Explanation of Symbols]

[0928] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of recognizing household items using an input device and acquiring related data, Means for storing data on the aforementioned articles and organizing and analyzing it based on predetermined conditions, A means for acquiring information based on the user's preferences and health status, and for generating suggestions corresponding to the said goods and user information, A means of presenting the aforementioned proposal to the user, obtaining additional information through dialogue, and making improvements, A means of providing product information visually using a smart device and generating visual alerts based on expiration dates, A system that includes this.

2. The system according to claim 1, characterized in that the proposed items include items to be used preferentially based on their expiration date and are displayed through a visual device.

3. The system according to claim 1, characterized in that the proposal includes cooking instructions configured based on predetermined meal needs, and generates a purchase record using information on shortages of items based on the instructions.

Images