system

Abstract

We provide the system. [Solution] An information processing means that receives voice input and converts the voice data into text data, An information processing means that analyzes the converted text data and checks the inventory status of the corresponding materials, An information processing system that selects the optimal materials based on user settings and places automatic orders, An information processing means that obtains expiration date information and notifies the user that the expiration date is approaching, An information processing tool that analyzes the purchase data of multiple users within a region and proposes collaborative purchasing, An information processing means for a household automated device that has the function of interacting with the user via voice and notifying them of inventory information, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor, the method including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In recent years, the management of consumables in households has played an important role in improving the efficiency of life and reducing waste. However, manual inventory checking and ordering require labor, and a great deal of time and effort are often spent, especially in managing expiration dates and identifying opportunities for joint purchases. In addition, there are problems such as the disposal of consumables whose expiration dates have passed and economic losses due to unnecessary inventory holding. Furthermore, from the perspective of improving the efficiency of logistics and sustainability within a region, efficient coordination of joint purchases and deliveries is required.

Means for Solving the Problems

[0005] This invention provides a system that enables users to efficiently manage consumables without effort by using a simple inventory management system via voice input and an AI-powered automatic ordering mechanism. Furthermore, it prevents the disposal of expired consumables by acquiring expiration date information and automatically notifying users. In addition, by analyzing data from users within a region to propose joint purchases and adjust delivery dates, it enables improved logistics efficiency and reduced environmental impact. The aim is to realize a more sustainable and economical lifestyle.

[0006] "Voice input" is a method of transmitting instructions and information to an information processing device via voice.

[0007] "Audio data" refers to data that represents voice input in a digital format.

[0008] "Text data" refers to information that has been converted into text format by speech recognition.

[0009] An "information processing device" is a device used to analyze and process given information, and includes digital devices and computers.

[0010] "Inventory status" refers to information indicating the current quantity and condition of a particular item being stored.

[0011] "User settings" refer to the conditions and priorities that users can pre-configure when using the system.

[0012] The "optimal item" is the product that best matches the user's specified criteria and is deemed suitable for purchase at this time.

[0013] "Automated ordering" refers to the process by which a system places an order for goods based on predetermined conditions, without requiring any user intervention.

[0014] "Expiration date information" refers to data indicating the date by which a purchased product becomes unsuitable for use.

[0015] "Joint purchase" refers to the act of multiple users within a region or community cooperating to purchase goods collectively.

[0016] "Delivery efficiency" is a measure indicating how quickly and with the least resources (such as time, fuel, etc.) an item can be delivered to its destination.

Brief Description of the Drawings

[0017] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which multiple emotions are mapped. [Figure 10] It shows an emotion map to which multiple emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0018] Hereinafter, an example of an embodiment of the system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0021] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0022] In the following embodiments, a numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0023] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0025] [First Embodiment]

[0026] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0027] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0030] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0033] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0037] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0038] One embodiment of this invention is a consumables management system based on voice input, which allows users to easily manage consumables using their voice. This system integrates and provides functions such as voice recognition, inventory management, ordering process, expiration date management, and regional group purchasing.

[0039] The device is equipped with a speech recognition function that converts voice input received from the user into text data. For example, if a user says to the device, "There's not enough milk," the device recognizes the voice as text and sends it to the server.

[0040] The server analyzes the received text data and identifies the corresponding consumable (in this case, milk). The server then accesses the inventory management database to check the inventory status of the identified consumable.

[0041] The server stores the user's purchasing criteria (for example, wanting to buy the cheapest product) and selects the optimal product based on inventory and price information. The server then automatically orders the selected product, allowing the user to live without worrying about stock shortages.

[0042] The server also integrates with online shopping sites and digital receipts to obtain expiration date information. For example, when the expiration date of fresh food purchased by a user approaches, the terminal notifies the user with an alert. This ensures that users can always use food in its freshest state without wasting any.

[0043] Furthermore, this system has a function to support joint purchases among users within the same region. The server analyzes the purchasing trends of users in the region and can improve delivery efficiency by coordinating bulk purchases and delivery dates.

[0044] In this way, the present invention streamlines the user's life and realizes economical and sustainable consumable management.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] The user issues a voice command regarding consumables to the terminal. The terminal receives the voice and converts the voice data into text data using its built-in voice recognition function.

[0048] Step 2:

[0049] The terminal sends the converted text data to the server.

[0050] Step 3:

[0051] The server analyzes the received text data and identifies the corresponding consumables. The server communicates with the inventory management database to check the latest inventory status of the identified consumables.

[0052] Step 4:

[0053] The server checks the user's pre-configured purchasing criteria (e.g., cheapest product, product from a specific manufacturer). Based on these criteria, the server searches for inventory and price information and selects the most suitable product.

[0054] Step 5:

[0055] The server initiates the automated ordering process, placing orders for selected items with online retailers. The purchase is then completed using the user's chosen payment method.

[0056] Step 6:

[0057] The server obtains expiration date information from online shopping sites and digital receipts. The server monitors the obtained expiration date information and notifies the user when the expiration date is approaching.

[0058] Step 7:

[0059] The server analyzes the purchasing trends of users within the region. It promotes opportunities for group buying by suggesting bulk purchases and delivery date adjustments to users.

[0060] Step 8:

[0061] When the device receives notifications or suggestions from the server, it presents that information to the user through the user interface. The user can respond to these notifications via the device.

[0062] (Example 1)

[0063] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0064] Traditional consumables management systems lacked efficiency due to limited user interaction using voice input and manual inventory management and ordering processes. Furthermore, insufficient features for expiration date management and group purchasing led to problems such as food waste and increased delivery costs. Additionally, the lack of visibility into purchase history and efficient logistics coordination meant users had limited means to effectively utilize the information.

[0065] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0066] In this invention, the server includes means for converting voice signals into text format, means for performing natural language processing to identify consumables, and means for automatically ordering selected products. This enables efficient inventory management and flexible expiration date management through voice input. Furthermore, it is possible to improve the user experience and contribute to reducing logistics costs through joint purchasing suggestions based on purchase data analysis and visualization of purchase history.

[0067] "Voice input" refers to the process of acquiring instructions and information from the user in the form of voice.

[0068] "Audio signal" is one of the formats used to process sound as digital data.

[0069] "Text format" refers to a format in which audio or other data has been converted into text data.

[0070] "Information processing means" refers to devices and programs that analyze digital data and perform corresponding functions.

[0071] "Natural language processing" refers to the technology of analyzing text data and extracting meaning.

[0072] "Consumable goods" refer to items used in daily life that decrease or are consumed through use.

[0073] "Automatic ordering methods" refer to the function of a system that orders necessary products online or by other means based on user instructions or conditions.

[0074] "Expiration date information" refers to data regarding the period during which a particular item can be used while maintaining its safety and quality.

[0075] "Group purchasing" refers to a system in which multiple users jointly purchase goods or services, thereby reducing costs and improving efficiency.

[0076] A "user interface" refers to the means or screens through which a system and a user interact when communicating information.

[0077] "Logistical efficiency" refers to the effective operation of goods delivery and distribution while minimizing time and costs.

[0078] This invention is a system for advanced consumable management that utilizes voice input, and helps users manage their daily necessities efficiently and automatically.

[0079] System Configuration

[0080] 1. Voice input and conversion

[0081] The device is equipped with a microphone to acquire voice input from the user. Speech recognition software (e.g., a typical cloud-based speech recognition service) is used to convert the voice signal into text format.

[0082] 2. Data analysis and consumable identification

[0083] The server analyzes the received text data using natural language processing. This analysis identifies consumable items (e.g., milk). A general-purpose AI model is used for natural language processing.

[0084] 3. Inventory management and automated ordering

[0085] The server accesses the database system to retrieve inventory information for consumables. If a shortage is detected, it automatically places an order for the most suitable product based on the user's purchase conditions. By utilizing the online shopping API, efficient product selection and ordering are possible.

[0086] 4. Expiration date management and notification

[0087] The server retrieves expiration date information for purchased products from online resources and notifies the user via their device when the expiration date is approaching.

[0088] 5. Optimizing group buying

[0089] The server analyzes the purchase history of multiple users and proposes cost-effective group purchases. It also improves logistics efficiency by coordinating deliveries among users within the same region.

[0090] Specific example

[0091] For example, if a user says "We're running low on milk," this information is converted to text using speech recognition, and after the server analyzes and checks inventory, the optimal amount of milk is automatically ordered. This process allows users to go about their daily lives without worrying about running out of consumables.

[0092] Example of a prompt

[0093] "A system that allows you to instantly purchase additional seasonings you realize you're running low on while cooking at home. It analyzes voice commands like 'I'm running low on tomato sauce,' selects the most suitable product, and automatically places an order."

[0094] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0095] Step 1:

[0096] The user provides voice input through the device's microphone. For example, they might say, "There's not enough milk." This voice input becomes the program's first input.

[0097] The device uses speech recognition software to convert the received speech signal into text data. This conversion process transforms the user's speech into a machine-readable text format.

[0098] Step 2:

[0099] The terminal sends the converted text data to the server. The text data becomes the input to the server.

[0100] The server uses natural language processing on the received text data to analyze keywords such as "few." This identifies a specific consumable item (in this case, "milk") and extracts the information necessary for the next processing step.

[0101] Step 3:

[0102] The server accesses the inventory management database to check the inventory of consumables (milk) identified by the text analysis results. This database query becomes the server input, and inventory information is output.

[0103] If the server has low or insufficient inventory, it uses a generative AI model to select the optimal order candidates based on the user's purchasing criteria. This results in a product list that matches the purchasing criteria.

[0104] Step 4:

[0105] The server uses an online shopping API to automatically place an order for the selected items. This API call is the next input to the server, and the order completion confirmation is the output.

[0106] This allows users to go about their daily lives with peace of mind, knowing that a new milk order has been automatically placed.

[0107] Step 5:

[0108] The server collects expiration date data and manages it in conjunction with digital receipts and other digital resources. When the expiration date is approaching, an alert is generated.

[0109] The device displays or sounds a warning to the user based on notifications from the server. This allows users to be more mindful of products nearing their expiration date and use them efficiently.

[0110] Step 6:

[0111] The server analyzes data from other users in the region to identify opportunities for group buying. The analysis of relevant data becomes the server's input, and proposed group buying plans are output.

[0112] Users can receive information about group purchases and, if they wish, participate to reduce costs.

[0113] (Application Example 1)

[0114] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0115] Modern households use a large number of consumables, and managing their inventory, ordering, and expiration dates presents challenges. Traditional methods often result in shortages or surpluses of consumables, making economical and efficient management difficult. Furthermore, efficient purchasing through cooperation with local residents is not adequately implemented. Therefore, it is necessary to address these issues, reduce resource waste within households, and improve the lives of users.

[0116] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0117] In this invention, the server includes information processing means for receiving voice input and converting voice data into text data; information processing means for analyzing the converted text data and confirming the inventory status of the corresponding materials; information processing means for selecting the optimal materials based on user settings and placing automatic orders; information processing means for acquiring expiration date information and notifying the user that the expiration date is approaching; information processing means for analyzing the purchase data of multiple users in the area and proposing cooperative purchases; and information processing means having a function to interact with the user via voice in a home automated device and notify them of inventory information. This makes inventory management of consumables more efficient, enables proper management of expiration dates, and allows for cooperative purchases with local residents.

[0118] "Voice input" refers to the operation of receiving voice signals from the user, which serve as the basic data for information processing.

[0119] "Information processing means" refers to devices and algorithms for converting audio data into text data, or for analyzing text data and processing the identified information.

[0120] "Inventory status" refers to information indicating the current quantity and condition of specific materials or goods.

[0121] "Materials" is a general term for consumables and necessities used in homes and organizations.

[0122] "Expiration date information" refers to information about the period during which a particular item can be used, and indicates the shelf life of food and consumables.

[0123] "Group purchasing" is a method in which multiple users cooperate to purchase goods in bulk, thereby increasing cost efficiency.

[0124] "Household automated devices" refer to machines or electrical devices used in the home that are designed to perform certain tasks automatically.

[0125] "Dialogue through voice" refers to a form of communication in which machines and humans exchange information using voice, employing speech synthesis and recognition technologies.

[0126] "User settings" refer to information that each user individually sets, serving as guidelines for operations and actions based on their preferences and conditions.

[0127] To implement this invention, first, a household automated device receives voice input from the user using a high-performance microphone. The voice input is converted into text data using speech recognition software. This converted text data is transmitted to a server via the internet. The server analyzes this text data and uses information processing means to check the inventory status of the corresponding materials.

[0128] After checking inventory status, the server selects the most suitable materials based on user settings, for example, prioritizing the cheapest items. The selected materials are automatically ordered, and the user is notified as needed. At this stage, the ordering process is simplified by utilizing the online shopping platform's API.

[0129] Furthermore, the server retrieves expiration date information for purchased materials and notifies users through automated home devices. This allows users to use materials efficiently and reduce waste. In addition, by analyzing purchase data from multiple users in the region, joint purchasing suggestions are made, resulting in cost reductions.

[0130] As a concrete example, if a home-use automated dispenser receives a voice input from a user saying "there's not enough milk," the server checks the inventory status and automatically orders the best option from nearby supermarkets or online stores. This process utilizes tools such as speech recognition APIs like Google® Cloud Speech-to-Text, database management systems like MySQL®, and Rakuten Market's API.

[0131] When providing this information to the user, a generative AI model can be used based on the following example prompt:

[0132] "Generate a scenario where a robotic consumables manager detects a milk shortage and selects and orders the most economical option."

[0133] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0134] Step 1:

[0135] The terminal receives voice input from the user. In this step, the home automated device collects the user's voice using a high-performance microphone and stores it as voice data. The input is a voice signal, and the output is digitized voice data.

[0136] Step 2:

[0137] The device uses a speech recognition API to convert audio data into text data. It analyzes the audio data using a speech recognition service such as Google Cloud Speech-to-Text and outputs it as text data. The input is audio data, and the output is text data.

[0138] Step 3:

[0139] The server receives text data and begins analysis. Based on the text data, it identifies the corresponding materials and prepares to check their inventory status. The input is text data, and the output is information about the items as a result of the analysis.

[0140] Step 4:

[0141] The server accesses the inventory management database to check the inventory status of the identified material. Here, the server uses an SQL query to retrieve the quantity information of the relevant material. The input is item information, and the output is the inventory status.

[0142] Step 5:

[0143] The server refers to user settings and determines the optimal ordering conditions. The server selects the most suitable materials based on price, past purchase history, and desired conditions. Inputs are inventory status and user settings, and output is a list of items to be ordered.

[0144] Step 6:

[0145] The server uses the API of the online shopping platform to execute automated orders. The server processes the order based on the selected items and confirms the transaction. The input is an item list, and the output is an order completion notification.

[0146] Step 7:

[0147] The server retrieves expiration date information and generates an alert regarding the expiration date. The server references digital receipts and online information and sends a notification to the user's device. The input is expiration date data, and the output is an alert message.

[0148] Step 8:

[0149] The server analyzes purchase data from multiple users within a region and proposes group purchases. It uses statistical methods to create an efficient collaborative purchase plan and notifies users. The input is purchase data, and the output is the proposed plan.

[0150] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0151] One embodiment of this invention is a consumables management system based on voice input that incorporates an emotion engine for analyzing the user's emotions. This system includes voice recognition, inventory management, automatic ordering, expiration date management, and a regional group purchasing function, as well as a function to optimize purchase suggestions that take into account the user's emotional state.

[0152] First, the device acquires voice input from the user and converts it into text data using its speech recognition function. At this time, the emotion engine analyzes the characteristics of the voice data and estimates various emotional states (e.g., satisfaction, anxiety, excitement, etc.). For example, if the user says, "I'm happy that the milk is all gone," the emotion engine analyzes that the user is in a positive emotional state.

[0153] The server identifies consumables based on the converted text data and checks their inventory status. It can then adjust purchase suggestions to best suit the user, taking into account the results of sentiment analysis. For example, if the user shows positive emotions, the server can suggest slightly more luxurious or new products in addition to their usual purchasing patterns.

[0154] Furthermore, the timing and importance of notifications can be adjusted according to the user's emotions, as recognized by the emotion engine. For example, if a user is feeling stressed, the server will notify them early about products nearing their expiration date to help alleviate their burden.

[0155] This system can analyze purchasing trends within a region and, taking emotions into account, can also suggest group purchases. For example, if many users have positive feelings when purchasing a particular product, the system aims to increase overall user satisfaction by suggesting that product as a target for group purchases.

[0156] In this way, by combining voice input and emotion analysis, the present invention improves the user experience and enables more efficient and emotionally responsive management of consumables.

[0157] The following describes the processing flow.

[0158] Step 1:

[0159] The user speaks a voice command regarding consumables to the terminal. The terminal receives this voice using a speech recognition module and processes it as voice data.

[0160] Step 2:

[0161] The device converts the acquired audio data into text data and simultaneously performs emotional analysis of the audio data using an emotion engine. This allows it to estimate the user's emotional state (e.g., satisfaction, stress, anxiety, etc.).

[0162] Step 3:

[0163] The terminal sends the converted text data and sentiment analysis results to the server.

[0164] Step 4:

[0165] The server analyzes the received text data and identifies the relevant consumables. The server then accesses the inventory management database to check the inventory status of the relevant consumables.

[0166] Step 5:

[0167] The server adjusts purchase suggestions based on the user's purchasing conditions and sentiment analysis results. For example, if the user is feeling stressed, it will suggest simple, relaxing consumables.

[0168] Step 6:

[0169] The server initiates an automated ordering process based on the selected products and completes the order with the online retailer using the user's preferred payment method.

[0170] Step 7:

[0171] The server retrieves product expiration date information from an online platform and adjusts the timing and content of notifications for products nearing their expiration date, taking into account the user's emotional state.

[0172] Step 8:

[0173] The server analyzes purchase data from other users in the region and suggests group purchases, taking their emotional state into consideration. If a user's emotional state is positive, it collaborates with other users with similar emotional states to encourage purchases.

[0174] Step 9:

[0175] The device receives notifications and suggestions from the server and presents them to the user in an easy-to-understand manner. The user makes decisions based on the presented information and, if necessary, engages in purchasing activities.

[0176] (Example 2)

[0177] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0178] In modern consumer goods management, challenges include the difficulty of responding flexibly to users' emotional states and the fact that inventory management and purchase recommendations do not take individual user feelings into consideration. Therefore, a more personalized approach is needed to improve user satisfaction and the purchasing experience. Furthermore, improving local group purchasing and delivery efficiency are also challenges.

[0179] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0180] In this invention, the server includes means for estimating the user's emotional state based on voice input, means for selecting the optimal consumables and making purchase suggestions based on the emotional state and user settings, and means for acquiring expiration date information and adjusting the timing of notifications according to the user's emotions. This enables efficient and personalized consumable management and purchase suggestions while taking the user's emotions into consideration.

[0181] "Voice input" is the process of acquiring acoustic signals from a user and converting that information into a data format that a machine can understand.

[0182] "Text data" refers to digital character data converted from speech, which holds information in a format that is easy for computers to process.

[0183] "Consumables" are items that require periodic replenishment to maintain their condition through use or consumption.

[0184] "Emotional state" refers to a state in which the user's emotional movements and feelings are analyzed and identified as a specific emotion, such as positive or negative.

[0185] A "purchase suggestion" is a guide or recommendation to provide appropriate products or services based on the user's interests and needs.

[0186] "Expiration date" refers to the date by which a particular item can maintain its quality without deterioration; after this date, there are concerns about the deterioration of its safety and quality.

[0187] "Local users" refers to a group of users who reside or work in a specific geographical area, and whose purchasing activities and trends are the subject of analysis.

[0188] "Group buying" is an activity in which multiple users cooperate to purchase the same or similar products together, aiming to reduce costs and improve purchasing conditions.

[0189] This invention is a consumables management system that combines voice input and emotion analysis, enabling efficient inventory management while improving the user experience. Specific embodiments are described below.

[0190] First, the device acquires the user's voice using an input device (e.g., a smart speaker or smartphone). This voice data is then converted into text data using speech recognition software (e.g., Google Cloud Speech-to-Text). The speech recognition process digitizes the acoustic signal and analyzes its content as text.

[0191] Next, the server uses the converted text data to identify consumables and check their inventory status. Here, database technology is used to manage product information and check inventory quantities and expiration dates. Furthermore, a sentiment analysis engine (e.g., IBM Watson® Tone Analyzer) is used to estimate the user's emotional state from the text, identifying positive or negative feelings.

[0192] The server optimizes consumable purchase suggestions by considering the user's emotional state. This utilizes an AI model to suggest products tailored to each individual user, based on their past purchase history and inventory information. Furthermore, it flexibly adjusts notification timing based on the user's emotions and inventory status, providing appropriate notifications to reduce stress for products nearing their expiration date.

[0193] Furthermore, the server analyzes purchasing data from multiple users within a region and proposes group purchases for specific consumables. For products that receive positive feedback from many users, the system aims to improve overall purchasing efficiency and satisfaction by offering group purchase options.

[0194] As a concrete example, a user might voice-input, "I want to buy a lot of bottled water." Based on this information, the server checks the inventory status and sentiment analysis results, and then suggests purchasing a large-capacity pack. At this point, by confirming that there is sufficient stock, it suggests immediate shipping, thus responding to the user's needs.

[0195] As an example of how the generative AI model can be used, we will show a prompt message. Using a prompt message such as, "Based on the user's purchase history and emotional state, identify appropriate consumables and create individually customized suggestions," the system will provide highly accurate suggestions.

[0196] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0197] Step 1:

[0198] The device acquires voice input from the user. This input is collected as an acoustic signal using the microphone of a smart speaker or smartphone. Next, this acoustic signal is converted into text data by speech recognition software. In this conversion process, the received audio is processed into a digital format and converted into text information by a speech recognition engine (e.g., Google Cloud Speech-to-Text). The output of this step is the text data converted from the audio.

[0199] Step 2:

[0200] The server receives text data sent from the terminal and identifies consumables and checks their inventory status. The input is text data, and the system uses database technology to check the inventory quantity and expiration date of the identified consumables. Database operations such as SQL are performed to search for data corresponding to the product information. The output of this step is inventory information for the identified consumables.

[0201] Step 3:

[0202] The server inputs the text data from the previous step into the sentiment analysis engine to estimate the user's emotional state. Here, natural language processing techniques (e.g., IBM Watson Tone Analyzer) are used to analyze emotional elements from the text. Specifically, an algorithm is applied that identifies emotions from keywords and context in the text. The output of this step is the estimated emotional state of the user.

[0203] Step 4:

[0204] The server makes optimal consumable purchase suggestions based on inventory information and the user's emotional state. The inputs here are inventory information and emotional state, which are used to generate product suggestions tailored to the user's preferences using an AI model. This process analyzes existing purchase history and emotional analysis results, inputting prompts into the AI ​​to generate appropriate suggestions. The output of this step is a personalized purchase suggestion for the user.

[0205] Step 5:

[0206] The server adjusts notification timing based on the user's emotional state. It uses emotional state and expiration date information as input to determine an appropriate notification schedule. Based on this identified schedule, it sends an alert in advance if the expiration date is approaching. The output of this step is a timely notification to the user.

[0207] Step 6:

[0208] The server evaluates purchase data and sentiment analysis results collected from users within a region to formulate group buying suggestions. The input is data from multiple users, which is analyzed to identify products that consistently receive high ratings. Based on this, the server provides users with group buying options. The output of this step is a group buying suggestion for all users.

[0209] (Application Example 2)

[0210] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0211] Conventional consumables management systems and electronic payment services often fail to consider the user's emotional state when making purchase suggestions, resulting in an inadequate user experience. Furthermore, automated ordering and expiration date management lack sufficient consideration for reducing the psychological burden on users. Therefore, there is a need for systems that provide appropriate product suggestions and notifications tailored to the user's current emotional state.

[0212] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0213] In this invention, the server includes means for receiving voice input and converting voice data into text data; means for analyzing the converted text data and checking the inventory status of the corresponding items; means for selecting the optimal items based on user settings and placing automatic orders; means for acquiring expiration date information and notifying the user that the expiration date is approaching; means for analyzing purchase data of multiple users in a region and proposing joint purchases; means for analyzing the characteristics of voice data and estimating the user's emotional state; and means for optimizing suggestions based on the emotional state and notifying the user of the information. This enables purchase suggestions and consumable management that are tailored to the user's emotions.

[0214] "Voice input" is a method by which users provide information to a system through their voice.

[0215] "Audio data" refers to audio signals obtained from audio input that have been converted into digital data.

[0216] "Text data" refers to string information obtained by analyzing audio data.

[0217] "Information processing means" refers to a device or function that processes data according to a set procedure and utilizes the results obtained.

[0218] "Inventory status" refers to information indicating the current level of a particular item or resource.

[0219] "Automatic ordering" is a process that automatically places orders for goods based on pre-set conditions.

[0220] "Expiration date information" refers to data indicating the date by which a particular product's quality is guaranteed.

[0221] "Group purchasing" is a form of purchasing where multiple users cooperate to buy the same item, aiming to reduce costs and improve efficiency.

[0222] "Emotion analysis" is a technology that estimates a person's emotional state using voice and text data.

[0223] "Optimizing proposals" is the process of presenting the most suitable options and information based on the user's needs and circumstances.

[0224] The system for implementing this invention mainly uses a server, a user terminal, speech recognition software, and an emotion analysis engine. The operation of each component is described below.

[0225] First, the user provides voice input to the user terminal. This terminal uses speech recognition software to convert the voice data into text data. Widely used software, such as the SpeechRecognition library, can be used for speech recognition. Once the text data is generated, the server receives it and checks the inventory status of the corresponding item. In the backend, database queries are used to connect with the inventory management system.

[0226] Furthermore, this server uses an emotion analysis engine to estimate the user's emotional state from the obtained voice data. Existing emotion analysis libraries and generative AI models can be used for this emotion analysis. For example, if a positive emotion is detected, the server will suggest a slightly luxurious item to the user. This suggestion is optimized by taking into account past purchase history and purchasing patterns.

[0227] When a user selects a suggested item, the server automatically places an order. This ordering process automates the purchase procedure based on the user's existing settings. Simultaneously, the server has a function to notify the user when an item is nearing its expiration date, based on expiration date information. This is an important feature to reduce the user's burden, and expiration date information is collected and managed via an API.

[0228] For example, if a user voice-inputs, "I'm in a good mood today, so I'd like to enjoy a special wine with dinner," the system will check the wine's inventory and suggest a slightly more elegant option than usual, along with a positive sentiment. Furthermore, based on the user's past preference patterns, it will determine if there are any limited-time offers or promotions available.

[0229] An example of a prompt for a generative AI model would be a question like, "Design logic to suggest luxury items when the user expresses an emotion they are enjoying." This prompt contributes to improving emotion-based purchase suggestion logic.

[0230] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0231] Step 1:

[0232] The user provides shopping requests to the terminal via voice input. The terminal uses voice recognition software to convert the voice data into text data. In this process, a voice recognition library analyzes the voice signal and generates corresponding string data.

[0233] Step 2:

[0234] The terminal sends the converted text data to the server. The server receives this text data and executes a database query to check the inventory status of the items. Here, the server takes the text data as input, retrieves the corresponding product information from the database, and outputs whether or not the item is in stock.

[0235] Step 3:

[0236] The server uses a generative AI model to analyze the user's emotional state from text data. The emotion analysis engine takes text data as input, analyzes features extracted from the text, and outputs an estimated result of the emotional state. For example, it can determine whether the user's statements are positive.

[0237] Step 4:

[0238] The server generates optimal product recommendations using the results of sentiment analysis. It selects suggested items considering the user's emotional state and past purchase history. The inputs used are emotional state and purchase history, and the server outputs a list of recommended products, including luxury goods and new products.

[0239] Step 5:

[0240] The user selects the suggested items. The server then places an automatic order based on this selection. In this step, the ordering process proceeds using the selected item information as input, and an order confirmation is output.

[0241] Step 6:

[0242] The server retrieves expiration date information and notifies the user. Here, it retrieves information about items nearing their expiration date from a database, uses that information as input, and outputs an expiration date notification. This includes specific actions such as notifying the user via email or in-app notification.

[0243] Step 7:

[0244] The server analyzes purchasing trends within the region and proposes group purchases to other users. This process takes purchase data as input and outputs suggestions for cost reductions and benefits through group purchasing. This is where actions that encourage collaborative behavior among users take place.

[0245] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0246] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0247] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0248] [Second Embodiment]

[0249] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0250] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0251] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0252] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0253] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0254] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0255] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0256] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0257] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0258] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0259] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0260] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0261] One embodiment of this invention is a consumables management system based on voice input, which allows users to easily manage consumables using their voice. This system integrates and provides functions such as voice recognition, inventory management, ordering process, expiration date management, and regional group purchasing.

[0262] The device is equipped with a speech recognition function that converts voice input received from the user into text data. For example, if a user says to the device, "There's not enough milk," the device recognizes the voice as text and sends it to the server.

[0263] The server analyzes the received text data and identifies the corresponding consumable (in this case, milk). The server then accesses the inventory management database to check the inventory status of the identified consumable.

[0264] The server stores the user's purchasing criteria (for example, wanting to buy the cheapest product) and selects the optimal product based on inventory and price information. The server then automatically orders the selected product, allowing the user to live without worrying about stock shortages.

[0265] The server also integrates with online shopping sites and digital receipts to obtain expiration date information. For example, when the expiration date of fresh food purchased by a user approaches, the terminal notifies the user with an alert. This ensures that users can always use food in its freshest state without wasting any.

[0266] Furthermore, this system has a function to support joint purchases among users within the same region. The server analyzes the purchasing trends of users in the region and can improve delivery efficiency by coordinating bulk purchases and delivery dates.

[0267] In this way, the present invention streamlines the user's life and realizes economical and sustainable consumable management.

[0268] The following describes the processing flow.

[0269] Step 1:

[0270] The user issues a voice command regarding consumables to the terminal. The terminal receives the voice and converts the voice data into text data using its built-in voice recognition function.

[0271] Step 2:

[0272] The terminal sends the converted text data to the server.

[0273] Step 3:

[0274] The server analyzes the received text data and identifies the corresponding consumables. The server communicates with the inventory management database to check the latest inventory status of the identified consumables.

[0275] Step 4:

[0276] The server checks the purchase conditions (e.g., lowest-price items, products of a specific manufacturer) set by the user in advance. Based on these conditions, the server searches for inventory and price information and selects the optimal products.

[0277] Step 5:

[0278] The server initiates the automatic ordering process and places an order for the selected products with the online retailer. The purchase is completed using the payment method set by the user.

[0279] Step 6:

[0280] The server obtains expiration date information through an online shopping site or a digital receipt. The server monitors the obtained expiration date information and notifies the user when the expiration date approaches.

[0281] Step 7:

[0282] The server analyzes the purchase trends of users within the region. By proposing bulk purchases and delivery date adjustments to users, it promotes opportunities for joint purchases.

[0283] Step 8:

[0284] When the terminal receives notifications and proposals from the server, it presents this information to the user through the user interface. The user can respond to these notifications via the terminal.

[0285] (Example 1)

[0286] Next, Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0287] Conventional consumable management systems have limited user interaction using voice input, and inventory management and product ordering processes were manual, lacking efficiency. Also, functions such as expiration date management and joint purchasing were insufficient, leading to problems such as increased food loss and delivery costs. Furthermore, since purchase history visualization and efficient logistics adjustment were not possible, users lacked means to effectively utilize information.

[0288] The specific processing by the specific processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0289] In this invention, the server includes means for converting a voice signal into text format, means for performing natural language processing to identify consumables, and means for automatically ordering selected products. This enables efficient inventory management and flexible expiration date management through voice input. Also, it is possible to improve the user experience and contribute to reducing logistics costs through proposals for joint purchasing and visualization of purchase history based on purchase data analysis.

[0290] "Voice input" refers to the process of acquiring instructions and information from the user as voice.

[0291] "Voice signal" is one form of processing voice as digital data.

[0292] "Text format" refers to the format in which voice or other data is converted into character data.

[0293] "Information processing means" refers to a device or program for analyzing digital data and executing corresponding functions.

[0294] "Natural language processing" refers to the technology of analyzing text data and extracting meaning.

[0295] "Consumable goods" refer to items used in daily life that decrease or are consumed through use.

[0296] "Automatic ordering methods" refer to the function of a system that orders necessary products online or by other means based on user instructions or conditions.

[0297] "Expiration date information" refers to data regarding the period during which a particular item can be used while maintaining its safety and quality.

[0298] "Group purchasing" refers to a system in which multiple users jointly purchase goods or services, thereby reducing costs and improving efficiency.

[0299] A "user interface" refers to the means or screens through which a system and a user interact when communicating information.

[0300] "Logistical efficiency" refers to the effective operation of goods delivery and distribution while minimizing time and costs.

[0301] This invention is a system for advanced consumable management that utilizes voice input, and helps users manage their daily necessities efficiently and automatically.

[0302] System Configuration

[0303] 1. Voice input and conversion

[0304] The device is equipped with a microphone to acquire voice input from the user. Speech recognition software (e.g., a typical cloud-based speech recognition service) is used to convert the voice signal into text format.

[0305] 2. Data analysis and consumable identification

[0306] The server analyzes the received text data using natural language processing. Through this analysis, consumables (such as milk) are identified. General AI models are used for natural language processing.

[0307] 3. Inventory Management and Automatic Ordering

[0308] The server accesses the database system to obtain inventory information of consumables. When a shortage is confirmed, automatic ordering of the optimal products is performed based on the user's purchase conditions. By leveraging the online shopping API, efficient product selection and ordering are possible.

[0309] 4. Expiration Date Management and Notification

[0310] The server obtains the expiration date information of purchased products from online resources and notifies the user through the terminal when the expiration date is approaching.

[0311] 5. Optimization of Group Purchases

[0312] The server analyzes the purchase histories of multiple users and proposes cost-effective group purchases. By adjusting the delivery among users within the region, logistics efficiency is improved.

[0313] Specific Example

[0314] For example, when the user says "There is little milk", this information is converted into text by voice recognition. After the server analyzes and checks the inventory, the optimal milk is automatically ordered. Through this process, the user can live their daily life without worrying about shortages of consumables.

[0315] Example of Prompt Sentence

[0316] "A system that can immediately re-purchase seasonings that are lacking during cooking at home. Analyze the voice instruction 'There is little tomato sauce' and select the optimal product for automatic ordering."

[0317] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0318] Step 1:

[0319] The user provides voice input through the device's microphone. For example, they might say, "There's not enough milk." This voice input becomes the program's first input.

[0320] The device uses speech recognition software to convert the received speech signal into text data. This conversion process transforms the user's speech into a machine-readable text format.

[0321] Step 2:

[0322] The terminal sends the converted text data to the server. The text data becomes the input to the server.

[0323] The server uses natural language processing on the received text data to analyze keywords such as "few." This identifies a specific consumable item (in this case, "milk") and extracts the information necessary for the next processing step.

[0324] Step 3:

[0325] The server accesses the inventory management database to check the inventory of consumables (milk) identified by the text analysis results. This database query becomes the server input, and inventory information is output.

[0326] If the server has low or insufficient inventory, it uses a generative AI model to select the optimal order candidates based on the user's purchasing criteria. This results in a product list that matches the purchasing criteria.

[0327] Step 4:

[0328] The server uses an online shopping API to automatically place an order for the selected items. This API call is the next input to the server, and the order completion confirmation is the output.

[0329] This allows users to go about their daily lives with peace of mind, knowing that a new milk order has been automatically placed.

[0330] Step 5:

[0331] The server collects expiration date data and manages it in conjunction with digital receipts and other digital resources. When the expiration date is approaching, an alert is generated.

[0332] The device displays or sounds a warning to the user based on notifications from the server. This allows users to be more mindful of products nearing their expiration date and use them efficiently.

[0333] Step 6:

[0334] The server analyzes data from other users in the region to identify opportunities for group buying. The analysis of relevant data becomes the server's input, and proposed group buying plans are output.

[0335] Users can receive information about group purchases and, if they wish, participate to reduce costs.

[0336] (Application Example 1)

[0337] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0338] Modern households use a large number of consumables, and managing their inventory, ordering, and expiration dates presents challenges. Traditional methods often result in shortages or surpluses of consumables, making economical and efficient management difficult. Furthermore, efficient purchasing through cooperation with local residents is not adequately implemented. Therefore, it is necessary to address these issues, reduce resource waste within households, and improve the lives of users.

[0339] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0340] In this invention, the server includes information processing means for receiving voice input and converting voice data into text data; information processing means for analyzing the converted text data and confirming the inventory status of the corresponding materials; information processing means for selecting the optimal materials based on user settings and placing automatic orders; information processing means for acquiring expiration date information and notifying the user that the expiration date is approaching; information processing means for analyzing the purchase data of multiple users in the area and proposing cooperative purchases; and information processing means having a function to interact with the user via voice in a home automated device and notify them of inventory information. This makes inventory management of consumables more efficient, enables proper management of expiration dates, and allows for cooperative purchases with local residents.

[0341] "Voice input" refers to the operation of receiving voice signals from the user, which serve as the basic data for information processing.

[0342] "Information processing means" refers to devices and algorithms for converting audio data into text data, or for analyzing text data and processing the identified information.

[0343] "Inventory status" refers to information indicating the current quantity and condition of specific materials or goods.

[0344] "Materials" is a general term for consumables and necessities used in homes and organizations.

[0345] "Expiration date information" refers to information about the period during which a particular item can be used, and indicates the shelf life of food and consumables.

[0346] "Group purchasing" is a method in which multiple users cooperate to purchase goods in bulk, thereby increasing cost efficiency.

[0347] "Household automated devices" refer to machines or electrical devices used in the home that are designed to perform certain tasks automatically.

[0348] "Dialogue through voice" refers to a form of communication in which machines and humans exchange information using voice, employing speech synthesis and recognition technologies.

[0349] "User settings" refer to information that each user individually sets, serving as guidelines for operations and actions based on their preferences and conditions.

[0350] To implement this invention, first, a household automated device receives voice input from the user using a high-performance microphone. The voice input is converted into text data using speech recognition software. This converted text data is transmitted to a server via the internet. The server analyzes this text data and uses information processing means to check the inventory status of the corresponding materials.

[0351] After checking inventory status, the server selects the most suitable materials based on user settings, for example, prioritizing the cheapest items. The selected materials are automatically ordered, and the user is notified as needed. At this stage, the ordering process is simplified by utilizing the online shopping platform's API.

[0352] Furthermore, the server retrieves expiration date information for purchased materials and notifies users through automated home devices. This allows users to use materials efficiently and reduce waste. In addition, by analyzing purchase data from multiple users in the region, joint purchasing suggestions are made, resulting in cost reductions.

[0353] As a concrete example, if a home automated system receives voice input from a user saying "there's not enough milk," the server checks the inventory status and automatically orders the best option from nearby supermarkets or online stores. This process utilizes tools such as speech recognition APIs like Google Cloud Speech-to-Text, database management systems like MySQL, and the Rakuten Market API.

[0354] When providing this information to the user, a generative AI model can be used based on the following example prompt:

[0355] "Generate a scenario where a robotic consumables manager detects a milk shortage and selects and orders the most economical option."

[0356] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0357] Step 1:

[0358] The terminal receives voice input from the user. In this step, the home automated device collects the user's voice using a high-performance microphone and stores it as voice data. The input is a voice signal, and the output is digitized voice data.

[0359] Step 2:

[0360] The device uses a speech recognition API to convert audio data into text data. It analyzes the audio data using a speech recognition service such as Google Cloud Speech-to-Text and outputs it as text data. The input is audio data, and the output is text data.

[0361] Step 3:

[0362] The server receives text data and begins analysis. Based on the text data, it identifies the corresponding materials and prepares to check their inventory status. The input is text data, and the output is information about the items as a result of the analysis.

[0363] Step 4:

[0364] The server accesses the inventory management database to check the inventory status of the identified material. Here, the server uses an SQL query to retrieve the quantity information of the relevant material. The input is item information, and the output is the inventory status.

[0365] Step 5:

[0366] The server refers to user settings and determines the optimal ordering conditions. The server selects the most suitable materials based on price, past purchase history, and desired conditions. Inputs are inventory status and user settings, and output is a list of items to be ordered.

[0367] Step 6:

[0368] The server uses the API of the online shopping platform to execute automated orders. The server processes the order based on the selected items and confirms the transaction. The input is an item list, and the output is an order completion notification.

[0369] Step 7:

[0370] The server retrieves expiration date information and generates an alert regarding the expiration date. The server references digital receipts and online information and sends a notification to the user's device. The input is expiration date data, and the output is an alert message.

[0371] Step 8:

[0372] The server analyzes purchase data from multiple users within a region and proposes group purchases. It uses statistical methods to create an efficient collaborative purchase plan and notifies users. The input is purchase data, and the output is the proposed plan.

[0373] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0374] One embodiment of this invention is a consumables management system based on voice input that incorporates an emotion engine for analyzing the user's emotions. This system includes voice recognition, inventory management, automatic ordering, expiration date management, and a regional group purchasing function, as well as a function to optimize purchase suggestions that take into account the user's emotional state.

[0375] First, the device acquires voice input from the user and converts it into text data using its speech recognition function. At this time, the emotion engine analyzes the characteristics of the voice data and estimates various emotional states (e.g., satisfaction, anxiety, excitement, etc.). For example, if the user says, "I'm happy that the milk is all gone," the emotion engine analyzes that the user is in a positive emotional state.

[0376] The server identifies consumables based on the converted text data and checks their inventory status. It can then adjust purchase suggestions to best suit the user, taking into account the results of sentiment analysis. For example, if the user shows positive emotions, the server can suggest slightly more luxurious or new products in addition to their usual purchasing patterns.

[0377] Furthermore, the timing and importance of notifications can be adjusted according to the user's emotions, as recognized by the emotion engine. For example, if a user is feeling stressed, the server will notify them early about products nearing their expiration date to help alleviate their burden.

[0378] This system can analyze purchasing trends within a region and, taking emotions into account, can also suggest group purchases. For example, if many users have positive feelings when purchasing a particular product, the system aims to increase overall user satisfaction by suggesting that product as a target for group purchases.

[0379] In this way, by combining voice input and emotion analysis, the present invention improves the user experience and enables more efficient and emotionally responsive management of consumables.

[0380] The following describes the processing flow.

[0381] Step 1:

[0382] The user speaks a voice command regarding consumables to the terminal. The terminal receives this voice using a speech recognition module and processes it as voice data.

[0383] Step 2:

[0384] The device converts the acquired audio data into text data and simultaneously performs emotional analysis of the audio data using an emotion engine. This allows it to estimate the user's emotional state (e.g., satisfaction, stress, anxiety, etc.).

[0385] Step 3:

[0386] The terminal sends the converted text data and sentiment analysis results to the server.

[0387] Step 4:

[0388] The server analyzes the received text data and identifies the relevant consumables. The server then accesses the inventory management database to check the inventory status of the relevant consumables.

[0389] Step 5:

[0390] The server adjusts purchase suggestions based on the user's purchasing conditions and sentiment analysis results. For example, if the user is feeling stressed, it will suggest simple, relaxing consumables.

[0391] Step 6:

[0392] The server initiates an automated ordering process based on the selected products and completes the order with the online retailer using the user's preferred payment method.

[0393] Step 7:

[0394] The server retrieves product expiration date information from an online platform and adjusts the timing and content of notifications for products nearing their expiration date, taking into account the user's emotional state.

[0395] Step 8:

[0396] The server analyzes purchase data from other users in the region and suggests group purchases, taking their emotional state into consideration. If a user's emotional state is positive, it collaborates with other users with similar emotional states to encourage purchases.

[0397] Step 9:

[0398] The device receives notifications and suggestions from the server and presents them to the user in an easy-to-understand manner. The user makes decisions based on the presented information and, if necessary, engages in purchasing activities.

[0399] (Example 2)

[0400] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0401] In modern consumer goods management, challenges include the difficulty of responding flexibly to users' emotional states and the fact that inventory management and purchase recommendations do not take individual user feelings into consideration. Therefore, a more personalized approach is needed to improve user satisfaction and the purchasing experience. Furthermore, improving local group purchasing and delivery efficiency are also challenges.

[0402] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0403] In this invention, the server includes means for estimating the user's emotional state based on voice input, means for selecting the optimal consumables and making purchase suggestions based on the emotional state and user settings, and means for acquiring expiration date information and adjusting the timing of notifications according to the user's emotions. This enables efficient and personalized consumable management and purchase suggestions while taking the user's emotions into consideration.

[0404] "Voice input" is the process of acquiring acoustic signals from a user and converting that information into a data format that a machine can understand.

[0405] "Text data" refers to digital character data converted from speech, which holds information in a format that is easy for computers to process.

[0406] "Consumables" are items that require periodic replenishment to maintain their condition through use or consumption.

[0407] "Emotional state" refers to a state in which the user's emotional movements and feelings are analyzed and identified as a specific emotion, such as positive or negative.

[0408] A "purchase suggestion" is a guide or recommendation to provide appropriate products or services based on the user's interests and needs.

[0409] "Expiration date" refers to the date by which a particular item can maintain its quality without deterioration; after this date, there are concerns about the deterioration of its safety and quality.

[0410] "Local users" refers to a group of users who reside or work in a specific geographical area, and whose purchasing activities and trends are the subject of analysis.

[0411] "Group buying" is an activity in which multiple users cooperate to purchase the same or similar products together, aiming to reduce costs and improve purchasing conditions.

[0412] This invention is a consumables management system that combines voice input and emotion analysis, enabling efficient inventory management while improving the user experience. Specific embodiments are described below.

[0413] First, the device acquires the user's voice using an input device (e.g., a smart speaker or smartphone). This voice data is then converted into text data using speech recognition software (e.g., Google Cloud Speech-to-Text). The speech recognition process digitizes the acoustic signal and analyzes its content as text.

[0414] Next, the server uses the converted text data to identify consumables and check their inventory status. Here, database technology is used to manage product information and check inventory quantities and expiration dates. Furthermore, a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) is used to estimate the user's emotional state from the text, identifying positive or negative feelings.

[0415] The server optimizes consumable purchase suggestions by considering the user's emotional state. This utilizes an AI model to suggest products tailored to each individual user, based on their past purchase history and inventory information. Furthermore, it flexibly adjusts notification timing based on the user's emotions and inventory status, providing appropriate notifications to reduce stress for products nearing their expiration date.

[0416] Furthermore, the server analyzes purchasing data from multiple users within a region and proposes group purchases for specific consumables. For products that receive positive feedback from many users, the system aims to improve overall purchasing efficiency and satisfaction by offering group purchase options.

[0417] As a concrete example, a user might voice-input, "I want to buy a lot of bottled water." Based on this information, the server checks the inventory status and sentiment analysis results, and then suggests purchasing a large-capacity pack. At this point, by confirming that there is sufficient stock, it suggests immediate shipping, thus responding to the user's needs.

[0418] As an example of how the generative AI model can be used, we will show a prompt message. Using a prompt message such as, "Based on the user's purchase history and emotional state, identify appropriate consumables and create individually customized suggestions," the system will provide highly accurate suggestions.

[0419] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0420] Step 1:

[0421] The device acquires voice input from the user. This input is collected as an acoustic signal using the microphone of a smart speaker or smartphone. Next, this acoustic signal is converted into text data by speech recognition software. In this conversion process, the received audio is processed into a digital format and converted into text information by a speech recognition engine (e.g., Google Cloud Speech-to-Text). The output of this step is the text data converted from the audio.

[0422] Step 2:

[0423] The server receives text data sent from the terminal and identifies consumables and checks their inventory status. The input is text data, and the system uses database technology to check the inventory quantity and expiration date of the identified consumables. Database operations such as SQL are performed to search for data corresponding to the product information. The output of this step is inventory information for the identified consumables.

[0424] Step 3:

[0425] The server inputs the text data from the previous step into the sentiment analysis engine to estimate the user's emotional state. Here, natural language processing techniques (e.g., IBM Watson Tone Analyzer) are used to analyze emotional elements from the text. Specifically, an algorithm is applied that identifies emotions from keywords and context in the text. The output of this step is the estimated emotional state of the user.

[0426] Step 4:

[0427] The server makes optimal consumable purchase suggestions based on inventory information and the user's emotional state. The inputs here are inventory information and emotional state, which are used to generate product suggestions tailored to the user's preferences using an AI model. This process analyzes existing purchase history and emotional analysis results, inputting prompts into the AI ​​to generate appropriate suggestions. The output of this step is a personalized purchase suggestion for the user.

[0428] Step 5:

[0429] The server adjusts notification timing based on the user's emotional state. It uses emotional state and expiration date information as input to determine an appropriate notification schedule. Based on this identified schedule, it sends an alert in advance if the expiration date is approaching. The output of this step is a timely notification to the user.

[0430] Step 6:

[0431] The server evaluates purchase data and sentiment analysis results collected from users within a region to formulate group buying suggestions. The input is data from multiple users, which is analyzed to identify products that consistently receive high ratings. Based on this, the server provides users with group buying options. The output of this step is a group buying suggestion for all users.

[0432] (Application Example 2)

[0433] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0434] Conventional consumables management systems and electronic payment services often fail to consider the user's emotional state when making purchase suggestions, resulting in an inadequate user experience. Furthermore, automated ordering and expiration date management lack sufficient consideration for reducing the psychological burden on users. Therefore, there is a need for systems that provide appropriate product suggestions and notifications tailored to the user's current emotional state.

[0435] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0436] In this invention, the server includes means for receiving voice input and converting voice data into text data; means for analyzing the converted text data and checking the inventory status of the corresponding items; means for selecting the optimal items based on user settings and placing automatic orders; means for acquiring expiration date information and notifying the user that the expiration date is approaching; means for analyzing purchase data of multiple users in a region and proposing joint purchases; means for analyzing the characteristics of voice data and estimating the user's emotional state; and means for optimizing suggestions based on the emotional state and notifying the user of the information. This enables purchase suggestions and consumable management that are tailored to the user's emotions.

[0437] "Voice input" is a method by which users provide information to a system through their voice.

[0438] "Audio data" refers to audio signals obtained from audio input that have been converted into digital data.

[0439] "Text data" refers to string information obtained by analyzing audio data.

[0440] "Information processing means" refers to a device or function that processes data according to a set procedure and utilizes the results obtained.

[0441] "Inventory status" refers to information indicating the current level of a particular item or resource.

[0442] "Automatic ordering" is a process that automatically places orders for goods based on pre-set conditions.

[0443] "Expiration date information" refers to data indicating the date by which a particular product's quality is guaranteed.

[0444] "Group purchasing" is a form of purchasing where multiple users cooperate to buy the same item, aiming to reduce costs and improve efficiency.

[0445] "Emotion analysis" is a technology that estimates a person's emotional state using voice and text data.

[0446] "Optimizing proposals" is the process of presenting the most suitable options and information based on the user's needs and circumstances.

[0447] The system for implementing this invention mainly uses a server, a user terminal, speech recognition software, and an emotion analysis engine. The operation of each component is described below.

[0448] First, the user provides voice input to the user terminal. This terminal uses speech recognition software to convert the voice data into text data. Widely used software, such as the SpeechRecognition library, can be used for speech recognition. Once the text data is generated, the server receives it and checks the inventory status of the corresponding item. In the backend, database queries are used to connect with the inventory management system.

[0449] Furthermore, this server uses an emotion analysis engine to estimate the user's emotional state from the obtained voice data. Existing emotion analysis libraries and generative AI models can be used for this emotion analysis. For example, if a positive emotion is detected, the server will suggest a slightly luxurious item to the user. This suggestion is optimized by taking into account past purchase history and purchasing patterns.

[0450] When a user selects a suggested item, the server automatically places an order. This ordering process automates the purchase procedure based on the user's existing settings. Simultaneously, the server has a function to notify the user when an item is nearing its expiration date, based on expiration date information. This is an important feature to reduce the user's burden, and expiration date information is collected and managed via an API.

[0451] For example, if a user voice-inputs, "I'm in a good mood today, so I'd like to enjoy a special wine with dinner," the system will check the wine's inventory and suggest a slightly more elegant option than usual, along with a positive sentiment. Furthermore, based on the user's past preference patterns, it will determine if there are any limited-time offers or promotions available.

[0452] An example of a prompt for a generative AI model would be a question like, "Design logic to suggest luxury items when the user expresses an emotion they are enjoying." This prompt contributes to improving emotion-based purchase suggestion logic.

[0453] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0454] Step 1:

[0455] The user provides shopping requests to the terminal via voice input. The terminal uses voice recognition software to convert the voice data into text data. In this process, a voice recognition library analyzes the voice signal and generates corresponding string data.

[0456] Step 2:

[0457] The terminal sends the converted text data to the server. The server receives this text data and executes a database query to check the inventory status of the items. Here, the server takes the text data as input, retrieves the corresponding product information from the database, and outputs whether or not the item is in stock.

[0458] Step 3:

[0459] The server uses a generative AI model to analyze the user's emotional state from text data. The emotion analysis engine takes text data as input, analyzes features extracted from the text, and outputs an estimated result of the emotional state. For example, it can determine whether the user's statements are positive.

[0460] Step 4:

[0461] The server generates optimal product recommendations using the results of sentiment analysis. It selects suggested items considering the user's emotional state and past purchase history. The inputs used are emotional state and purchase history, and the server outputs a list of recommended products, including luxury goods and new products.

[0462] Step 5:

[0463] The user selects the suggested items. The server then places an automatic order based on this selection. In this step, the ordering process proceeds using the selected item information as input, and an order confirmation is output.

[0464] Step 6:

[0465] The server retrieves expiration date information and notifies the user. Here, it retrieves information about items nearing their expiration date from a database, uses that information as input, and outputs an expiration date notification. This includes specific actions such as notifying the user via email or in-app notification.

[0466] Step 7:

[0467] The server analyzes purchasing trends within the region and proposes group purchases to other users. This process takes purchase data as input and outputs suggestions for cost reductions and benefits through group purchasing. This is where actions that encourage collaborative behavior among users take place.

[0468] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0469] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0470] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0471] [Third Embodiment]

[0472] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0473] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0474] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0475] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0476] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0477] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0478] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0479] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0480] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0481] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0482] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0483] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0484] One embodiment of this invention is a consumables management system based on voice input, which allows users to easily manage consumables using their voice. This system integrates and provides functions such as voice recognition, inventory management, ordering process, expiration date management, and regional group purchasing.

[0485] The device is equipped with a speech recognition function that converts voice input received from the user into text data. For example, if a user says to the device, "There's not enough milk," the device recognizes the voice as text and sends it to the server.

[0486] The server analyzes the received text data and identifies the corresponding consumable (in this case, milk). The server then accesses the inventory management database to check the inventory status of the identified consumable.

[0487] The server stores the user's purchasing criteria (for example, wanting to buy the cheapest product) and selects the optimal product based on inventory and price information. The server then automatically orders the selected product, allowing the user to live without worrying about stock shortages.

[0488] The server also integrates with online shopping sites and digital receipts to obtain expiration date information. For example, when the expiration date of fresh food purchased by a user approaches, the terminal notifies the user with an alert. This ensures that users can always use food in its freshest state without wasting any.

[0489] Furthermore, this system has a function to support joint purchases among users within the same region. The server analyzes the purchasing trends of users in the region and can improve delivery efficiency by coordinating bulk purchases and delivery dates.

[0490] In this way, the present invention streamlines the user's life and realizes economical and sustainable consumable management.

[0491] The following describes the processing flow.

[0492] Step 1:

[0493] The user issues a voice command regarding consumables to the terminal. The terminal receives the voice and converts the voice data into text data using its built-in voice recognition function.

[0494] Step 2:

[0495] The terminal sends the converted text data to the server.

[0496] Step 3:

[0497] The server analyzes the received text data and identifies the corresponding consumables. The server communicates with the inventory management database to check the latest inventory status of the identified consumables.

[0498] Step 4:

[0499] The server checks the user's pre-configured purchasing criteria (e.g., cheapest product, product from a specific manufacturer). Based on these criteria, the server searches for inventory and price information and selects the most suitable product.

[0500] Step 5:

[0501] The server initiates the automated ordering process, placing orders for selected items with online retailers. The purchase is then completed using the user's chosen payment method.

[0502] Step 6:

[0503] The server obtains expiration date information from online shopping sites and digital receipts. The server monitors the obtained expiration date information and notifies the user when the expiration date is approaching.

[0504] Step 7:

[0505] The server analyzes the purchasing trends of users within the region. It promotes opportunities for group buying by suggesting bulk purchases and delivery date adjustments to users.

[0506] Step 8:

[0507] When the device receives notifications or suggestions from the server, it presents that information to the user through the user interface. The user can respond to these notifications via the device.

[0508] (Example 1)

[0509] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0510] Traditional consumables management systems lacked efficiency due to limited user interaction using voice input and manual inventory management and ordering processes. Furthermore, insufficient features for expiration date management and group purchasing led to problems such as food waste and increased delivery costs. Additionally, the lack of visibility into purchase history and efficient logistics coordination meant users had limited means to effectively utilize the information.

[0511] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0512] In this invention, the server includes means for converting voice signals into text format, means for performing natural language processing to identify consumables, and means for automatically ordering selected products. This enables efficient inventory management and flexible expiration date management through voice input. Furthermore, it is possible to improve the user experience and contribute to reducing logistics costs through joint purchasing suggestions based on purchase data analysis and visualization of purchase history.

[0513] "Voice input" refers to the process of acquiring instructions and information from the user in the form of voice.

[0514] "Audio signal" is one of the formats used to process sound as digital data.

[0515] "Text format" refers to a format in which audio or other data has been converted into text data.

[0516] "Information processing means" refers to devices and programs that analyze digital data and perform corresponding functions.

[0517] "Natural language processing" refers to the technology of analyzing text data and extracting meaning.

[0518] "Consumable goods" refer to items used in daily life that decrease or are consumed through use.

[0519] "Automatic ordering methods" refer to the function of a system that orders necessary products online or by other means based on user instructions or conditions.

[0520] "Expiration date information" refers to data regarding the period during which a particular item can be used while maintaining its safety and quality.

[0521] "Group purchasing" refers to a system in which multiple users jointly purchase goods or services, thereby reducing costs and improving efficiency.

[0522] A "user interface" refers to the means or screens through which a system and a user interact when communicating information.

[0523] "Logistical efficiency" refers to the effective operation of goods delivery and distribution while minimizing time and costs.

[0524] This invention is a system for advanced consumable management that utilizes voice input, and helps users manage their daily necessities efficiently and automatically.

[0525] System Configuration

[0526] 1. Voice input and conversion

[0527] The device is equipped with a microphone to acquire voice input from the user. Speech recognition software (e.g., a typical cloud-based speech recognition service) is used to convert the voice signal into text format.

[0528] 2. Data analysis and consumable identification

[0529] The server analyzes the received text data using natural language processing. This analysis identifies consumable items (e.g., milk). A general-purpose AI model is used for natural language processing.

[0530] 3. Inventory management and automated ordering

[0531] The server accesses the database system to retrieve inventory information for consumables. If a shortage is detected, it automatically places an order for the most suitable product based on the user's purchase conditions. By utilizing the online shopping API, efficient product selection and ordering are possible.

[0532] 4. Expiration date management and notification

[0533] The server retrieves expiration date information for purchased products from online resources and notifies the user via their device when the expiration date is approaching.

[0534] 5. Optimizing group buying

[0535] The server analyzes the purchase history of multiple users and proposes cost-effective group purchases. It also improves logistics efficiency by coordinating deliveries among users within the same region.

[0536] Specific example

[0537] For example, if a user says "We're running low on milk," this information is converted to text using speech recognition, and after the server analyzes and checks inventory, the optimal amount of milk is automatically ordered. This process allows users to go about their daily lives without worrying about running out of consumables.

[0538] Example of a prompt

[0539] "A system that allows you to instantly purchase additional seasonings you realize you're running low on while cooking at home. It analyzes voice commands like 'I'm running low on tomato sauce,' selects the most suitable product, and automatically places an order."

[0540] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0541] Step 1:

[0542] The user provides voice input through the device's microphone. For example, they might say, "There's not enough milk." This voice input becomes the program's first input.

[0543] The device uses speech recognition software to convert the received speech signal into text data. This conversion process transforms the user's speech into a machine-readable text format.

[0544] Step 2:

[0545] The terminal sends the converted text data to the server. The text data becomes the input to the server.

[0546] The server uses natural language processing on the received text data to analyze keywords such as "few." This identifies a specific consumable item (in this case, "milk") and extracts the information necessary for the next processing step.

[0547] Step 3:

[0548] The server accesses the inventory management database to check the inventory of consumables (milk) identified by the text analysis results. This database query becomes the server input, and inventory information is output.

[0549] If the server has low or insufficient inventory, it uses a generative AI model to select the optimal order candidates based on the user's purchasing criteria. This results in a product list that matches the purchasing criteria.

[0550] Step 4:

[0551] The server uses an online shopping API to automatically place an order for the selected items. This API call is the next input to the server, and the order completion confirmation is the output.

[0552] This allows users to go about their daily lives with peace of mind, knowing that a new milk order has been automatically placed.

[0553] Step 5:

[0554] The server collects expiration date data and manages it in conjunction with digital receipts and other digital resources. When the expiration date is approaching, an alert is generated.

[0555] The device displays or sounds a warning to the user based on notifications from the server. This allows users to be more mindful of products nearing their expiration date and use them efficiently.

[0556] Step 6:

[0557] The server analyzes data from other users in the region to identify opportunities for group buying. The analysis of relevant data becomes the server's input, and proposed group buying plans are output.

[0558] Users can receive information about group purchases and, if they wish, participate to reduce costs.

[0559] (Application Example 1)

[0560] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0561] Modern households use a large number of consumables, and managing their inventory, ordering, and expiration dates presents challenges. Traditional methods often result in shortages or surpluses of consumables, making economical and efficient management difficult. Furthermore, efficient purchasing through cooperation with local residents is not adequately implemented. Therefore, it is necessary to address these issues, reduce resource waste within households, and improve the lives of users.

[0562] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0563] In this invention, the server includes information processing means for receiving voice input and converting voice data into text data; information processing means for analyzing the converted text data and confirming the inventory status of the corresponding materials; information processing means for selecting the optimal materials based on user settings and placing automatic orders; information processing means for acquiring expiration date information and notifying the user that the expiration date is approaching; information processing means for analyzing the purchase data of multiple users in the area and proposing cooperative purchases; and information processing means having a function to interact with the user via voice in a home automated device and notify them of inventory information. This makes inventory management of consumables more efficient, enables proper management of expiration dates, and allows for cooperative purchases with local residents.

[0564] "Voice input" refers to the operation of receiving voice signals from the user, which serve as the basic data for information processing.

[0565] "Information processing means" refers to devices and algorithms for converting audio data into text data, or for analyzing text data and processing the identified information.

[0566] "Inventory status" refers to information indicating the current quantity and condition of specific materials or goods.

[0567] "Materials" is a general term for consumables and necessities used in homes and organizations.

[0568] "Expiration date information" refers to information about the period during which a particular item can be used, and indicates the shelf life of food and consumables.

[0569] "Group purchasing" is a method in which multiple users cooperate to purchase goods in bulk, thereby increasing cost efficiency.

[0570] "Household automated devices" refer to machines or electrical devices used in the home that are designed to perform certain tasks automatically.

[0571] "Dialogue through voice" refers to a form of communication in which machines and humans exchange information using voice, employing speech synthesis and recognition technologies.

[0572] "User settings" refer to information that each user individually sets, serving as guidelines for operations and actions based on their preferences and conditions.

[0573] To implement this invention, first, a household automated device receives voice input from the user using a high-performance microphone. The voice input is converted into text data using speech recognition software. This converted text data is transmitted to a server via the internet. The server analyzes this text data and uses information processing means to check the inventory status of the corresponding materials.

[0574] After checking inventory status, the server selects the most suitable materials based on user settings, for example, prioritizing the cheapest items. The selected materials are automatically ordered, and the user is notified as needed. At this stage, the ordering process is simplified by utilizing the online shopping platform's API.

[0575] Furthermore, the server retrieves expiration date information for purchased materials and notifies users through automated home devices. This allows users to use materials efficiently and reduce waste. In addition, by analyzing purchase data from multiple users in the region, joint purchasing suggestions are made, resulting in cost reductions.

[0576] As a concrete example, if a home automated system receives voice input from a user saying "there's not enough milk," the server checks the inventory status and automatically orders the best option from nearby supermarkets or online stores. This process utilizes tools such as speech recognition APIs like Google Cloud Speech-to-Text, database management systems like MySQL, and the Rakuten Market API.

[0577] When providing this information to the user, a generative AI model can be used based on the following example prompt:

[0578] "Generate a scenario where a robotic consumables manager detects a milk shortage and selects and orders the most economical option."

[0579] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0580] Step 1:

[0581] The terminal receives voice input from the user. In this step, the home automated device collects the user's voice using a high-performance microphone and stores it as voice data. The input is a voice signal, and the output is digitized voice data.

[0582] Step 2:

[0583] The device uses a speech recognition API to convert audio data into text data. It analyzes the audio data using a speech recognition service such as Google Cloud Speech-to-Text and outputs it as text data. The input is audio data, and the output is text data.

[0584] Step 3:

[0585] The server receives text data and begins analysis. Based on the text data, it identifies the corresponding materials and prepares to check their inventory status. The input is text data, and the output is information about the items as a result of the analysis.

[0586] Step 4:

[0587] The server accesses the inventory management database to check the inventory status of the identified material. Here, the server uses an SQL query to retrieve the quantity information of the relevant material. The input is item information, and the output is the inventory status.

[0588] Step 5:

[0589] The server refers to user settings and determines the optimal ordering conditions. The server selects the most suitable materials based on price, past purchase history, and desired conditions. Inputs are inventory status and user settings, and output is a list of items to be ordered.

[0590] Step 6:

[0591] The server uses the API of the online shopping platform to execute automated orders. The server processes the order based on the selected items and confirms the transaction. The input is an item list, and the output is an order completion notification.

[0592] Step 7:

[0593] The server retrieves expiration date information and generates an alert regarding the expiration date. The server references digital receipts and online information and sends a notification to the user's device. The input is expiration date data, and the output is an alert message.

[0594] Step 8:

[0595] The server analyzes purchase data from multiple users within a region and proposes group purchases. It uses statistical methods to create an efficient collaborative purchase plan and notifies users. The input is purchase data, and the output is the proposed plan.

[0596] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0597] One embodiment of this invention is a consumables management system based on voice input that incorporates an emotion engine for analyzing the user's emotions. This system includes voice recognition, inventory management, automatic ordering, expiration date management, and a regional group purchasing function, as well as a function to optimize purchase suggestions that take into account the user's emotional state.

[0598] First, the device acquires voice input from the user and converts it into text data using its speech recognition function. At this time, the emotion engine analyzes the characteristics of the voice data and estimates various emotional states (e.g., satisfaction, anxiety, excitement, etc.). For example, if the user says, "I'm happy that the milk is all gone," the emotion engine analyzes that the user is in a positive emotional state.

[0599] The server identifies consumables based on the converted text data and checks their inventory status. It can then adjust purchase suggestions to best suit the user, taking into account the results of sentiment analysis. For example, if the user shows positive emotions, the server can suggest slightly more luxurious or new products in addition to their usual purchasing patterns.

[0600] Furthermore, the timing and importance of notifications can be adjusted according to the user's emotions, as recognized by the emotion engine. For example, if a user is feeling stressed, the server will notify them early about products nearing their expiration date to help alleviate their burden.

[0601] This system can analyze purchasing trends within a region and, taking emotions into account, can also suggest group purchases. For example, if many users have positive feelings when purchasing a particular product, the system aims to increase overall user satisfaction by suggesting that product as a target for group purchases.

[0602] In this way, by combining voice input and emotion analysis, the present invention improves the user experience and enables more efficient and emotionally responsive management of consumables.

[0603] The following describes the processing flow.

[0604] Step 1:

[0605] The user speaks a voice command regarding consumables to the terminal. The terminal receives this voice using a speech recognition module and processes it as voice data.

[0606] Step 2:

[0607] The device converts the acquired audio data into text data and simultaneously performs emotional analysis of the audio data using an emotion engine. This allows it to estimate the user's emotional state (e.g., satisfaction, stress, anxiety, etc.).

[0608] Step 3:

[0609] The terminal sends the converted text data and sentiment analysis results to the server.

[0610] Step 4:

[0611] The server analyzes the received text data and identifies the relevant consumables. The server then accesses the inventory management database to check the inventory status of the relevant consumables.

[0612] Step 5:

[0613] The server adjusts purchase suggestions based on the user's purchasing conditions and sentiment analysis results. For example, if the user is feeling stressed, it will suggest simple, relaxing consumables.

[0614] Step 6:

[0615] The server initiates an automated ordering process based on the selected products and completes the order with the online retailer using the user's preferred payment method.

[0616] Step 7:

[0617] The server retrieves product expiration date information from an online platform and adjusts the timing and content of notifications for products nearing their expiration date, taking into account the user's emotional state.

[0618] Step 8:

[0619] The server analyzes purchase data from other users in the region and suggests group purchases, taking their emotional state into consideration. If a user's emotional state is positive, it collaborates with other users with similar emotional states to encourage purchases.

[0620] Step 9:

[0621] The device receives notifications and suggestions from the server and presents them to the user in an easy-to-understand manner. The user makes decisions based on the presented information and, if necessary, engages in purchasing activities.

[0622] (Example 2)

[0623] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0624] In modern consumer goods management, challenges include the difficulty of responding flexibly to users' emotional states and the fact that inventory management and purchase recommendations do not take individual user feelings into consideration. Therefore, a more personalized approach is needed to improve user satisfaction and the purchasing experience. Furthermore, improving local group purchasing and delivery efficiency are also challenges.

[0625] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0626] In this invention, the server includes means for estimating the user's emotional state based on voice input, means for selecting the optimal consumables and making purchase suggestions based on the emotional state and user settings, and means for acquiring expiration date information and adjusting the timing of notifications according to the user's emotions. This enables efficient and personalized consumable management and purchase suggestions while taking the user's emotions into consideration.

[0627] "Voice input" is the process of acquiring acoustic signals from a user and converting that information into a data format that a machine can understand.

[0628] "Text data" refers to digital character data converted from speech, which holds information in a format that is easy for computers to process.

[0629] "Consumables" are items that require periodic replenishment to maintain their condition through use or consumption.

[0630] "Emotional state" refers to a state in which the user's emotional movements and feelings are analyzed and identified as a specific emotion, such as positive or negative.

[0631] A "purchase suggestion" is a guide or recommendation to provide appropriate products or services based on the user's interests and needs.

[0632] "Expiration date" refers to the date by which a particular item can maintain its quality without deterioration; after this date, there are concerns about the deterioration of its safety and quality.

[0633] "Local users" refers to a group of users who reside or work in a specific geographical area, and whose purchasing activities and trends are the subject of analysis.

[0634] "Group buying" is an activity in which multiple users cooperate to purchase the same or similar products together, aiming to reduce costs and improve purchasing conditions.

[0635] This invention is a consumables management system that combines voice input and emotion analysis, enabling efficient inventory management while improving the user experience. Specific embodiments are described below.

[0636] First, the device acquires the user's voice using an input device (e.g., a smart speaker or smartphone). This voice data is then converted into text data using speech recognition software (e.g., Google Cloud Speech-to-Text). The speech recognition process digitizes the acoustic signal and analyzes its content as text.

[0637] Next, the server uses the converted text data to identify consumables and check their inventory status. Here, database technology is used to manage product information and check inventory quantities and expiration dates. Furthermore, a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) is used to estimate the user's emotional state from the text, identifying positive or negative feelings.

[0638] The server optimizes consumable purchase suggestions by considering the user's emotional state. This utilizes an AI model to suggest products tailored to each individual user, based on their past purchase history and inventory information. Furthermore, it flexibly adjusts notification timing based on the user's emotions and inventory status, providing appropriate notifications to reduce stress for products nearing their expiration date.

[0639] Furthermore, the server analyzes purchasing data from multiple users within a region and proposes group purchases for specific consumables. For products that receive positive feedback from many users, the system aims to improve overall purchasing efficiency and satisfaction by offering group purchase options.

[0640] As a concrete example, a user might voice-input, "I want to buy a lot of bottled water." Based on this information, the server checks the inventory status and sentiment analysis results, and then suggests purchasing a large-capacity pack. At this point, by confirming that there is sufficient stock, it suggests immediate shipping, thus responding to the user's needs.

[0641] As an example of how the generative AI model can be used, we will show a prompt message. Using a prompt message such as, "Based on the user's purchase history and emotional state, identify appropriate consumables and create individually customized suggestions," the system will provide highly accurate suggestions.

[0642] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0643] Step 1:

[0644] The device acquires voice input from the user. This input is collected as an acoustic signal using the microphone of a smart speaker or smartphone. Next, this acoustic signal is converted into text data by speech recognition software. In this conversion process, the received audio is processed into a digital format and converted into text information by a speech recognition engine (e.g., Google Cloud Speech-to-Text). The output of this step is the text data converted from the audio.

[0645] Step 2:

[0646] The server receives text data sent from the terminal and identifies consumables and checks their inventory status. The input is text data, and the system uses database technology to check the inventory quantity and expiration date of the identified consumables. Database operations such as SQL are performed to search for data corresponding to the product information. The output of this step is inventory information for the identified consumables.

[0647] Step 3:

[0648] The server inputs the text data from the previous step into the sentiment analysis engine to estimate the user's emotional state. Here, natural language processing techniques (e.g., IBM Watson Tone Analyzer) are used to analyze emotional elements from the text. Specifically, an algorithm is applied that identifies emotions from keywords and context in the text. The output of this step is the estimated emotional state of the user.

[0649] Step 4:

[0650] The server makes optimal consumable purchase suggestions based on inventory information and the user's emotional state. The inputs here are inventory information and emotional state, which are used to generate product suggestions tailored to the user's preferences using an AI model. This process analyzes existing purchase history and emotional analysis results, inputting prompts into the AI ​​to generate appropriate suggestions. The output of this step is a personalized purchase suggestion for the user.

[0651] Step 5:

[0652] The server adjusts notification timing based on the user's emotional state. It uses emotional state and expiration date information as input to determine an appropriate notification schedule. Based on this identified schedule, it sends an alert in advance if the expiration date is approaching. The output of this step is a timely notification to the user.

[0653] Step 6:

[0654] The server evaluates purchase data and sentiment analysis results collected from users within a region to formulate group buying suggestions. The input is data from multiple users, which is analyzed to identify products that consistently receive high ratings. Based on this, the server provides users with group buying options. The output of this step is a group buying suggestion for all users.

[0655] (Application Example 2)

[0656] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0657] Conventional consumables management systems and electronic payment services often fail to consider the user's emotional state when making purchase suggestions, resulting in an inadequate user experience. Furthermore, automated ordering and expiration date management lack sufficient consideration for reducing the psychological burden on users. Therefore, there is a need for systems that provide appropriate product suggestions and notifications tailored to the user's current emotional state.

[0658] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0659] In this invention, the server includes means for receiving voice input and converting voice data into text data; means for analyzing the converted text data and checking the inventory status of the corresponding items; means for selecting the optimal items based on user settings and placing automatic orders; means for acquiring expiration date information and notifying the user that the expiration date is approaching; means for analyzing purchase data of multiple users in a region and proposing joint purchases; means for analyzing the characteristics of voice data and estimating the user's emotional state; and means for optimizing suggestions based on the emotional state and notifying the user of the information. This enables purchase suggestions and consumable management that are tailored to the user's emotions.

[0660] "Voice input" is a method by which users provide information to a system through their voice.

[0661] "Audio data" refers to audio signals obtained from audio input that have been converted into digital data.

[0662] "Text data" refers to string information obtained by analyzing audio data.

[0663] "Information processing means" refers to a device or function that processes data according to a set procedure and utilizes the results obtained.

[0664] "Inventory status" refers to information indicating the current level of a particular item or resource.

[0665] "Automatic ordering" is a process that automatically places orders for goods based on pre-set conditions.

[0666] "Expiration date information" refers to data indicating the date by which a particular product's quality is guaranteed.

[0667] "Group purchasing" is a form of purchasing where multiple users cooperate to buy the same item, aiming to reduce costs and improve efficiency.

[0668] "Emotion analysis" is a technology that estimates a person's emotional state using voice and text data.

[0669] "Optimizing proposals" is the process of presenting the most suitable options and information based on the user's needs and circumstances.

[0670] The system for implementing this invention mainly uses a server, a user terminal, speech recognition software, and an emotion analysis engine. The operation of each component is described below.

[0671] First, the user provides voice input to the user terminal. This terminal uses speech recognition software to convert the voice data into text data. Widely used software, such as the SpeechRecognition library, can be used for speech recognition. Once the text data is generated, the server receives it and checks the inventory status of the corresponding item. In the backend, database queries are used to connect with the inventory management system.

[0672] Furthermore, this server uses an emotion analysis engine to estimate the user's emotional state from the obtained voice data. Existing emotion analysis libraries and generative AI models can be used for this emotion analysis. For example, if a positive emotion is detected, the server will suggest a slightly luxurious item to the user. This suggestion is optimized by taking into account past purchase history and purchasing patterns.

[0673] When a user selects a suggested item, the server automatically places an order. This ordering process automates the purchase procedure based on the user's existing settings. Simultaneously, the server has a function to notify the user when an item is nearing its expiration date, based on expiration date information. This is an important feature to reduce the user's burden, and expiration date information is collected and managed via an API.

[0674] For example, if a user voice-inputs, "I'm in a good mood today, so I'd like to enjoy a special wine with dinner," the system will check the wine's inventory and suggest a slightly more elegant option than usual, along with a positive sentiment. Furthermore, based on the user's past preference patterns, it will determine if there are any limited-time offers or promotions available.

[0675] An example of a prompt for a generative AI model would be a question like, "Design logic to suggest luxury items when the user expresses an emotion they are enjoying." This prompt contributes to improving emotion-based purchase suggestion logic.

[0676] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0677] Step 1:

[0678] The user provides shopping requests to the terminal via voice input. The terminal uses voice recognition software to convert the voice data into text data. In this process, a voice recognition library analyzes the voice signal and generates corresponding string data.

[0679] Step 2:

[0680] The terminal sends the converted text data to the server. The server receives this text data and executes a database query to check the inventory status of the items. Here, the server takes the text data as input, retrieves the corresponding product information from the database, and outputs whether or not the item is in stock.

[0681] Step 3:

[0682] The server uses a generative AI model to analyze the user's emotional state from text data. The emotion analysis engine takes text data as input, analyzes features extracted from the text, and outputs an estimated result of the emotional state. For example, it can determine whether the user's statements are positive.

[0683] Step 4:

[0684] The server generates optimal product recommendations using the results of sentiment analysis. It selects suggested items considering the user's emotional state and past purchase history. The inputs used are emotional state and purchase history, and the server outputs a list of recommended products, including luxury goods and new products.

[0685] Step 5:

[0686] The user selects the suggested items. The server then places an automatic order based on this selection. In this step, the ordering process proceeds using the selected item information as input, and an order confirmation is output.

[0687] Step 6:

[0688] The server retrieves expiration date information and notifies the user. Here, it retrieves information about items nearing their expiration date from a database, uses that information as input, and outputs an expiration date notification. This includes specific actions such as notifying the user via email or in-app notification.

[0689] Step 7:

[0690] The server analyzes purchasing trends within the region and proposes group purchases to other users. This process takes purchase data as input and outputs suggestions for cost reductions and benefits through group purchasing. This is where actions that encourage collaborative behavior among users take place.

[0691] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0692] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0693] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0694] [Fourth Embodiment]

[0695] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0696] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0697] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0698] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0699] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0700] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0701] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0702] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0703] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0704] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0705] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0706] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0707] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0708] One embodiment of this invention is a consumables management system based on voice input, which allows users to easily manage consumables using their voice. This system integrates and provides functions such as voice recognition, inventory management, ordering process, expiration date management, and regional group purchasing.

[0709] The device is equipped with a speech recognition function that converts voice input received from the user into text data. For example, if a user says to the device, "There's not enough milk," the device recognizes the voice as text and sends it to the server.

[0710] The server analyzes the received text data and identifies the corresponding consumable (in this case, milk). The server then accesses the inventory management database to check the inventory status of the identified consumable.

[0711] The server stores the user's purchasing criteria (for example, wanting to buy the cheapest product) and selects the optimal product based on inventory and price information. The server then automatically orders the selected product, allowing the user to live without worrying about stock shortages.

[0712] The server also integrates with online shopping sites and digital receipts to obtain expiration date information. For example, when the expiration date of fresh food purchased by a user approaches, the terminal notifies the user with an alert. This ensures that users can always use food in its freshest state without wasting any.

[0713] Furthermore, this system has a function to support joint purchases among users within the same region. The server analyzes the purchasing trends of users in the region and can improve delivery efficiency by coordinating bulk purchases and delivery dates.

[0714] In this way, the present invention streamlines the user's life and realizes economical and sustainable consumable management.

[0715] The following describes the processing flow.

[0716] Step 1:

[0717] The user issues a voice command regarding consumables to the terminal. The terminal receives the voice and converts the voice data into text data using its built-in voice recognition function.

[0718] Step 2:

[0719] The terminal sends the converted text data to the server.

[0720] Step 3:

[0721] The server analyzes the received text data and identifies the corresponding consumables. The server communicates with the inventory management database to check the latest inventory status of the identified consumables.

[0722] Step 4:

[0723] The server checks the user's pre-configured purchasing criteria (e.g., cheapest product, product from a specific manufacturer). Based on these criteria, the server searches for inventory and price information and selects the most suitable product.

[0724] Step 5:

[0725] The server initiates the automated ordering process, placing orders for selected items with online retailers. The purchase is then completed using the user's chosen payment method.

[0726] Step 6:

[0727] The server obtains expiration date information from online shopping sites and digital receipts. The server monitors the obtained expiration date information and notifies the user when the expiration date is approaching.

[0728] Step 7:

[0729] The server analyzes the purchasing trends of users within the region. It promotes opportunities for group buying by suggesting bulk purchases and delivery date adjustments to users.

[0730] Step 8:

[0731] When the device receives notifications or suggestions from the server, it presents that information to the user through the user interface. The user can respond to these notifications via the device.

[0732] (Example 1)

[0733] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0734] Traditional consumables management systems lacked efficiency due to limited user interaction using voice input and manual inventory management and ordering processes. Furthermore, insufficient features for expiration date management and group purchasing led to problems such as food waste and increased delivery costs. Additionally, the lack of visibility into purchase history and efficient logistics coordination meant users had limited means to effectively utilize the information.

[0735] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0736] In this invention, the server includes means for converting voice signals into text format, means for performing natural language processing to identify consumables, and means for automatically ordering selected products. This enables efficient inventory management and flexible expiration date management through voice input. Furthermore, it is possible to improve the user experience and contribute to reducing logistics costs through joint purchasing suggestions based on purchase data analysis and visualization of purchase history.

[0737] "Voice input" refers to the process of acquiring instructions and information from the user in the form of voice.

[0738] "Audio signal" is one of the formats used to process sound as digital data.

[0739] "Text format" refers to a format in which audio or other data has been converted into text data.

[0740] "Information processing means" refers to devices and programs that analyze digital data and perform corresponding functions.

[0741] "Natural language processing" refers to the technology of analyzing text data and extracting meaning.

[0742] "Consumable goods" refer to items used in daily life that decrease or are consumed through use.

[0743] "Automatic ordering methods" refer to the function of a system that orders necessary products online or by other means based on user instructions or conditions.

[0744] "Expiration date information" refers to data regarding the period during which a particular item can be used while maintaining its safety and quality.

[0745] "Group purchasing" refers to a system in which multiple users jointly purchase goods or services, thereby reducing costs and improving efficiency.

[0746] A "user interface" refers to the means or screens through which a system and a user interact when communicating information.

[0747] "Logistical efficiency" refers to the effective operation of goods delivery and distribution while minimizing time and costs.

[0748] This invention is a system for advanced consumable management that utilizes voice input, and helps users manage their daily necessities efficiently and automatically.

[0749] System Configuration

[0750] 1. Voice input and conversion

[0751] The device is equipped with a microphone to acquire voice input from the user. Speech recognition software (e.g., a typical cloud-based speech recognition service) is used to convert the voice signal into text format.

[0752] 2. Data analysis and consumable identification

[0753] The server analyzes the received text data using natural language processing. This analysis identifies consumable items (e.g., milk). A general-purpose AI model is used for natural language processing.

[0754] 3. Inventory management and automated ordering

[0755] The server accesses the database system to retrieve inventory information for consumables. If a shortage is detected, it automatically places an order for the most suitable product based on the user's purchase conditions. By utilizing the online shopping API, efficient product selection and ordering are possible.

[0756] 4. Expiration date management and notification

[0757] The server retrieves expiration date information for purchased products from online resources and notifies the user via their device when the expiration date is approaching.

[0758] 5. Optimizing group buying

[0759] The server analyzes the purchase history of multiple users and proposes cost-effective group purchases. It also improves logistics efficiency by coordinating deliveries among users within the same region.

[0760] Specific example

[0761] For example, if a user says "We're running low on milk," this information is converted to text using speech recognition, and after the server analyzes and checks inventory, the optimal amount of milk is automatically ordered. This process allows users to go about their daily lives without worrying about running out of consumables.

[0762] Example of a prompt

[0763] "A system that allows you to instantly purchase additional seasonings you realize you're running low on while cooking at home. It analyzes voice commands like 'I'm running low on tomato sauce,' selects the most suitable product, and automatically places an order."

[0764] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0765] Step 1:

[0766] The user provides voice input through the device's microphone. For example, they might say, "There's not enough milk." This voice input becomes the program's first input.

[0767] The device uses speech recognition software to convert the received speech signal into text data. This conversion process transforms the user's speech into a machine-readable text format.

[0768] Step 2:

[0769] The terminal sends the converted text data to the server. The text data becomes the input to the server.

[0770] The server uses natural language processing on the received text data to analyze keywords such as "few." This identifies a specific consumable item (in this case, "milk") and extracts the information necessary for the next processing step.

[0771] Step 3:

[0772] The server accesses the inventory management database to check the inventory of consumables (milk) identified by the text analysis results. This database query becomes the server input, and inventory information is output.

[0773] If the server has low or insufficient inventory, it uses a generative AI model to select the optimal order candidates based on the user's purchasing criteria. This results in a product list that matches the purchasing criteria.

[0774] Step 4:

[0775] The server uses an online shopping API to automatically place an order for the selected items. This API call is the next input to the server, and the order completion confirmation is the output.

[0776] This allows users to go about their daily lives with peace of mind, knowing that a new milk order has been automatically placed.

[0777] Step 5:

[0778] The server collects expiration date data and manages it in conjunction with digital receipts and other digital resources. When the expiration date is approaching, an alert is generated.

[0779] The device displays or sounds a warning to the user based on notifications from the server. This allows users to be more mindful of products nearing their expiration date and use them efficiently.

[0780] Step 6:

[0781] The server analyzes data from other users in the region to identify opportunities for group buying. The analysis of relevant data becomes the server's input, and proposed group buying plans are output.

[0782] Users can receive information about group purchases and, if they wish, participate to reduce costs.

[0783] (Application Example 1)

[0784] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0785] Modern households use a large number of consumables, and managing their inventory, ordering, and expiration dates presents challenges. Traditional methods often result in shortages or surpluses of consumables, making economical and efficient management difficult. Furthermore, efficient purchasing through cooperation with local residents is not adequately implemented. Therefore, it is necessary to address these issues, reduce resource waste within households, and improve the lives of users.

[0786] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0787] In this invention, the server includes information processing means for receiving voice input and converting voice data into text data; information processing means for analyzing the converted text data and confirming the inventory status of the corresponding materials; information processing means for selecting the optimal materials based on user settings and placing automatic orders; information processing means for acquiring expiration date information and notifying the user that the expiration date is approaching; information processing means for analyzing the purchase data of multiple users in the area and proposing cooperative purchases; and information processing means having a function to interact with the user via voice in a home automated device and notify them of inventory information. This makes inventory management of consumables more efficient, enables proper management of expiration dates, and allows for cooperative purchases with local residents.

[0788] "Voice input" refers to the operation of receiving voice signals from the user, which serve as the basic data for information processing.

[0789] "Information processing means" refers to devices and algorithms for converting audio data into text data, or for analyzing text data and processing the identified information.

[0790] "Inventory status" refers to information indicating the current quantity and condition of specific materials or goods.

[0791] "Materials" is a general term for consumables and necessities used in homes and organizations.

[0792] "Expiration date information" refers to information about the period during which a particular item can be used, and indicates the shelf life of food and consumables.

[0793] "Group purchasing" is a method in which multiple users cooperate to purchase goods in bulk, thereby increasing cost efficiency.

[0794] "Household automated devices" refer to machines or electrical devices used in the home that are designed to perform certain tasks automatically.

[0795] "Dialogue through voice" refers to a form of communication in which machines and humans exchange information using voice, employing speech synthesis and recognition technologies.

[0796] "User settings" refer to information that each user individually sets, serving as guidelines for operations and actions based on their preferences and conditions.

[0797] To implement this invention, first, a household automated device receives voice input from the user using a high-performance microphone. The voice input is converted into text data using speech recognition software. This converted text data is transmitted to a server via the internet. The server analyzes this text data and uses information processing means to check the inventory status of the corresponding materials.

[0798] After checking inventory status, the server selects the most suitable materials based on user settings, for example, prioritizing the cheapest items. The selected materials are automatically ordered, and the user is notified as needed. At this stage, the ordering process is simplified by utilizing the online shopping platform's API.

[0799] Furthermore, the server retrieves expiration date information for purchased materials and notifies users through automated home devices. This allows users to use materials efficiently and reduce waste. In addition, by analyzing purchase data from multiple users in the region, joint purchasing suggestions are made, resulting in cost reductions.

[0800] As a concrete example, if a home automated system receives voice input from a user saying "there's not enough milk," the server checks the inventory status and automatically orders the best option from nearby supermarkets or online stores. This process utilizes tools such as speech recognition APIs like Google Cloud Speech-to-Text, database management systems like MySQL, and the Rakuten Market API.

[0801] When providing this information to the user, a generative AI model can be used based on the following example prompt:

[0802] "Generate a scenario where a robotic consumables manager detects a milk shortage and selects and orders the most economical option."

[0803] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0804] Step 1:

[0805] The terminal receives voice input from the user. In this step, the home automated device collects the user's voice using a high-performance microphone and stores it as voice data. The input is a voice signal, and the output is digitized voice data.

[0806] Step 2:

[0807] The device uses a speech recognition API to convert audio data into text data. It analyzes the audio data using a speech recognition service such as Google Cloud Speech-to-Text and outputs it as text data. The input is audio data, and the output is text data.

[0808] Step 3:

[0809] The server receives text data and begins analysis. Based on the text data, it identifies the corresponding materials and prepares to check their inventory status. The input is text data, and the output is information about the items as a result of the analysis.

[0810] Step 4:

[0811] The server accesses the inventory management database to check the inventory status of the identified material. Here, the server uses an SQL query to retrieve the quantity information of the relevant material. The input is item information, and the output is the inventory status.

[0812] Step 5:

[0813] The server refers to user settings and determines the optimal ordering conditions. The server selects the most suitable materials based on price, past purchase history, and desired conditions. Inputs are inventory status and user settings, and output is a list of items to be ordered.

[0814] Step 6:

[0815] The server uses the API of the online shopping platform to execute automated orders. The server processes the order based on the selected items and confirms the transaction. The input is an item list, and the output is an order completion notification.

[0816] Step 7:

[0817] The server retrieves expiration date information and generates an alert regarding the expiration date. The server references digital receipts and online information and sends a notification to the user's device. The input is expiration date data, and the output is an alert message.

[0818] Step 8:

[0819] The server analyzes purchase data from multiple users within a region and proposes group purchases. It uses statistical methods to create an efficient collaborative purchase plan and notifies users. The input is purchase data, and the output is the proposed plan.

[0820] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0821] One embodiment of this invention is a consumables management system based on voice input that incorporates an emotion engine for analyzing the user's emotions. This system includes voice recognition, inventory management, automatic ordering, expiration date management, and a regional group purchasing function, as well as a function to optimize purchase suggestions that take into account the user's emotional state.

[0822] First, the device acquires voice input from the user and converts it into text data using its speech recognition function. At this time, the emotion engine analyzes the characteristics of the voice data and estimates various emotional states (e.g., satisfaction, anxiety, excitement, etc.). For example, if the user says, "I'm happy that the milk is all gone," the emotion engine analyzes that the user is in a positive emotional state.

[0823] The server identifies consumables based on the converted text data and checks their inventory status. It can then adjust purchase suggestions to best suit the user, taking into account the results of sentiment analysis. For example, if the user shows positive emotions, the server can suggest slightly more luxurious or new products in addition to their usual purchasing patterns.

[0824] Furthermore, the timing and importance of notifications can be adjusted according to the user's emotions, as recognized by the emotion engine. For example, if a user is feeling stressed, the server will notify them early about products nearing their expiration date to help alleviate their burden.

[0825] This system can analyze purchasing trends within a region and, taking emotions into account, can also suggest group purchases. For example, if many users have positive feelings when purchasing a particular product, the system aims to increase overall user satisfaction by suggesting that product as a target for group purchases.

[0826] In this way, by combining voice input and emotion analysis, the present invention improves the user experience and enables more efficient and emotionally responsive management of consumables.

[0827] The following describes the processing flow.

[0828] Step 1:

[0829] The user speaks a voice command regarding consumables to the terminal. The terminal receives this voice using a speech recognition module and processes it as voice data.

[0830] Step 2:

[0831] The device converts the acquired audio data into text data and simultaneously performs emotional analysis of the audio data using an emotion engine. This allows it to estimate the user's emotional state (e.g., satisfaction, stress, anxiety, etc.).

[0832] Step 3:

[0833] The terminal sends the converted text data and sentiment analysis results to the server.

[0834] Step 4:

[0835] The server analyzes the received text data and identifies the relevant consumables. The server then accesses the inventory management database to check the inventory status of the relevant consumables.

[0836] Step 5:

[0837] The server adjusts purchase suggestions based on the user's purchasing conditions and sentiment analysis results. For example, if the user is feeling stressed, it will suggest simple, relaxing consumables.

[0838] Step 6:

[0839] The server initiates an automated ordering process based on the selected products and completes the order with the online retailer using the user's preferred payment method.

[0840] Step 7:

[0841] The server retrieves product expiration date information from an online platform and adjusts the timing and content of notifications for products nearing their expiration date, taking into account the user's emotional state.

[0842] Step 8:

[0843] The server analyzes purchase data from other users in the region and suggests group purchases, taking their emotional state into consideration. If a user's emotional state is positive, it collaborates with other users with similar emotional states to encourage purchases.

[0844] Step 9:

[0845] The device receives notifications and suggestions from the server and presents them to the user in an easy-to-understand manner. The user makes decisions based on the presented information and, if necessary, engages in purchasing activities.

[0846] (Example 2)

[0847] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0848] In modern consumer goods management, challenges include the difficulty of responding flexibly to users' emotional states and the fact that inventory management and purchase recommendations do not take individual user feelings into consideration. Therefore, a more personalized approach is needed to improve user satisfaction and the purchasing experience. Furthermore, improving local group purchasing and delivery efficiency are also challenges.

[0849] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0850] In this invention, the server includes means for estimating the user's emotional state based on voice input, means for selecting the optimal consumables and making purchase suggestions based on the emotional state and user settings, and means for acquiring expiration date information and adjusting the timing of notifications according to the user's emotions. This enables efficient and personalized consumable management and purchase suggestions while taking the user's emotions into consideration.

[0851] "Voice input" is the process of acquiring acoustic signals from a user and converting that information into a data format that a machine can understand.

[0852] "Text data" refers to digital character data converted from speech, which holds information in a format that is easy for computers to process.

[0853] "Consumables" are items that require periodic replenishment to maintain their condition through use or consumption.

[0854] "Emotional state" refers to a state in which the user's emotional movements and feelings are analyzed and identified as a specific emotion, such as positive or negative.

[0855] A "purchase suggestion" is a guide or recommendation to provide appropriate products or services based on the user's interests and needs.

[0856] "Expiration date" refers to the date by which a particular item can maintain its quality without deterioration; after this date, there are concerns about the deterioration of its safety and quality.

[0857] "Local users" refers to a group of users who reside or work in a specific geographical area, and whose purchasing activities and trends are the subject of analysis.

[0858] "Group buying" is an activity in which multiple users cooperate to purchase the same or similar products together, aiming to reduce costs and improve purchasing conditions.

[0859] This invention is a consumables management system that combines voice input and emotion analysis, enabling efficient inventory management while improving the user experience. Specific embodiments are described below.

[0860] First, the device acquires the user's voice using an input device (e.g., a smart speaker or smartphone). This voice data is then converted into text data using speech recognition software (e.g., Google Cloud Speech-to-Text). The speech recognition process digitizes the acoustic signal and analyzes its content as text.

[0861] Next, the server uses the converted text data to identify consumables and check their inventory status. Here, database technology is used to manage product information and check inventory quantities and expiration dates. Furthermore, a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) is used to estimate the user's emotional state from the text, identifying positive or negative feelings.

[0862] The server optimizes consumable purchase suggestions by considering the user's emotional state. This utilizes an AI model to suggest products tailored to each individual user, based on their past purchase history and inventory information. Furthermore, it flexibly adjusts notification timing based on the user's emotions and inventory status, providing appropriate notifications to reduce stress for products nearing their expiration date.

[0863] Furthermore, the server analyzes purchasing data from multiple users within a region and proposes group purchases for specific consumables. For products that receive positive feedback from many users, the system aims to improve overall purchasing efficiency and satisfaction by offering group purchase options.

[0864] As a concrete example, a user might voice-input, "I want to buy a lot of bottled water." Based on this information, the server checks the inventory status and sentiment analysis results, and then suggests purchasing a large-capacity pack. At this point, by confirming that there is sufficient stock, it suggests immediate shipping, thus responding to the user's needs.

[0865] As an example of how the generative AI model can be used, we will show a prompt message. Using a prompt message such as, "Based on the user's purchase history and emotional state, identify appropriate consumables and create individually customized suggestions," the system will provide highly accurate suggestions.

[0866] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0867] Step 1:

[0868] The device acquires voice input from the user. This input is collected as an acoustic signal using the microphone of a smart speaker or smartphone. Next, this acoustic signal is converted into text data by speech recognition software. In this conversion process, the received audio is processed into a digital format and converted into text information by a speech recognition engine (e.g., Google Cloud Speech-to-Text). The output of this step is the text data converted from the audio.

[0869] Step 2:

[0870] The server receives text data sent from the terminal and identifies consumables and checks their inventory status. The input is text data, and the system uses database technology to check the inventory quantity and expiration date of the identified consumables. Database operations such as SQL are performed to search for data corresponding to the product information. The output of this step is inventory information for the identified consumables.

[0871] Step 3:

[0872] The server inputs the text data from the previous step into the sentiment analysis engine to estimate the user's emotional state. Here, natural language processing techniques (e.g., IBM Watson Tone Analyzer) are used to analyze emotional elements from the text. Specifically, an algorithm is applied that identifies emotions from keywords and context in the text. The output of this step is the estimated emotional state of the user.

[0873] Step 4:

[0874] The server makes optimal consumable purchase suggestions based on inventory information and the user's emotional state. The inputs here are inventory information and emotional state, which are used to generate product suggestions tailored to the user's preferences using an AI model. This process analyzes existing purchase history and emotional analysis results, inputting prompts into the AI ​​to generate appropriate suggestions. The output of this step is a personalized purchase suggestion for the user.

[0875] Step 5:

[0876] The server adjusts notification timing based on the user's emotional state. It uses emotional state and expiration date information as input to determine an appropriate notification schedule. Based on this identified schedule, it sends an alert in advance if the expiration date is approaching. The output of this step is a timely notification to the user.

[0877] Step 6:

[0878] The server evaluates purchase data and sentiment analysis results collected from users within a region to formulate group buying suggestions. The input is data from multiple users, which is analyzed to identify products that consistently receive high ratings. Based on this, the server provides users with group buying options. The output of this step is a group buying suggestion for all users.

[0879] (Application Example 2)

[0880] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0881] Conventional consumables management systems and electronic payment services often fail to consider the user's emotional state when making purchase suggestions, resulting in an inadequate user experience. Furthermore, automated ordering and expiration date management lack sufficient consideration for reducing the psychological burden on users. Therefore, there is a need for systems that provide appropriate product suggestions and notifications tailored to the user's current emotional state.

[0882] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0883] In this invention, the server includes means for receiving voice input and converting voice data into text data; means for analyzing the converted text data and checking the inventory status of the corresponding items; means for selecting the optimal items based on user settings and placing automatic orders; means for acquiring expiration date information and notifying the user that the expiration date is approaching; means for analyzing purchase data of multiple users in a region and proposing joint purchases; means for analyzing the characteristics of voice data and estimating the user's emotional state; and means for optimizing suggestions based on the emotional state and notifying the user of the information. This enables purchase suggestions and consumable management that are tailored to the user's emotions.

[0884] "Voice input" is a method by which users provide information to a system through their voice.

[0885] "Audio data" refers to audio signals obtained from audio input that have been converted into digital data.

[0886] "Text data" refers to string information obtained by analyzing audio data.

[0887] "Information processing means" refers to a device or function that processes data according to a set procedure and utilizes the results obtained.

[0888] "Inventory status" refers to information indicating the current level of a particular item or resource.

[0889] "Automatic ordering" is a process that automatically places orders for goods based on pre-set conditions.

[0890] "Expiration date information" refers to data indicating the date by which a particular product's quality is guaranteed.

[0891] "Group purchasing" is a form of purchasing where multiple users cooperate to buy the same item, aiming to reduce costs and improve efficiency.

[0892] "Emotion analysis" is a technology that estimates a person's emotional state using voice and text data.

[0893] "Optimizing proposals" is the process of presenting the most suitable options and information based on the user's needs and circumstances.

[0894] The system for implementing this invention mainly uses a server, a user terminal, speech recognition software, and an emotion analysis engine. The operation of each component is described below.

[0895] First, the user provides voice input to the user terminal. This terminal uses speech recognition software to convert the voice data into text data. Widely used software, such as the SpeechRecognition library, can be used for speech recognition. Once the text data is generated, the server receives it and checks the inventory status of the corresponding item. In the backend, database queries are used to connect with the inventory management system.

[0896] Furthermore, this server uses an emotion analysis engine to estimate the user's emotional state from the obtained voice data. Existing emotion analysis libraries and generative AI models can be used for this emotion analysis. For example, if a positive emotion is detected, the server will suggest a slightly luxurious item to the user. This suggestion is optimized by taking into account past purchase history and purchasing patterns.

[0897] When a user selects a suggested item, the server automatically places an order. This ordering process automates the purchase procedure based on the user's existing settings. Simultaneously, the server has a function to notify the user when an item is nearing its expiration date, based on expiration date information. This is an important feature to reduce the user's burden, and expiration date information is collected and managed via an API.

[0898] For example, if a user voice-inputs, "I'm in a good mood today, so I'd like to enjoy a special wine with dinner," the system will check the wine's inventory and suggest a slightly more elegant option than usual, along with a positive sentiment. Furthermore, based on the user's past preference patterns, it will determine if there are any limited-time offers or promotions available.

[0899] An example of a prompt for a generative AI model would be a question like, "Design logic to suggest luxury items when the user expresses an emotion they are enjoying." This prompt contributes to improving emotion-based purchase suggestion logic.

[0900] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0901] Step 1:

[0902] The user provides shopping requests to the terminal via voice input. The terminal uses voice recognition software to convert the voice data into text data. In this process, a voice recognition library analyzes the voice signal and generates corresponding string data.

[0903] Step 2:

[0904] The terminal sends the converted text data to the server. The server receives this text data and executes a database query to check the inventory status of the items. Here, the server takes the text data as input, retrieves the corresponding product information from the database, and outputs whether or not the item is in stock.

[0905] Step 3:

[0906] The server uses a generative AI model to analyze the user's emotional state from text data. The emotion analysis engine takes text data as input, analyzes features extracted from the text, and outputs an estimated result of the emotional state. For example, it can determine whether the user's statements are positive.

[0907] Step 4:

[0908] The server generates optimal product recommendations using the results of sentiment analysis. It selects suggested items considering the user's emotional state and past purchase history. The inputs used are emotional state and purchase history, and the server outputs a list of recommended products, including luxury goods and new products.

[0909] Step 5:

[0910] The user selects the suggested items. The server then places an automatic order based on this selection. In this step, the ordering process proceeds using the selected item information as input, and an order confirmation is output.

[0911] Step 6:

[0912] The server retrieves expiration date information and notifies the user. Here, it retrieves information about items nearing their expiration date from a database, uses that information as input, and outputs an expiration date notification. This includes specific actions such as notifying the user via email or in-app notification.

[0913] Step 7:

[0914] The server analyzes purchasing trends within the region and proposes group purchases to other users. This process takes purchase data as input and outputs suggestions for cost reductions and benefits through group purchasing. This is where actions that encourage collaborative behavior among users take place.

[0915] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0916] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0917] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0918] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0919] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0920] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0921] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0922] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0923] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0924] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0925] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0926] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0927] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0928] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0929] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0930] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0931] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0932] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0933] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0934] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0935] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0936] The following is further disclosed regarding the embodiments described above.

[0937] (Claim 1)

[0938] An information processing device that receives voice input and converts voice data into text data,

[0939] An information processing device that analyzes converted text data and checks the inventory status of the corresponding items,

[0940] An information processing device that selects the most suitable items based on user settings and places an automatic order,

[0941] An information processing device that acquires expiration date information and notifies the user that the expiration date is approaching,

[0942] An information processing device that analyzes purchase data from multiple users within a region and proposes joint purchasing,

[0943] A system that includes this.

[0944] (Claim 2)

[0945] The system according to claim 1, further comprising means for visualizing the user's purchase history and inventory status on a user interface.

[0946] (Claim 3)

[0947] The system according to claim 1, comprising means for coordinating delivery dates within a region and improving delivery efficiency.

[0948] "Example 1"

[0949] (Claim 1)

[0950] Information processing means that acquires audio input and converts the audio signal into text format,

[0951] A means for receiving converted text data and performing natural language processing to identify consumables,

[0952] A method for checking the inventory status of identified consumables from the data structure and selecting the optimal product,

[0953] A means of automatically ordering products selected based on user settings,

[0954] A means of obtaining expiration date information and notifying the user that the expiration date is approaching,

[0955] A means of analyzing purchase data from multiple users and providing opportunities for group purchases,

[0956] A system that includes this.

[0957] (Claim 2)

[0958] The system according to claim 1, further comprising means for visually presenting the user's purchase history and inventory status in a user interface.

[0959] (Claim 3)

[0960] The system according to claim 1, comprising means for coordinating delivery dates within a region and improving logistics efficiency.

[0961] "Application Example 1"

[0962] (Claim 1)

[0963] An information processing means that receives voice input and converts the voice data into text data,

[0964] An information processing means that analyzes the converted text data and checks the inventory status of the corresponding materials,

[0965] An information processing system that selects the optimal materials based on user settings and places automatic orders,

[0966] An information processing means that obtains expiration date information and notifies the user that the expiration date is approaching,

[0967] An information processing tool that analyzes the purchase data of multiple users within a region and proposes collaborative purchasing,

[0968] An information processing means for a household automated device that has the function of interacting with the user via voice and notifying them of inventory information,

[0969] A system that includes this.

[0970] (Claim 2)

[0971] The system according to claim 1, comprising means for visualizing the user's purchase history and inventory status on a user interface.

[0972] (Claim 3)

[0973] The system according to claim 1, comprising means for coordinating delivery dates within a region and improving logistics efficiency.

[0974] "Example 2 of combining an emotion engine"

[0975] (Claim 1)

[0976] A device that receives voice input and converts voice data into text data,

[0977] A device that analyzes converted text data and checks the inventory status of the corresponding consumables,

[0978] A device that estimates the user's emotional state based on voice input,

[0979] A device that selects the optimal consumables and makes purchasing suggestions based on emotional state and user settings,

[0980] A device that acquires expiration date information and adjusts the timing of notifications according to the user's emotions,

[0981] A device that analyzes the purchase data and emotional state of multiple users within a region and proposes group purchases,

[0982] A system that includes this.

[0983] (Claim 2)

[0984] The system according to claim 1, comprising means for visualizing the user's purchase history and inventory status on a user interface, and for displaying content including sentiment analysis results.

[0985] (Claim 3)

[0986] The system according to claim 1, comprising means for coordinating delivery dates within a region, improving delivery efficiency, and providing flexible delivery options that respond to the user's emotional state.

[0987] "Application example 2 when combining with an emotional engine"

[0988] (Claim 1)

[0989] An information processing means that receives voice input and converts the voice data into text data,

[0990] Information processing means for analyzing converted text data and confirming the inventory status of the corresponding items,

[0991] An information processing means that selects the most suitable items based on user settings and places an automatic order,

[0992] An information processing means that obtains expiration date information and notifies the user that the expiration date is approaching,

[0993] An information processing tool that analyzes purchase data from multiple users within a region and proposes joint purchasing,

[0994] An emotion analysis method that analyzes the characteristics of voice data to estimate the user's emotional state,

[0995] An information processing means that optimizes suggestions and notifies information based on emotional state,

[0996] A system that includes this.

[0997] (Claim 2)

[0998] The system according to claim 1, further comprising means for visualizing the user's purchase history and inventory status on a user interface.

[0999] (Claim 3)

[1000] The system according to claim 1, comprising means for coordinating delivery dates within a region and improving delivery efficiency. [Explanation of symbols]

[1001] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. An information processing means that receives voice input and converts the voice data into text data, An information processing means that analyzes the converted text data and checks the inventory status of the corresponding materials, An information processing system that selects the optimal materials based on user settings and places automatic orders, An information processing means that obtains expiration date information and notifies the user that the expiration date is approaching, An information processing tool that analyzes the purchase data of multiple users within a region and proposes collaborative purchasing, An information processing means for a household automated device that has the function of interacting with the user via voice and notifying them of inventory information, A system that includes this.

2. The system according to claim 1, comprising means for visualizing the user's purchase history and inventory status on a user interface.

3. The system according to claim 1, comprising means for coordinating delivery dates within a region and improving logistics efficiency.

Images